fix(resolver): mart locali nello scan + perf find + discovery dir≠slug - #460
Merged
Conversation
Cleanup audit: due file senza consumer reali. - cmd_batch.py: comando 'toolkit batch' deprecato da tempo, sostituito da 'toolkit run --batch' (cmd_run.py). Unico riferimento: registrazione in app.py. - sql_dry_run.py: shim di reindirizzamento verso core.sql_validation; nessun import (cmd_run usa sql_validation direttamente). Rimossa registrazione register_batch da app.py. Nessun test dipendeva dai file rimossi (1324 verdi invariati), ruff pulito, zero riferimenti orfani.
Tre fix del resolver, emersi da audit e prove cross-repo:
1. mart locali: _scan_workspace_parquets scansionava solo *_clean.parquet —
i mart locali (bucket local-mart, naming canonico mart_{table}.parquet)
non erano risolvibili in catalog mode; dedup per (slug,year,bucket,table)
per non collidere tabelle diverse; gestiti flat multi-anno (year=None).
2. perf find: scan parquet solo sui repo dati (registry/ o datasets/) —
esclusi project residui (open-conto-annuale, out/, data/, ...) con
rglob costosi; cache YAML mtime-based in read_yaml (PyYAML ~7ms/file,
~1.1s su 158 dataset.yml a ogni istanza).
3. discovery: resolve_config_path Stage 3 cercava {slug} letterale ma le
dir possono essere hyphen ≠ slug underscore (es. eurostat-nrg-chddr2-a)
— ora prova entrambe le forme. 3/39 dataset eurostat/bologna colpiti.
Test: +5 (scan mart con/flat/skip non-data, resolve mart locale+gcs,
resolve dir≠slug). 1329 verdi, ruff pulito.
…ionale) CI: mypy falliva su catalog.py — dedup_key riusata con 3-tuple (clean) e 4-tuple (mart) e year None su variabile int. Variabili dedicate (mart_slug/mart_year/mart_key) + tipo union per seen.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tipo
root fix (3 problemi del resolver emersi da audit + prove cross-repo MCP).
Problema reale
Mart locali non leggibili:
_scan_workspace_parquetsscansionava solo*_clean.parquet— i mart su disco (bologna/eurostat/DI, 479 file) non erano risolvibili in catalog mode.resolve_slug(layer='mart')mostrava solo i mart GCS. Inoltre la dedup(slug, year, bucket)faceva collidere 2 mart dello stesso dataset+anno con tabelle diverse (vinceva il primo).Find lento:
_scan_workspace_parquetsfacevarglobsu tutti i repo del workspace (48 dir), inclusi project residui non-dati (open-conto-annuale 106 parquet, open-siope, out/, data/) → 3.8s al primo find di ogni istanza MCP.read_yaml(PyYAML ~7ms/file) riparsava 158 dataset.yml a ogni istanza.Discovery dir≠slug:
resolve_config_pathStage 3 cercava{section}/{slug}/dataset.ymlcon lo slug letterale (underscore), ma le dir reali possono essere hyphen ≠ slug (es.eurostat-nrg-chddr2-a-nuts3vs slugeurostat_nrg_chddr2_a_nuts3). Colpiva 3/39 dataset eurostat/bologna.Contratto riusato/sostituito
mart_{table}.parquet(già usato dalla pipeline per scrivere);repo_dataset_dirs/euristica repo-dati (già usata da_scan_workspace_configs); iltablefilter diresolve_slug(già usato datoolkit_layer).read_yamlsenza cache → cache mtime-based.Implementazione
toolkit/domain/catalog.pylocal-mart, dedup con table, flat multi-annoyear=None); filtro repo dati nello scan parquet;_matches_layerriconosce local-mart; escluso None dayears(crash sorted)toolkit/core/io.pyread_yaml/read_yaml_or_none(keyed su path+mtime_ns+size)toolkit/core/discovery.pyresolve_config_pathprova entrambe le forme slug (underscore/hyphen)tests/test_catalog_ops.pytests/test_paths.pyVerifica
centraline_ariarisolve 2 mart locali + 2 GCS;spire_traffico3 locali (2 per-anno + 1 flat multi-anno) + 3 GCS; querytoolkit_layer table=mart_trendsu eurostat funziona sul localeresolve_config_path('eurostat_nrg_chddr2_a_nuts3')ora risolve; casi standard invariatiRischio residuo
spire_traffico/mart_spire_trend.parquetsenza anno) sono esposti conyear=None— coerente col fatto che il registry GCS li pubblica flat.gs://inesistenti (35 file) — il resolver le espone comehas_remote=True. Noto e fuori scope: fix al wrapper build_registry di bologna (derive_mode check-gcs), repo separato.resolve_config_pathrisolve per dir (hyphen o underscore); non risolve slug presenti solo nel registry senza dir locale (es. chddr2_m) — serve la dir o il fallback registry.Follow-up obbligatorio
--check-gcsnel wrapper per non committare location GCS inesistenti.