Publish qsv profiles in the Fair Store, deploy it to Compute Engine, and manage it from Verikan

[minhajuddin2510]

Problem

The Fair Store mirrored source metadata, but none of the qsv profiling behind Verikan's search — stats, frequency tables, AI descriptions — reached it. It also ran only locally. Deploying and verifying it against the full catalogs (WPRDC, data.dathere.com, data.pa.gov) exposed further problems:

• Partial reads looked like empty results. CKANClient.action returns {} for any failure. One 502 while paging a source's organizations would have moved the unseen organizations' datasets to the portal root on --apply. A failed Socrata Discovery call would have dropped every data.pa.gov agency.

• Retries could double DataStore rows. A retried insert, or a failed datastore_delete on a rerun, could append a table's rows a second time.

• Some qsv output described the wrong file. Onboarding writes qsv output once per dataset directory, so two WPRDC datasets carried stats, frequency and AI text computed from a different CSV. describegpt also sometimes answers Tags with a capital T, which dropped the AI tags from 21 datasets.

• describegpt's provenance block leaked into the summaries. Its command line and model details appeared in about a third of them; the API key was already redacted.

• Running it needed a terminal. Mirroring, checking and configuring the Fair Store meant SSH, a token file and the CLI. Verikan's chat couldn't use it either: its records hold metadata only, so loading a mirrored resource from the Fair Store returns nothing.

• The Admin button disappeared from the chat page. The homepage simplification moved it into the sidebar, which is collapsed by default.

Change

Mirror (scripts/populate_fairstore.py)

• --site all mirrors every registered portal, CKAN and DCAT. The Socrata Discovery API supplies the owning agency (the hierarchy) and the column list.
• Each qsv profile is published as:
  • qsv_* dataset fields: the AI summary (prose only), AI tags, row and column counts, the profiled columns as a JSON list, model and version;
  • four resources: a data dictionary, and the qsv stats and qsv frequency tables (DataStore tables with table views), plus the describegpt JSON.
• The vetting step (_vet_profile) publishes stats, frequency and describegpt output only when it matches the profiled file's header.
• Reads fail closed:
  • reads go through the new CKANClient.call / CKANActionError, which distinguish transient from not-found errors (action() keeps its old contract for the app);
  • paging is sorted by name and cross-checked against the full name listing;
  • Socrata enrichment is required under --apply.
• Writes are retry-safe:
  • a create is resolved by its UUID and never double-counted;
  • DataStore loads are verified by row count and rebuilt, and a table's digest is written only after its rows are verified;
  • view creation re-lists after a failure, so it never adds a duplicate view.
• Reruns write only what changed, using mirror_digest and qsv_digest.
  • Values the source cleared are cleared here too: source resources are replaced, not merged.
  • Source renames are applied. DCAT-derived names stay pinned so URLs don't change.
  • Records withdrawn at the source are reported, never deleted.
• Run control:
  • one portal's failure no longer stops the others (the exit status is non-zero);
  • a portal that is itself the target is skipped;
  • the token and default target come from FAIRSTORE_API_KEY / FAIRSTORE_URL, not the app's CKAN_* settings.

Container (docker/fairstore)

• CKAN 2.11.3 → 2.11.6, which fixes the advisories published since 2.11.3.
• Hooks:
  • pin the signing secrets, so API tokens survive a recreated container;
  • add an AI-summary template;
  • set Secure cookies on HTTPS;
  • set SameSite=Lax on the remember-me cookie.
• CKAN__CSRF_PROTECTION__IGNORE_EXTENSIONS=false and CKAN__UPLOADS_ENABLED=true are set explicitly.
• Redis and the DataStore views are added to the local compose profile.

Deployment (deploy/fairstore, new)

• PROJECT=<id> deploy/fairstore/deploy.sh is idempotent. It runs one Compute Engine VM with CKAN on uwsgi, Postgres, Solr, Redis, and Caddy for automatic HTTPS (<ip>.sslip.io until DNS is pointed at the VM).
• Hardening:
  • no service account;
  • containers blocked from the metadata server, except DNS;
  • HSTS, nosniff, Referrer-Policy and X-Frame-Options (view pages stay embeddable);
  • no Server header and no HTTP/3 advert.
• Operations:
  • access logs with log rotation;
  • daily disk snapshots and deletion protection;
  • secrets generated on the VM and never shipped;
  • bounded waits, the site's saved host kept across redeploys, and a search reindex when the host changes.
• Caddy mounts its config as a directory, so a redeploy's Caddyfile actually takes effect. Upstream keep-alive is off, which fixes the spurious 502s on POSTs to uwsgi.

Verikan ↔ Fair Store (gateway/fairstore.py, Admin → Fair Store)

• Connection: the Fair Store URL and a sysadmin token (write-only in the UI), seeded from FAIRSTORE_URL / FAIRSTORE_API_KEY.
  • The token is bound to the origin (scheme, host and port) it was saved for, so a URL edit can never send it to another server.
  • A URL that is already a registered source portal is refused, so a mirror can never write into one.
• Mirror runs: dry runs or writes, for every portal or one, with a live log, a per-portal summary and cancel.
  • They run under the onboarding job runner (one job at a time, token in the environment, never argv).
  • Running jobs heartbeat their record and use absolute log offsets, so any Cloud Run instance can show or cancel a run.
• Runs on Cloud Run read qsv profiles from the storage backend (--qsv-source):
  • sync_to_storage now writes sync_manifest.json (CSV headers, dataset directories, qsv outputs present), last;
  • JSON is stored with API keys scrubbed;
  • staging fails closed when the manifest or a listed object is missing.
• Live status (reachability, token check, datasets per source portal, last run) and the Fair Store's runtime site settings (title, description, intro, about, logo, custom CSS).
• Chat source: the Fair Store can be offered in chat.
  • The agent searches its catalog and loads rows from the portal each dataset was mirrored from: the same UUID for a CKAN origin, the file for a DCAT one (CSV/TSV only).
  • Citations, agent-log sources and notebook code name the portal that served each call. The notebook editor shares the same per-call path.
• The chat page and the data dictionary link to the Fair Store.

Agent hardening found while testing

• Numeric tool arguments sent as strings ("limit": "100") crashed loads and were written raw into notebook code.
• A portal with DataStore SQL disabled is detected from the response body.
• DCAT not-found / no-rows answers count as failures.
• Calls that fetched nothing are commented out in notebooks.

Admin button

• Restored to the chat page's top bar (icon-only on phones); the sidebar link stays.

Validation

• Ruff passes. The full suite passes: 1014 passed, 2 skipped. 35 new focused tests cover:

  • fail-closed reads and required Socrata enrichment;
  • renames, evictions and collisions;
  • partial-load recovery and digest skipping;
  • the vetting of qsv output against the profiled file;
  • prose extraction;
  • per-portal failure isolation.

• A deployed instance was mirrored from all three portals and checked against a local reference store, dataset by dataset:

  • 921 datasets, 79 organizations under three portal roots and 44 groups;
  • 429 qsv resources on 110 WPRDC datasets;
  • every DataStore table's row count matches the onboarding output;
  • 0 API-key hits across all published metadata and files.

• Idempotency: a converged rerun plans 0 writes except genuine source drift, and takes about 2 minutes.

• Resilience: a full VM reboot recovers by itself (site back in about 70 s, firewall rules re-applied, tokens still valid).

• Several rounds of review and verification checked the code and the live deployment, with each finding re-checked by a second pass that tried to refute it. Everything confirmed is fixed here.

• 134 more tests (tests/unit/test_fairstore_link.py) cover the Fair Store link:

  • settings, including token/origin binding, conflicts and unreadable storage;
  • mirror jobs, including cross-instance heartbeats, cancels and log offsets;
  • storage staging;
  • agent routing, attribution and argument repair;
  • the notebook editor path.

• Local and storage-staged mirror runs of the 110 WPRDC profiles produced identical writes and digests, removals included. A dry run staged from the storage backend against the deployed Fair Store planned 0 qsv changes, the same as the local run.

• Chat questions answered through the deployed Fair Store:

  • Pittsburgh parks by acreage, with rows from WPRDC;
  • Pennsylvania county assistance offices, with rows from data.pa.gov.
  • Both have correct citations and notebook code.

• The admin pane was checked in a browser against the deployed Fair Store, and the Admin button on desktop and at phone width.

Not in this PR

• Project-level GCP hardening: the default SSH/RDP firewall rules and the default compute service account's roles.
• Re-mirroring data.dathere.com, whose WPRDC copy predates the upload-link fix.
• Rebuilding the WPRDC onboarding index and Pinecone records for the 21 tag sets that _extract_qsv_tags previously missed.