Publish qsv profiles in the Fair Store, deploy it to Compute Engine, and manage it from Verikan

.env.example

@@ -78,6 +78,12 @@ DATA_COMMONS_API_KEY=
 CKAN_URL=https://data.dathere.com
 CKAN_API_KEY=
 
+# Fair Store — the CKAN that mirrors every registered portal. Seeds for the
+# admin panel's Fair Store page (values saved there take precedence). The token
+# is a CKAN sysadmin API token; it is only ever sent to FAIRSTORE_URL's host.
+# FAIRSTORE_URL=https://fairstore.example.org
+# FAIRSTORE_API_KEY=
+
 # WPRDC (Western PA Regional Data Center) - City of Pittsburgh open data
 WPRDC_CKAN_URL=https://data.wprdc.org
 WPRDC_ORGANIZATION=city-of-pittsburgh

.gitignore

@@ -158,11 +158,17 @@ logs/
 *.log
 .secrets/
 
+# API token files (deploy/fairstore/deploy.sh writes the mirror's outside the
+# repository; these catch one saved here by mistake)
+fairstore-token*
+*token.txt
+
 # Runtime-seeded local storage (contains user data / secrets — never commit)
 /users.json
 /roles.json
 /notebook_verification/
 /github_settings.json
+/fairstore_settings.json
 /landing_settings.json
 /system_prompt_settings.json
 /runtime_settings.json

README.md

@@ -360,36 +360,127 @@ examples/                    # Sample generated notebooks
 
 ## CKAN Fair Store mirror
 
-Start the local CKAN 2.11 Fair Store with organization hierarchy support:
+Start the local CKAN 2.11 Fair Store with organization hierarchy support.
+Set `FAIRSTORE_SECRET_KEY` (any long random string, e.g. in `.env`) so API
+tokens and sessions survive the container being recreated:
 
 ```bash
 docker compose --profile fairstore up -d --build fairstore
 ```
 
-Preview a registered CKAN portal before writing anything:
+On first run, create a sysadmin and an API token for the mirror (keep the token
+out of the repository):
 
 ```bash
-python -m scripts.populate_fairstore \
-  --site wprdc \
-  --target-url http://localhost:5001
+docker compose exec fairstore ckan -c /srv/app/ckan.ini user add fairadmin email=fairadmin@localhost.localdomain password=<password>
+docker compose exec fairstore ckan -c /srv/app/ckan.ini sysadmin add fairadmin
+docker compose exec fairstore ckan -c /srv/app/ckan.ini user token add fairadmin mirror
+```
+
+Preview every registered portal (CKAN and DCAT) before writing anything, or
+pass one `--site` ID:
+
+```bash
+python -m scripts.populate_fairstore --site all --target-url http://localhost:5001
 ```
 
-To apply the mirror, create a target CKAN sysadmin token, expose it through
-`CKAN_API_KEY` (or a protected file), and add `--apply`. The command performs a
-collision preflight first, then preserves source organization, dataset, and
-resource names and UUIDs. It creates one parent organization for the source
-portal, attaches source organizations beneath it, and adds any matching qsv
-profile to the resource as separate metadata. Re-running patches the preserved
-UUIDs, so it does not duplicate resources.
+Add `--apply` with the token in `FAIRSTORE_API_KEY` or `--api-key-file` to
+write (`FAIRSTORE_URL` sets the default `--target-url`; the app's own
+`CKAN_URL`/`CKAN_API_KEY` are deliberately not used, and a portal that is the
+target is never mirrored into itself). Every read fails closed — a portal that
+errors mid-read is not mirrored from a partial snapshot — and with `--site all`
+one portal's failure does not stop the others (the exit status is non-zero).
+The command runs a collision preflight first, then:
+
+- creates one parent organization per source portal and attaches the portal's
+  publishers beneath it (ckanext-hierarchy);
+- for a **CKAN** portal, preserves organization, group, dataset, and resource
+  names and UUIDs;
+- for a **DCAT** portal (`/data.json` from Socrata, DKAN, ArcGIS Hub), turns
+  publishers into organizations, themes into groups, and distributions into
+  resources, with UUIDv5 identifiers derived from the catalog's own IDs. A
+  Socrata catalog is enriched from the Socrata Discovery API with the owning
+  agency (the hierarchy) and the column list (`source_data_dictionary`); pass
+  `--no-enrich` to skip it;
+- keeps every source field: anything the Fair Store's default schema has no
+  column for (ckanext-scheming fields, DCAT-US fields) becomes an extra, and
+  links to files uploaded to the source keep pointing at the source;
+- publishes each dataset's qsv profile (from `data/{ckan,dcat}_onboard/`): the
+  AI description (its prose; describegpt's provenance block stays in the raw
+  output), AI tags, row and column counts and column names become `qsv_*`
+  fields on the dataset, and four resources are added to it — a data
+  dictionary (types, AI labels and descriptions, key statistics), the full
+  `qsv stats` and `qsv frequency` tables (DataStore tables, shown as sortable
+  tables and downloadable as CSV/JSON), and the raw describegpt JSON. API keys
+  that describegpt writes into its attribution are redacted, and stats,
+  frequency or describegpt output that does not match the profiled file's
+  header (left behind by another file's profile in the same directory) is not
+  published. The source data itself is never copied;
+- skips records the source itself mirrored from another portal, so each
+  dataset is mirrored once from its origin.
+
+Re-running patches the preserved UUIDs, so it does not duplicate anything,
+and it only writes what changed: each mirrored record carries a digest of what
+was last written (`mirror_digest`, `qsv_digest`), so an unchanged dataset,
+resource or qsv table is left alone. A record the source renamed is renamed
+(DCAT names are derived, so those keep the name they were published under);
+records the source no longer publishes are reported (`withdrawn_at_source`),
+never deleted. The dry run reports the same created/updated/unchanged plan.
 
 ```bash
 python -m scripts.populate_fairstore \
-  --site wprdc \
+  --site all \
   --target-url http://localhost:5001 \
   --api-key-file /path/to/protected-token \
   --apply
 ```
 
+### Managing the Fair Store from Verikan
+
+The admin panel's **Fair Store** page links Verikan to it:
+
+- **Connection** — the Fair Store URL and a sysadmin API token (`ckan user
+  token add <sysadmin> verikan` on the Fair Store). `FAIRSTORE_URL` and
+  `FAIRSTORE_API_KEY` seed these; values saved on the page take precedence.
+  The token is never shown again and is only ever sent to the host it was saved
+  for: moving the URL to another host drops it.
+- **Chat source** — optionally offers the Fair Store as a data source in chat.
+  The agent searches its catalog and loads each dataset's rows from the portal
+  it was mirrored from.
+- **Mirror runs** — dry runs and writes of every portal or one, with a live log
+  and a per-portal summary. On Cloud Run the qsv profiles come from the storage
+  backend that onboarding syncs to (`--qsv-source`).
+- **Site settings** — the Fair Store's title, description, home page and about
+  text, logo and custom CSS, read and saved live.
+
+The chat sidebar and the data dictionary link to the Fair Store once it is set.
+
+### Deploying the Fair Store on GCP
+
+`deploy/fairstore/deploy.sh` runs the same stack on one Compute Engine VM next
+to Verikan (in `us-central1` by default; `PROJECT` names the GCP project): CKAN on uwsgi
+(built from `ckan/ckan-base`), Postgres, Solr, Redis, and Caddy for automatic
+HTTPS. It is idempotent — the first run creates the static IP, firewall rule,
+VM and a daily snapshot schedule; later runs re-ship `docker/fairstore` and
+restart the stack. Secrets are generated on the VM (`/opt/fairstore/.env`) and
+never leave it. The host is `FAIRSTORE_HOST` if set, else the one already
+saved on the VM, else `<ip>.sslip.io` (first deploy); point a DNS name at the IP
+and rerun with `FAIRSTORE_HOST` set to move it (the search index is rebuilt
+for the new URLs).
+
+The VM has no service account (the site calls no Google API; a VM that still
+has one is stopped once to remove it), and containers are blocked from the
+metadata server except for DNS.
+
+```bash
+gcloud auth login
+PROJECT=<gcp-project-id> deploy/fairstore/deploy.sh
+```
+
+The script prints the command that mints the mirror's API token into
+`~/.config/verikan/fairstore-token` (outside the repository); then run the
+mirror above with `--target-url https://<host>` and that `--api-key-file`.
+
 ## Development
 
 ```bash

deploy/fairstore/caddy/Caddyfile

@@ -0,0 +1,48 @@
+# HTTPS for the Fair Store; Caddy obtains and renews the certificate itself.
+#
+# Mounted as a directory (./caddy:/etc/caddy), not as a single file: a
+# redeploy's tar replaces the file with a new inode, and a single-file bind
+# mount keeps serving the old one. remote-up.sh runs `caddy reload` afterwards.
+{
+	servers {
+		# Only TCP 443 is published and allowed through the firewall, so
+		# don't advertise HTTP/3 (UDP) to clients that would try it first.
+		protocols h1 h2
+	}
+}
+
+{$FAIRSTORE_HOST} {
+	encode zstd gzip
+
+	# Access log with the client's address; CKAN only ever sees Caddy's.
+	log {
+		output stdout
+		format json
+	}
+
+	header {
+		Strict-Transport-Security "max-age=31536000"
+		X-Content-Type-Options nosniff
+		Referrer-Policy strict-origin-when-cross-origin
+		# Don't advertise the server software (Caddy adds both).
+		-Server
+		-Via
+	}
+
+	# CKAN's "Embed" button hands out an iframe of a resource view page
+	# (/dataset/<id>/resource/<id>/view/<id>), so only those stay frameable by
+	# other sites.
+	@not_view not path */view/*
+	header @not_view {
+		X-Frame-Options SAMEORIGIN
+		defer
+	}
+
+	reverse_proxy fairstore:5000 {
+		# uwsgi's http router closes idle keep-alive connections; a POST sent on
+		# a pooled one fails with a 502, and Go never retries a non-idempotent request.
+		transport http {
+			keepalive off
+		}
+	}
+}

deploy/fairstore/deploy.sh

@@ -0,0 +1,174 @@
+#!/usr/bin/env bash
+# Deploy the Fair Store (CKAN 2.11 + ckanext-hierarchy) next to Verikan.
+#
+# One Compute Engine VM runs the same stack as the local `fairstore` compose
+# profile — CKAN, Postgres, Solr, Redis — plus Caddy for automatic HTTPS. A VM
+# rather than Cloud Run because Postgres and Solr need persistent disks.
+#
+# Idempotent: creates the static IP, firewall rule, VM and daily disk snapshots
+# on the first run, then (re)ships docker/fairstore + this directory and
+# restarts the stack. Secrets are generated on the VM and stay there.
+#
+#   gcloud auth login              # the session expires; re-auth is interactive
+#   PROJECT=<gcp-project-id> deploy/fairstore/deploy.sh
+#
+# The VM runs without a service account: the site calls no Google API, and the
+# default one is a project Editor. A VM that still has one is stopped, stripped
+# of it and started again — a few minutes of downtime, once.
+#
+# The host is FAIRSTORE_HOST if set, else the one the VM already serves (its
+# .env), else <ip>.sslip.io, a wildcard DNS name that resolves to the VM so
+# Caddy can get a certificate before a real domain exists. Point a DNS name at
+# the IP and rerun with FAIRSTORE_HOST set to move it.
+set -euo pipefail
+
+PROJECT="${PROJECT:?set PROJECT to the GCP project id to deploy into}"
+REGION="${REGION:-us-central1}"
+ZONE="${ZONE:-us-central1-a}"
+VM="${VM:-verikan-fairstore}"
+MACHINE_TYPE="${MACHINE_TYPE:-e2-medium}"
+DISK_SIZE="${DISK_SIZE:-50GB}"
+NETWORK_TAG="${VM}-web"
+REMOTE_DIR=/opt/fairstore
+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+STARTUP_SCRIPT="$ROOT/deploy/fairstore/vm-startup.sh"
+# Outside the repository, so the mirror's sysadmin token can never be committed.
+TOKEN_FILE="${XDG_CONFIG_HOME:-$HOME/.config}/verikan/fairstore-token"
+
+gc() { gcloud --project "$PROJECT" --quiet "$@"; }
+on_vm() { gc compute ssh "$VM" --zone "$ZONE" --command "$1"; }
+vm_field() { gc compute instances describe "$VM" --zone "$ZONE" --format="value($1)"; }
+
+# wait_for <timeout seconds> <seconds between tries> <what> <command...>
+# Prints a dot per failed try and, on timeout, the last try's output (the why).
+wait_for() {
+    local limit="$1" pause="$2" what="$3" deadline out
+    shift 3
+    deadline=$((SECONDS + limit))
+    printf 'Waiting for %s ' "$what"
+    until out="$("$@" 2>&1)"; do
+        if ((SECONDS >= deadline)); then
+            echo
+            echo "deploy: gave up waiting for $what after $((limit / 60)) min; last try said:" >&2
+            printf '%s\n' "$out" | tail -n 20 >&2
+            exit 1
+        fi
+        printf .
+        sleep "$pause"
+    done
+    echo " ok"
+}
+
+gc projects describe "$PROJECT" --format='value(projectId)' >/dev/null \
+    || { echo "gcloud cannot reach $PROJECT; run: gcloud auth login" >&2; exit 1; }
+
+if ! gc compute addresses describe "${VM}-ip" --region "$REGION" >/dev/null 2>&1; then
+    gc compute addresses create "${VM}-ip" --region "$REGION"
+fi
+IP="$(gc compute addresses describe "${VM}-ip" --region "$REGION" --format='value(address)')"
+
+if ! gc compute firewall-rules describe "${VM}-web" >/dev/null 2>&1; then
+    gc compute firewall-rules create "${VM}-web" --network default --direction INGRESS \
+        --allow tcp:80,tcp:443 --target-tags "$NETWORK_TAG" --source-ranges 0.0.0.0/0
+fi
+
+if ! gc compute instances describe "$VM" --zone "$ZONE" >/dev/null 2>&1; then
+    gc compute instances create "$VM" --zone "$ZONE" --machine-type "$MACHINE_TYPE" \
+        --image-family debian-12 --image-project debian-cloud \
+        --boot-disk-size "$DISK_SIZE" --boot-disk-type pd-balanced \
+        --address "$IP" --tags "$NETWORK_TAG" \
+        --no-service-account --no-scopes --deletion-protection \
+        --labels app=verikan,component=fairstore \
+        --metadata-from-file startup-script="$STARTUP_SCRIPT"
+else
+    status="$(vm_field status)"
+    sa="$(vm_field 'serviceAccounts[].email')"
+    if [ -n "$sa" ]; then
+        echo "$VM runs as $sa, which the site never uses; removing it."
+        echo "That needs the VM stopped: the Fair Store is down for a few minutes."
+        [ "$status" = TERMINATED ] || gc compute instances stop "$VM" --zone "$ZONE"
+        gc compute instances set-service-account "$VM" --zone "$ZONE" \
+            --no-service-account --no-scopes
+        status=TERMINATED
+    fi
+    # The boot disk holds every database: guard the VM against deletion.
+    gc compute instances update "$VM" --zone "$ZONE" --deletion-protection
+    # GCE runs the startup script stored in instance metadata, so ship the
+    # current one; it applies from the next boot (remote-up.sh covers this one).
+    gc compute instances add-metadata "$VM" --zone "$ZONE" \
+        --metadata-from-file startup-script="$STARTUP_SCRIPT"
+    if [ "$status" = TERMINATED ]; then
+        echo "Starting $VM..."
+        gc compute instances start "$VM" --zone "$ZONE"
+    fi
+fi
+
+# Postgres, Solr and uploads all live on the boot disk: keep a week of daily
+# snapshots.
+if ! gc compute resource-policies describe "${VM}-daily" --region "$REGION" >/dev/null 2>&1; then
+    gc compute resource-policies create snapshot-schedule "${VM}-daily" --region "$REGION" \
+        --daily-schedule --start-time 07:00 --max-retention-days 7 \
+        --on-source-disk-delete keep-auto-snapshots
+fi
+policies="$(gc compute disks describe "$VM" --zone "$ZONE" --format='value(resourcePolicies)')"
+case ";${policies};" in
+    *"/resourcePolicies/${VM}-daily;"*) ;;
+    *) gc compute disks add-resource-policies "$VM" --zone "$ZONE" --resource-policies "${VM}-daily" ;;
+esac
+
+# `compose ls` needs both the compose plugin and a running daemon.
+wait_for 900 15 "Docker on $VM" on_vm "sudo docker compose ls"
+
+# Default to the host the VM already serves: falling back to sslip.io on every
+# run would silently move a site that has been given a real domain.
+if [ -n "${FAIRSTORE_HOST:-}" ]; then
+    HOST="$FAIRSTORE_HOST"
+else
+    HOST="$(on_vm "if sudo test -f $REMOTE_DIR/.env; then sudo sed -n 's/^FAIRSTORE_HOST=//p' $REMOTE_DIR/.env; fi" \
+        | tail -n 1 | tr -d '\r')"
+    HOST="${HOST:-${IP//./-}.sslip.io}"
+fi
+[[ "$HOST" =~ ^[A-Za-z0-9.-]+$ ]] || { echo "deploy: bad host name: '$HOST'" >&2; exit 1; }
+echo "Deploying to https://$HOST"
+
+bundle="$(mktemp -d)"
+trap 'rm -rf "$bundle"' EXIT
+mkdir -p "$bundle/stack/ckan"
+cp "$ROOT"/deploy/fairstore/{docker-compose.yml,remote-up.sh} "$bundle/stack/"
+cp -R "$ROOT"/deploy/fairstore/caddy "$bundle/stack/"
+cp -R "$ROOT"/docker/fairstore/. "$bundle/stack/ckan/"
+# macOS tar would add AppleDouble ._* files and xattrs, and record the local
+# user as owner. Members are named rather than ".", so extracting never resets
+# the mode of /opt/fairstore itself.
+COPYFILE_DISABLE=1 tar --no-xattrs --owner=root:0 --group=root:0 --exclude .DS_Store \
+    -C "$bundle/stack" -czf "$bundle/fairstore.tgz" docker-compose.yml remote-up.sh caddy ckan
+gc compute scp "$bundle/fairstore.tgz" "$VM:/tmp/fairstore.tgz" --zone "$ZONE"
+# ckan/ is only a build context (its init-db.sh runs only when Postgres
+# initialises an empty volume), so it is replaced whole and files deleted here
+# disappear there. .env is never touched, and neither is the caddy/ directory
+# itself: Caddy bind-mounts it, so only the files inside it are replaced.
+# The first deploys' bundles left macOS ._* files and the operator's ownership
+# at the top level; both are tidied here.
+on_vm "sudo mkdir -p $REMOTE_DIR && sudo chown root:root $REMOTE_DIR \
+    && sudo find $REMOTE_DIR -maxdepth 1 -name '._*' -delete \
+    && sudo rm -rf $REMOTE_DIR/ckan \
+    && sudo tar --no-same-owner -xzf /tmp/fairstore.tgz -C $REMOTE_DIR \
+    && rm -f /tmp/fairstore.tgz \
+    && sudo bash $REMOTE_DIR/remote-up.sh $HOST"
+
+wait_for 600 10 "https://$HOST" \
+    curl -fsS -o /dev/null --max-time 20 "https://$HOST/api/3/action/status_show"
+cat <<EOF
+
+Fair Store is up: https://$HOST
+
+Mint an API token for the mirror into a private file outside the repository:
+  (umask 077 && mkdir -p "${TOKEN_FILE%/*}" && gcloud compute ssh $VM --zone $ZONE --project $PROJECT --command \\
+    "cd $REMOTE_DIR && sudo docker compose exec -T fairstore ckan -c /srv/app/ckan.ini user token add fairadmin mirror" \\
+    | tail -1 > "$TOKEN_FILE")
+then load every registered portal:
+  python -m scripts.populate_fairstore --site all --apply \\
+    --target-url https://$HOST --api-key-file "$TOKEN_FILE"
+
+The fairadmin password is FAIRSTORE_ADMIN_PASSWORD in $REMOTE_DIR/.env on the VM.
+EOF