fix(metadata): re-vendor @jspsych/metadata from upstream main + sync tooling

[htsukamoto5]

Summary

DataPipe's vendored @jspsych/metadata at functions/metadata/ was a June-2024 fork of v0.0.1, frozen since the standalone jspsych/metadata repo took over development. This PR re-vendors it from upstream main (pinned to 224d336) and adds tooling so future re-syncs are a deliberate one-command step instead of another multi-year drift.

Why not just depend on the npm package?

Published @jspsych/metadata@0.0.3 predates the fixes we need (nested-data expansion, extraction API, array-accepting generate() — landed in upstream PRs #115/#116/#117/#120 but never released). Worse, 0.0.3's generate() internally JSON.parses its input and early-returns on failure, so a naive ^0.0.3 bump would make DataPipe (which pre-parses the data) silently emit only the default metadata template, dropping all real variables. Pinned re-vendor of upstream main is the safest path until upstream cuts a new release.

What this fixes

• Data-loss bug: vendored 0.0.1 collapsed nested object/array fields (e.g. survey response objects, mouse-tracking arrays) to contentless stubs. The re-vendored lib expands them into dotted sub-variables (response.Q0, mouse_tracking_data.x/y/t) with correct types/min/max.
• JSON-LD "type" → "@type" on variableMeasured entries (verified safe: DataPipe's merge logic never reads .type).
• Consistent "number" typing (0.0.1 hardcoded "numeric" for system fields).
• Failed plugin-description fetches now report "unknown" instead of accumulating comma-joined compound keys.
• Exposes getExtractedArrays()/getExtractedObjects()/getArrayJoinKeys() — needed for the follow-up sidecar-CSV work (PR B, coming separately on top of this branch).

Changes

• functions/metadata/ — now built dist only (pinned to upstream main 224d336), with VENDORED_FROM.json provenance + generated README; src/tests/build-configs removed.
• functions/scripts/sync-metadata.mjs (+ npm run sync:metadata) — clones upstream at a given ref, builds packages/metadata, copies dist + sanitized package.json + LICENSE, writes provenance. Sanitization strips upstream's prepare script (would break npm install of a dist-only file: dep) and devDeps, keeps the csv-parse runtime dep.
• .github/workflows/metadata-drift-check.yml — weekly non-blocking check; opens/updates a tracking issue when upstream main moves past the pinned commit.
• functions/src/metadata-production.ts — adapts to the new generate() signature (3rd arg boolean → ext: 'json'|'csv').
• functions/src/__tests__/metadata-production.test.js — fixture updated to real new output; data-derived levels/min-max double as a silent-drop guard; fixed a pre-existing aliasing bug in the options test.
• functions/package.json — dep stays file:metadata; added explicit typescript devDep (build previously relied on it transitively via the old fork).
• .gitignore — un-ignores the vendored dist.

Testing

• metadata-production (2), metadata-update (10), metadata-process (2) tests all pass.
• functions tsc build clean; fresh npm install clean (no prepare-script breakage).
• Remaining suite failures are env-only (Firestore-emulator-dependent tests with no emulator running), unrelated to this change.

Note: please don't merge yet — a follow-up PR (sidecar CSVs + transaction refactor) builds on this branch and we'd like to land them in a coordinated way.

🤖 Generated with Claude Code