diff --git a/.github/workflows/marketplace-ingest.yml b/.github/workflows/marketplace-ingest.yml new file mode 100644 index 00000000..7ad48966 --- /dev/null +++ b/.github/workflows/marketplace-ingest.yml @@ -0,0 +1,44 @@ +# AI Artefact Marketplace — push your catalog from GitHub Actions (spec 077) +# Drop this into a repo as .github/workflows/marketplace-ingest.yml. +# +# Prereqs (only ONE thing is mandatory): +# 1. Copy docs/marketplace/templates/map-to-envelope.mjs into your repo at ci/map-to-envelope.mjs +# (zero-dependency Node mapper; set the `kind` arg below for your repo type). +# 2. Generate an ingest token in the Marketplace UI (your marketplace detail → Ingest token) and store it +# as the repo secret MARKETPLACE_INGEST_TOKEN. ← the only required setup. +# (optional) Override the target by setting a repo variable MARKETPLACE_INGEST_URL; otherwise it defaults +# to the public ingest sidecar below. +# +# NETWORK: the default URL is the PUBLIC ingest sidecar (infaz-prod-0-0-func-mktingest) — reachable from +# GitHub-hosted runners. (The hub vp.unic.com is IP-allowlisted and would block hosted runners; only point +# MARKETPLACE_INGEST_URL there if you run on a self-hosted Unic-network runner.) + +name: Marketplace ingest +on: + push: + branches: [main] + # Tighten to release tags if you only want to publish on release: tags: ['*@*'] + workflow_dispatch: {} + +jobs: + ingest: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: '20' } + + - name: Map repo → artefact envelope + run: node ci/map-to-envelope.mjs unic-agents . > body.json + + - name: Push full set to the marketplace + env: + # Defaults to the public ingest sidecar; override via repo variable if needed. + INGEST_URL: ${{ vars.MARKETPLACE_INGEST_URL || 'https://infaz-prod-0-0-func-mktingest.azurewebsites.net' }} + VP_TOKEN: ${{ secrets.MARKETPLACE_INGEST_TOKEN }} + run: | + echo "Pushing $(node -e 'console.log(require("./body.json").artefacts.length)') artefacts to $INGEST_URL …" + curl -sS -f -X POST "$INGEST_URL/api/marketplace/ingest" \ + -H "X-VP-Token: $VP_TOKEN" \ + -H "Content-Type: application/json" \ + --data-binary @body.json diff --git a/AGENTS.md b/AGENTS.md index 64746117..166640cf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,6 +30,7 @@ docs/ ├── issues/ # Grilled and scoped feature issues ├── process/ # Process and workflow guides └── research/ # Research notes and explorations +ci/ # Vendored marketplace mapper — copied verbatim, never edited ``` ## Navigation @@ -38,6 +39,7 @@ docs/ - Shared release scripts: `packages/release-tools/scripts/` - Architectural decisions: `docs/adr/` - Process templates: `docs/process/` +- Marketplace mapper: `ci/map-to-envelope.mjs` (vendored — see [Marketplace ingest](#marketplace-ingest)) ## Commands @@ -134,6 +136,26 @@ To ship a new plugin version: | Push to `develop` | ✓ | ✓ (changed packages) | — | | Push to `main` | ✓ | ✓ (changed packages) | — | +## Marketplace ingest + +Every push to `main` publishes this repository's six plugin cards to the Unic AI Artefact Marketplace, source `unic-agents-plugins`. `.github/workflows/marketplace-ingest.yml` maps the catalogue with `ci/map-to-envelope.mjs` and posts it to the public ingest sidecar. The mapper kind is `unic-agents`, because this is a Claude-plugin monorepo. The full documentation is [Push your catalog](https://vp.unic.com/docs/#/marketplace/), which needs a VP login. Run it by hand to see the envelope: + +```sh +node ci/map-to-envelope.mjs unic-agents . > body.json +``` + +**Read that output, never POST it.** Off CI the mapper resolves `commit`, `ref` and `author` to empty strings, because it reads them from the CI environment. Posting such an envelope is how this repository's six cards got a version history with a blank author in the first place, and VP cannot repair one afterwards. The local Python tool now refuses a push it cannot attribute ([UNICGRAPH-572](https://uniccom.atlassian.net/browse/UNICGRAPH-572)), but that guard lives in that tool: neither this mapper nor the ingest endpoint will stop you. Only the workflow may push. + +A push sends the **full set**. VP diffs it and soft-removes any artefact the push omits, so never narrow the set to the plugins that changed. + +**Both files are upstream templates, copied verbatim.** Never hand-edit them and never reformat them. To update either one, re-copy it from its VP URL — [`map-to-envelope.mjs`](https://vp.unic.com/docs/marketplace/templates/map-to-envelope.mjs) and [`pipeline-github-action.yml`](https://vp.unic.com/docs/marketplace/templates/pipeline-github-action.yml). Both answer `401` without a signed-in VP session, so a person must fetch them; an agent cannot. The documentation warns twice that a hand-built envelope is how `provenance.author` goes missing, and this repository's six cards already carry an empty author from one such push on 10 June 2026. The same warning covers the artefact `content` block, which does not apply here: `mapUnicAgents` emits no `content` for any plugin, because plugins are `pointer` artefacts obtainable via `/plugin install`. + +`biome.json` excludes `ci/map-to-envelope.mjs` so that rule can hold: Biome reports two errors on the vendored mapper and reformats it. Keep the exclusion, and keep it scoped to that one path. A mapper reformatted to repo style makes the next re-copy read as a diff, and a directory-wide `!ci` would silently un-check any file added to `ci/` later. + +The mapper resolves `provenance.author` to `GITHUB_ACTOR`, the person who triggered the run, not the commit author. That is wrong on a merge-commit repository like this one, and it is filed upstream as [UNICGRAPH-575](https://uniccom.atlassian.net/browse/UNICGRAPH-575). **Do not patch it here.** A local fix would hide the defect instead of surfacing it. + +The ingest token lives only as the repository secret `MARKETPLACE_INGEST_TOKEN` (Settings → Secrets and variables → Actions). It never goes into a file, a commit or an agent conversation. The token also decides which marketplace the push lands in — the request body cannot name another source. + ## Feature-driven development New work enters through the issue tracker as Features. Plan with `/wayfinder` when the work is too big for one agent session, or `/grill-with-docs` when it fits in one; then `/to-spec` → `/to-tickets` → `/archon-rollout`. `/to-tickets` iterates the breakdown with you and applies `ready-for-agent` once you approve it — that in-session approval is the checkpoint, and nothing re-checks it before dispatch. So keep the ready queue short: grill late, dispatch soon, and re-grill a ticket that has sat for more than a few days rather than trusting it. Use `/tdd` and `/implement` for individual issues. See [`docs/process/ai-development.md`](docs/process/ai-development.md) for the mental model. diff --git a/biome.json b/biome.json index 38b5695c..2e61297c 100644 --- a/biome.json +++ b/biome.json @@ -10,7 +10,8 @@ "!**/.claude", "!**/.ralph", "!**/.remember", - "!**/*.min.js" + "!**/*.min.js", + "!ci/map-to-envelope.mjs" ] } } diff --git a/ci/map-to-envelope.mjs b/ci/map-to-envelope.mjs new file mode 100644 index 00000000..b508c7eb --- /dev/null +++ b/ci/map-to-envelope.mjs @@ -0,0 +1,230 @@ +#!/usr/bin/env node +// 077 — repo → artefact-envelope mapper for CI pipelines. Zero dependencies (Node 18+). +// Emits {provenance, artefacts:[...]} JSON to stdout; pipe it to the ingest endpoint. +// +// node map-to-envelope.mjs skillrepo . > body.json +// node map-to-envelope.mjs unic-agents . > body.json +// +// This is the JS port of specs/077-artefact-registry/tools/ingest_local.py — same envelope, so the +// catalog accepts pipeline pushes identically to today's local push. The transport (curl to the +// ingest sidecar / blob+SAS) is the pipeline's job; this only does the mapping. + +import { readFileSync, readdirSync, existsSync } from 'node:fs'; +import { join, relative, basename, dirname } from 'node:path'; + +// ── tiny YAML front-matter parser (handles the SKILL.md shape: quoted multi-line scalars + +// one nested `metadata:` map). Avoids a js-yaml dependency so this drops into any pipeline. ── +function parseFrontMatter(text) { + const m = text.match(/^---\s*\r?\n([\s\S]*?)\r?\n---/); + if (!m) return {}; + const lines = m[1].split(/\r?\n/); + const data = {}; + let parent = null; + for (let i = 0; i < lines.length; i++) { + const raw = lines[i]; + if (!raw.trim()) continue; + const km = raw.match(/^(\s*)([\w-]+):\s?(.*)$/); + if (!km) continue; + const indent = km[1].length, key = km[2]; + let val = km[3]; + if (indent === 0) parent = null; + if (val === '' && key === 'metadata') { data.metadata = {}; parent = 'metadata'; continue; } + if (/^[>|][+-]?$/.test(val)) { // YAML block scalar (folded > / literal |) + const literal = val[0] === '|'; + const block = []; + while (i + 1 < lines.length) { + const next = lines[i + 1]; + if (next.trim() === '') { block.push(''); i++; continue; } + if ((next.length - next.trimStart().length) <= indent) break; // dedent → block ends + block.push(next.trimStart()); i++; + } + while (block.length && block[block.length - 1] === '') block.pop(); + if (literal) { val = block.join('\n'); } + else { + let out = ''; let prevBlank = true; + for (const bl of block) { + if (bl === '') { out += '\n\n'; prevBlank = true; } + else { out += (prevBlank ? '' : ' ') + bl; prevBlank = false; } + } + val = out.trim(); + } + } else if (val.startsWith('"')) { // quoted, maybe multi-line + let content = val; + while (!(content.trimEnd().endsWith('"') && content.length > 1) && i + 1 < lines.length) { + i++; content += ' ' + lines[i].trim(); + } + val = content.trim().replace(/^"/, '').replace(/"$/, '').trim(); + } else { + val = val.trim().replace(/^"|"$/g, ''); + } + if (parent === 'metadata' && indent >= 2) data.metadata[key] = val; + else data[key] = val; + } + return data; +} + +function walk(dir, name, out = []) { + for (const e of readdirSync(dir, { withFileTypes: true })) { + if (e.isDirectory()) { + if (e.name === '.git' || e.name === 'node_modules') continue; // skip noise + walk(join(dir, e.name), name, out); + } else if (e.name === name) out.push(join(dir, e.name)); + } + return out; +} + +// ── content (spec 115) ──────────────────────────────────────────────────────────────────────── +// The pushing pipeline is the ONLY place with guaranteed read access to its own repo, so it ships +// the artefact BODY — not just a link to it. Consumers then need nothing but their VP login. +// Caps mirror the server (which truncates + counts rather than rejecting, but declaring truncation +// here keeps the numbers honest). +const MAX_BODY = 256 * 1024; +const MAX_TOTAL = 1024 * 1024; +const MAX_FILES = 25; + +// Never read these into a payload that becomes company-wide readable — a checked-in secret must not +// be published by accident. The mapper is the right place for this check: it is the only step with +// filesystem context. +const SECRET_RE = /(^|\/)(\.env(\..*)?|id_(rsa|dsa|ecdsa|ed25519)(\.pub)?)$|\.(key|pem|pfx|p12|keystore|jks)$/i; +const TEXT_RE = /\.(md|txt|json|ya?ml|toml|py|sh|bash|ps1|js|mjs|cjs|ts|sql|csv|html|css|xml|ini|cfg|conf)$/i; + +function listFiles(dir, out = [], depth = 0) { + if (depth > 4) return out; + for (const e of readdirSync(dir, { withFileTypes: true })) { + const p = join(dir, e.name); + if (e.isDirectory()) { + if (['.git', 'node_modules', '__pycache__', '.venv'].includes(e.name)) continue; + listFiles(p, out, depth + 1); + } else out.push(p); + } + return out; +} + +// Everything in the artefact's own folder except the primary file: scripts/, references/, templates. +function collectCompanions(primaryPath) { + const dir = dirname(primaryPath); + const files = []; + let total = 0; + let truncated = false; + for (const p of listFiles(dir).sort()) { + if (p === primaryPath) continue; + const rel = relative(dir, p).replace(/\\/g, '/'); + if (SECRET_RE.test(rel)) { console.error(`! skipped (secret-shaped): ${rel}`); truncated = true; continue; } + if (files.length >= MAX_FILES) { truncated = true; break; } + const buf = readFileSync(p); + if (buf.length > MAX_BODY || total + buf.length > MAX_TOTAL) { truncated = true; continue; } + total += buf.length; + const isText = TEXT_RE.test(rel); + files.push({ + path: rel, bytes: buf.length, + encoding: isText ? 'utf-8' : 'base64', + format: isText ? 'text' : 'binary', + body: isText ? buf.toString('utf8') : buf.toString('base64'), + }); + } + return { files, truncated }; +} + +// One SKILL.md (Agent-Skills standard) → envelope. Tolerates minimal front matter (name + description +// only — version/keywords/owner default cleanly when there's no `metadata:` block). +function skillFromFile(repo, path) { + const src = readFileSync(path, 'utf8'); + const fm = parseFrontMatter(src); + const meta = fm.metadata || {}; + const name = fm.name || basename(dirname(path)); + const keywords = String(meta.tags || '').split(',').map((s) => s.trim()).filter(Boolean); + const rel = relative(repo, path).replace(/\\/g, '/'); + const bytes = Buffer.byteLength(src, 'utf8'); + const { files, truncated } = collectCompanions(path); + return { + artefact_id: name, type: 'skill', name, + // An explicit `version` WINS over `last_verified`, and a top-level one over a nested one. + // This read `metadata.last_verified` and nothing else, so an author who wrote + // `version: "1.0.0"` saw the skill published as a date — or, with no `last_verified` at all, + // as "unversioned" while the version sat right there in the file (six skills in UNICIS-skills + // alone). The two are not the same statement: `version` is what the artefact IS, + // `last_verified` is when somebody last checked it. Same precedence as the hand-upload path in + // marketplace_contrib.py — two doors into one registry must not disagree about what a file + // says (UNICGRAPH-438). + version: String(fm.version || meta.version || meta.last_verified || 'unversioned'), + description: fm.description || '', + keywords, owner: meta.owner || '', homepage: '', + entrypoints: [{ kind: 'skill', ref: rel, label: 'SKILL.md' }], + links: [], spec: fm, + // The skill itself — this is what makes it obtainable without repo access. + content: { + format: 'markdown', path: rel, encoding: 'utf-8', + body: bytes <= MAX_BODY ? src : undefined, + bytes, files, truncated: truncated || bytes > MAX_BODY, + }, + }; +} + +// Agent-Skills repo, skills under `skills/` (e.g. skillrepo-monorepo). +function mapSkillrepo(repo) { + return walk(join(repo, 'skills'), 'SKILL.md').sort().map((p) => skillFromFile(repo, p)); +} + +// Agent-Skills repo, ANY layout — every SKILL.md anywhere (e.g. UNICIS-skills: /SKILL.md at root). +function mapSkills(repo) { + return walk(repo, 'SKILL.md').sort().map((p) => skillFromFile(repo, p)); +} + +function mapUnicAgents(repo) { + const mk = JSON.parse(readFileSync(join(repo, '.claude-plugin', 'marketplace.json'), 'utf8')); + const market = mk.name || 'unic-agent-plugins'; + const out = []; + for (const p of mk.plugins || []) { + const pj = join(repo, p.source || '', '.claude-plugin', 'plugin.json'); + if (!existsSync(pj)) { console.error(`! missing ${pj}`); continue; } + const j = JSON.parse(readFileSync(pj, 'utf8')); + const name = j.name || p.name; + const eps = [{ kind: 'install', ref: `/plugin install ${name}@${market}`, label: 'Install' }]; + for (const c of j.commands || []) eps.push({ kind: 'command', ref: c }); + for (const a of j.agents || []) eps.push({ kind: 'agent', ref: a }); + out.push({ + artefact_id: name, type: 'claude-plugin', name, version: j.version || '0.0.0', + description: j.description || '', keywords: j.keywords || [], + owner: (j.author || {}).name || '', homepage: j.homepage || '', + entrypoints: eps, links: j.homepage ? [{ label: 'Homepage', url: j.homepage }] : [], spec: j, + }); + } + return out; +} + +const MAPPERS = { skills: mapSkills, skillrepo: mapSkillrepo, 'unic-agents': mapUnicAgents }; +const REPO_URLS = { + skillrepo: 'https://dev.azure.com/unicag/CloudProjectHub/_git/skillrepo-monorepo', + 'unic-agents': 'https://github.com/unic/unic-agents-plugins', +}; + +// ADO's Build.Repository.Uri carries the org as basic-auth userinfo +// (https://unicag@dev.azure.com/...). Strip it so the published link is clean & shareable. +function cleanRepoUrl(u) { + return String(u || '').replace(/^(https?:\/\/)[^/@]+@/, '$1'); +} + +const [kind, repo = '.'] = process.argv.slice(2); +if (!MAPPERS[kind]) { console.error(`usage: map-to-envelope.mjs <${Object.keys(MAPPERS).join('|')}> `); process.exit(2); } +const artefacts = MAPPERS[kind](repo); +console.error(`mapped ${artefacts.length} artefacts from ${kind}`); +process.stdout.write(JSON.stringify({ + provenance: { + // Prefer the CI-provided repo URL (works for any repo); fall back to the known map. + repo_url: cleanRepoUrl(process.env.BUILD_REPOSITORY_URI) + || (process.env.GITHUB_REPOSITORY ? `https://github.com/${process.env.GITHUB_REPOSITORY}` : '') + || REPO_URLS[kind] || '', + commit: process.env.GIT_COMMIT || process.env.BUILD_SOURCEVERSION || process.env.GITHUB_SHA || '', + ref: process.env.BUILD_SOURCEBRANCH || process.env.GITHUB_REF || '', + // WHO changed it. Without this the marketplace can show a new version and a commit hash but no + // person, and the only name on the artefact page is its OWNER — which reads as an attribution + // and is not one (UNICGRAPH-438). The commit AUTHOR is preferred over whoever started the + // build: a queued or re-run pipeline says nothing about who wrote the change. VP cannot look + // this up afterwards — commit search is unavailable on the unicag organisation — so the push + // is the only moment the name exists. + author: process.env.BUILD_SOURCEVERSIONAUTHOR || process.env.GITHUB_ACTOR + || process.env.BUILD_REQUESTEDFOR || '', + pipeline: 'ci:map-to-envelope.mjs', + }, + artefacts, +}));