diff --git a/.changeset/remote-schema-sources.md b/.changeset/remote-schema-sources.md new file mode 100644 index 0000000000..fc336e38b6 --- /dev/null +++ b/.changeset/remote-schema-sources.md @@ -0,0 +1,5 @@ +--- +"@fission-ai/openspec": minor +--- + +Add Git-backed project schema sources with explicit `schema sync`, deterministic lockfiles, integrity-verified local caching, and network-free normal resolution. diff --git a/docs/cli.md b/docs/cli.md index 7c5a75291b..ab073f6e56 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -15,7 +15,7 @@ The OpenSpec CLI (`openspec`) provides terminal commands for project setup, vali | **Validation** | `validate` | Check changes and specs for issues | | **Lifecycle** | `archive` | Finalize completed changes | | **Workflow** | `new change`, `status`, `instructions`, `templates`, `schemas` | Artifact-driven workflow support | -| **Schemas** | `schema init`, `schema fork`, `schema validate`, `schema which` | Create and manage custom workflows | +| **Schemas** | `schema init`, `schema fork`, `schema validate`, `schema which`, `schema sync` | Create, inspect, and synchronize custom workflows | | **Config** | `config` | View and modify settings | | **Utility** | `feedback`, `completion` | Feedback and shell integration | @@ -51,6 +51,7 @@ These commands support `--json` output for programmatic use by AI agents and scr | `openspec instructions` | Get next steps | `--json` for agent instructions | | `openspec templates` | Find template paths | `--json` for path resolution | | `openspec schemas` | List available schemas | `--json` for schema discovery; `--store ` to select a registered root | +| `openspec schema sync [name]` | Synchronize declared Git schemas | `--json` for structured sync results | | `openspec store setup ` | Create and register a local store | `--json` with explicit inputs for structured setup output | | `openspec store register ` | Register an existing store | `--json` for structured registration output | | `openspec store unregister ` | Forget a local store registration | `--json` for structured cleanup output | @@ -892,14 +893,21 @@ openspec templates --json ``` Schema: spec-driven +Source: package -Templates: - proposal → ~/.openspec/schemas/spec-driven/templates/proposal.md - specs → ~/.openspec/schemas/spec-driven/templates/specs.md - design → ~/.openspec/schemas/spec-driven/templates/design.md - tasks → ~/.openspec/schemas/spec-driven/templates/tasks.md +proposal: + /path/to/openspec/schemas/spec-driven/templates/proposal.md +specs: + /path/to/openspec/schemas/spec-driven/templates/specs.md +design: + /path/to/openspec/schemas/spec-driven/templates/design.md +tasks: + /path/to/openspec/schemas/spec-driven/templates/tasks.md ``` +The source is one of `project`, `remote`, `user`, or `package`. Synchronized +remote templates report `remote` in both text and JSON output. + --- ### `openspec schemas` @@ -943,6 +951,93 @@ Available schemas: Commands for creating and managing custom workflow schemas. +### `openspec schema sync` + +Synchronize Git schema sources declared in `openspec/config.yaml`. + +```text +openspec schema sync [name] [options] +``` + +| Option | Description | +|--------|-------------| +| `--locked` | Restore or verify the exact commit and digest already in the lockfile | +| `--json` | Emit exactly one JSON result document | + +Omit `name` to synchronize every declared source. Update mode resolves a +configured branch or tag to an immutable commit, validates the complete bundle, +installs it in the local content-addressed cache, and atomically updates +`openspec/schemas.lock.yaml`. + +The command searches upward from the current directory for the nearest +consumer repository containing `openspec/`. Its `config.yaml` and +`schemas.lock.yaml` remain authoritative even when the repository selects a +planning store. Concurrent schema sync processes for the same consumer +repository are serialized so named updates cannot overwrite each other. +Runtime coordination lives beneath a self-ignored +`openspec/.schemas.lock/` directory. OpenSpec publishes participant records +atomically and recovers aged malformed records, so interrupted syncs neither +dirty Git status nor require manual lock cleanup. + +```bash +# Update one source to the current configured ref +openspec schema sync qeda-sdd + +# Update every source +openspec schema sync + +# CI: restore the exact committed lock state +openspec schema sync --locked --json +``` + +Public HTTPS and private SSH declarations: + +```yaml +schemaSources: + public-flow: + git: https://github.com/example/team-schemas.git + ref: v1.2.0 + path: schemas/public-flow + private-flow: + git: git@github.com:acme/private-schemas.git + ref: main + path: schemas/private-flow +``` + +Private access reuses system Git SSH and credential helpers. OpenSpec preserves +an existing `GIT_SSH_COMMAND` while enforcing non-interactive SSH with +`BatchMode=yes`. An explicit `StrictHostKeyChecking` value is preserved; +`accept-new` is added only when the inherited command has no host-key policy. +Never embed a token in a URL. Commit the lockfile, but do not commit the global +cache. Ordinary commands are network-free and continue to use the old locked +version after a remote branch advances. Run update mode explicitly to upgrade. +A missing or corrupt cache reports a `schema sync --locked` fix; CI must either +restore the global cache or run that command while the source is reachable +before entering an offline phase. + +Example JSON success: + +```json +{ + "synced": true, + "locked": false, + "lockfile": "/workspace/openspec/schemas.lock.yaml", + "schemas": [ + { + "name": "qeda-sdd", + "git": "https://github.com/example/team-schemas.git", + "requestedRef": "v1.2.0", + "resolvedCommit": "0123456789abcdef0123456789abcdef01234567", + "bundlePath": "schemas/qeda-sdd", + "integrity": "sha256:...", + "cachePath": "...", + "restored": false + } + ], + "status": [] +} +``` + ### `openspec schema init` Create a new project-local schema. @@ -1094,11 +1189,16 @@ spec-driven resolves from: package Source: /usr/local/lib/node_modules/@fission-ai/openspec/schemas/spec-driven ``` -**Schema precedence:** +**Schema authority and precedence:** + +For a name without a remote declaration, precedence remains project, user, then +package. A `schemaSources.` declaration owns that name: a same-named +project schema is a configuration conflict, a valid lock/cache resolves as +remote, and an unavailable remote never falls through to user or package. -1. Project: `openspec/schemas//` -2. User: `~/.local/share/openspec/schemas//` -3. Package: Built-in schemas +Declared remote sources fail closed when unavailable and never trigger an +implicit network request. `schema which --all` reports an unavailable remote as +one structured entry and continues listing healthy schemas. --- diff --git a/docs/customization.md b/docs/customization.md index b1143b9276..df376e1b0a 100644 --- a/docs/customization.md +++ b/docs/customization.md @@ -1,11 +1,12 @@ # Customization -OpenSpec provides three levels of customization: +OpenSpec provides four levels of customization: | Level | What it does | Best for | |-------|--------------|----------| | **Project Config** | Set defaults, inject context/rules | Most teams | | **Custom Schemas** | Define your own workflow artifacts | Teams with unique processes | +| **Remote Schemas** | Pin a team-owned Git schema across repositories | Multi-repository teams | | **Global Overrides** | Share schemas across all projects | Power users | --- @@ -344,6 +345,98 @@ Path: /path/to/project/openspec/schemas/my-workflow > **Note:** OpenSpec also supports user-level schemas at `~/.local/share/openspec/schemas/` for sharing across projects, but project-level schemas in `openspec/schemas/` are recommended since they're version-controlled with your code. +## Remote Team Schemas + +A remote schema is a complete schema bundle maintained in a Git repository and +declared by a project. It solves a different problem from the other locations: + +- A **project-local schema** is copied into one repository and can evolve there. +- A **remote schema** is shared by many repositories and pinned by each consumer. +- A **user-level schema** is a machine-local override and is not reproducible for a team. +- A **package schema** ships with the installed OpenSpec version. + +Declare the source without changing the existing string-valued `schema` field: + +```yaml +# openspec/config.yaml +schema: qeda-sdd + +schemaSources: + qeda-sdd: + git: https://github.com/example/QEDASDD.git + ref: v1.0.0 + path: schemas/qeda-sdd +``` + +Then synchronize explicitly: + +```bash +# Resolve the configured ref and update the lock +openspec schema sync qeda-sdd + +# Synchronize every declared source +openspec schema sync + +# Restore/verify the exact lockfile state, for example in CI +openspec schema sync --locked +``` + +Commit `openspec/schemas.lock.yaml`. It records the requested ref, resolved +commit SHA, bundle path, and SHA-256 content integrity. Do not commit the +machine cache under the OpenSpec global data directory. Normal OpenSpec +commands never fetch: they only use a matching lock entry and verified cache. +If the cache is absent, run `schema sync --locked` while the Git source is +reachable, then ordinary commands work offline. + +The consumer repository owns both `openspec/config.yaml` and +`openspec/schemas.lock.yaml`. Running sync from a nested directory searches +upward for that repository. A configured planning store does not own or redirect +remote schema sources. Sync processes for one consumer repository are +serialized, so concurrent named updates cannot lose lockfile entries. +The `openspec/.schemas.lock/` coordination directory ignores its own runtime +files. Participant records are published atomically, and aged malformed +records left by an interrupted process are reclaimed without manual cleanup. + +Branches and tags are allowed, but they move only when `schema sync` is run +without `--locked`. A remote update therefore cannot change a normal command's +workflow unexpectedly. + +Private repositories use the system Git client's existing SSH agent or +credential helper: + +```yaml +schemaSources: + private-flow: + git: git@github.com:acme/private-schemas.git + ref: main + path: schemas/private-flow +``` + +Do not put credentials or tokens in configuration. Credential-bearing HTTPS +URLs are rejected, and lockfiles contain no authentication material. OpenSpec +preserves existing `GIT_SSH_COMMAND` options while enforcing `BatchMode=yes`. +An explicit `StrictHostKeyChecking` value is preserved; OpenSpec adds +`StrictHostKeyChecking=accept-new` only when no host-key policy is present. + +Schema authority is name-based: + +- Without a remote declaration, precedence remains project-local, user-level, + then package built-in. +- Once `schemaSources.` is declared, the remote owns that name. +- A same-named project-local bundle is a configuration conflict; OpenSpec does + not silently choose or shadow either bundle. + +A declared remote source fails closed when its lock or cache is missing, +stale, or corrupt; OpenSpec does not silently select a same-named user or +package schema. `schema which --all` reports each unavailable remote separately +while continuing to list healthy schemas. Bundle paths must stay inside the Git +repository. Absolute paths, traversal, symlinks, submodules, case-colliding +paths, invalid names, incomplete schemas, and bundles over 1,000 files or +10 MiB are rejected. These portable fail-closed checks apply to remote bundles; +existing project-local schema validation retains its legacy path and symlink +behavior. Remote schemas are complete bundles; inheritance and schema merging +are not supported. + --- ## Examples @@ -414,7 +507,7 @@ Then edit `schema.yaml` to add: OpenSpec also supports community-maintained schemas distributed via standalone repositories. These provide opinionated workflows that integrate OpenSpec with other tools or systems, similar to how [github/spec-kit's community extension catalog](https://github.com/github/spec-kit/tree/main/extensions) works for spec-kit. -Community schemas are not vendored into OpenSpec core — they live in their own repositories with their own release cadence. To use one, copy the schema bundle into your project's `openspec/schemas//` directory (each repo's README has install instructions). +Community schemas are not vendored into OpenSpec core — they live in their own repositories with their own release cadence. You can either declare one as a remote schema and pin it with `openspec schema sync`, or copy its bundle into your project's `openspec/schemas//` directory. | Schema | Maintainer | Repository | Description | |--------|-----------|-----------|-------------| diff --git a/openspec/changes/add-remote-schema-sources/.openspec.yaml b/openspec/changes/add-remote-schema-sources/.openspec.yaml new file mode 100644 index 0000000000..2bc06e0e51 --- /dev/null +++ b/openspec/changes/add-remote-schema-sources/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-26 diff --git a/openspec/changes/add-remote-schema-sources/design.md b/openspec/changes/add-remote-schema-sources/design.md new file mode 100644 index 0000000000..ad4f72ee45 --- /dev/null +++ b/openspec/changes/add-remote-schema-sources/design.md @@ -0,0 +1,212 @@ +## Context + +OpenSpec currently resolves complete schema bundles from three directory tiers: project-local, user-level, and package built-in. Project configuration selects a schema by name, but it cannot describe where a team-maintained schema comes from. Teams therefore copy bundles across repositories or use external automation, neither of which records the exact schema revision consumed by normal commands. + +The feature spans resilient project config parsing, schema resolution, CLI schema management, Git transport, local data storage, integrity validation, cross-platform path handling, and documentation. Normal commands must remain synchronous and network-free; only the explicit sync command may invoke Git. + +## Goals / Non-Goals + +**Goals:** + +- Make a named Git repository path a reproducible schema source for a project. +- Resolve mutable refs to immutable commits only during explicit synchronization. +- Commit stable lock metadata while keeping downloaded content machine-local. +- Preserve existing schema selection and all non-remote resolver APIs. +- Fail closed when a declared remote source is not reproducible locally. +- Use system Git for HTTPS, SSH, credential helpers, SSH agents, and local Git repositories. +- Validate and install bundles atomically with portable security boundaries. +- Provide human and one-document JSON output suitable for CI and agents. + +**Non-Goals:** + +- Hosted registries, central services, or OpenSpec-managed credentials. +- Implicit clone, fetch, pull, or update during ordinary commands. +- Schema inheritance, merging, partial overrides, or dependency graphs between schemas. +- Automatic cache garbage collection in the MVP. +- Committing downloaded cache contents to a project. + +## Decisions + +### 1. Keep schema selection separate from source declaration + +Project configuration retains the existing scalar selection and adds a provisional source map: + +```yaml +schema: qeda-sdd + +schemaSources: + qeda-sdd: + git: https://github.com/example/QEDASDD.git + ref: v1.0.0 + path: schemas/qeda-sdd +``` + +`schemaSources` is parsed field-by-field into `Record`. Names use the existing kebab-case schema grammar. Each entry requires non-empty `git`, `ref`, and `path`. Valid Git forms include HTTPS, `ssh://`, scp-style SSH, and `file://` for local repositories. HTTPS user information is rejected so a password or token cannot be persisted. SSH usernames such as `git@github.com` remain valid. + +Alternatives considered: + +- Replace `schema: ` with an object: rejected because it breaks existing configuration and couples selection to transport. +- Put sources in a separate manifest: rejected because it adds another hand-edited project file before locking provides value. + +### 2. Commit a project lock and keep a global content-addressed cache + +The consumer repository owns both the source declarations and the lock. The lock path is tracked by one constant and is always `openspec/schemas.lock.yaml` relative to the nearest consumer repository root: + +```yaml +version: 1 +schemas: + qeda-sdd: + git: https://github.com/example/QEDASDD.git + requestedRef: v1.0.0 + resolvedCommit: 0123456789abcdef0123456789abcdef01234567 + bundlePath: schemas/qeda-sdd + integrity: sha256:0123456789abcdef... +``` + +The cache lives under `getGlobalDataDir()/schema-cache/v1/sha256/`. The lock never stores a machine-specific absolute cache path. Identical bundle content is reused across projects and commits, and a failed upgrade cannot overwrite the old content-addressed directory. + +Schema source ownership does not follow a selected planning store. Commands started in a nested directory search upward for the nearest repository containing `openspec/` and use that consumer root for configuration, local schemas, and the lockfile. A `store:` pointer or global default store may redirect planning artifacts, but it does not redirect remote schema sources in V1. + +The digest is computed over every regular file beneath the bundle using sorted Git-style relative paths, explicit byte lengths, and file bytes. Paths are normalized to `/` only for the canonical digest and lock contract; filesystem access uses `path.join`/`path.resolve`. + +Alternatives considered: + +- Project-local ignored cache: easier to discover but dirties project trees and duplicates content across repositories. +- Commit the downloaded bundle: works offline from a clone but recreates vendoring and schema-copy drift. +- Cache by source/ref: mutable refs could overwrite prior state and make rollback unsafe. + +### 3. Use two explicit synchronization modes + +`openspec schema sync [name]` is update mode. It fetches each configured ref, resolves the fetched object to a commit, validates the selected tree, installs content, and updates the lock. Omitting the name operates on every valid declaration. + +`openspec schema sync [name] --locked` is restore mode. It requires matching config and lock metadata, fetches the exact `resolvedCommit`, reconstructs and verifies the expected bundle, and leaves the lockfile byte-identical. This separates intentional upgrades from deterministic CI cache restoration. + +For multiple selected sources, all fetch/extract/validation operations finish before one atomic lock replacement. A successful cache directory that becomes unreferenced because a later source fails is harmless content-addressed data; no prior lock or cache is removed. + +All update and locked synchronization runs under one project-scoped interprocess lock covering lockfile read, Git/cache work, merge, and lockfile write. Beneath the consumer repository's `openspec/.schemas.lock` coordination directory, each contender creates unique choosing and immutable ticket files containing a random token, PID, hostname, acquisition time, and bakery number. The directory contains a self-ignore rule so coordination, staging, and stale runtime files never appear as Git candidates in consumer repositories. + +Participant state is written to a uniquely named staging file and atomically published through an exclusive filesystem link. Ticket validation requires a positive bakery number; choosing records intentionally omit it. The filesystem bakery ordering prevents a stale reclaimer from moving a shared lock path and temporarily admitting two owners. Acquisition retries for a bounded period, immediately reclaims the exact unique files of a same-host process that is no longer alive, and reclaims unparseable choosing or ticket files only after their filesystem age exceeds the acquisition timeout. Release removes only the current token's ticket and leaves the self-ignoring coordination directory available for later runs. This deliberately serializes network work in V1 so two named sync processes cannot both merge against stale lock state. + +### 4. Fetch with system Git and extract tracked objects, not a working-tree copy + +A focused Git adapter uses `execFile` argument arrays and a temporary repository: + +1. `git init` +2. `git remote add origin ` +3. update mode: `git fetch --depth=1 --no-tags origin ` +4. locked mode: fetch the advertised requested ref with history, then verify the locked commit exists and is its ancestor +5. `git rev-parse ^{commit}` +6. `git merge-base --is-ancestor ` in locked mode +7. `git ls-tree -r -z -- ` +8. `git cat-file --batch` for accepted regular files + +Reading Git objects rather than recursively copying a checkout prevents `.git`, untracked content, and followed filesystem symlinks from entering the bundle. Tree modes identify and reject symlinks and submodules before content extraction. Every process has bounded output and duration. + +Git runs with `GIT_TERMINAL_PROMPT=0`. SSH transports preserve the user's existing `GIT_SSH_COMMAND` command, identity, proxy, and other arguments while normalizing and enforcing `BatchMode=yes`. An explicit user `StrictHostKeyChecking` policy is preserved, including the stricter `yes` policy; `StrictHostKeyChecking=accept-new` is appended only when no policy exists. Authentication and passphrase questions therefore cannot block automation, while OpenSpec does not weaken an explicitly selected host-key policy. The constructed SSH command is never included in diagnostics. + +Git stderr is treated as untrusted and is not copied verbatim into user diagnostics. Errors use stable codes and sanitized source labels. This prevents a transport, credential helper, or malicious remote from reflecting secrets while keeping the actionable fixes (`check Git credentials`, `schema sync`, or `schema sync --locked`). + +Alternative considered: shallow clone plus filesystem traversal. Rejected because checkout behavior, symlink targets, submodules, and untracked administrative content make the trust boundary harder to prove cross-platform. + +### 5. Apply a portable bundle boundary before schema validation + +The source `path` is parsed as a Git tree path, not an operating-system path. It must be relative and contain only safe non-empty segments; POSIX absolute paths, Windows drive/UNC paths, `.`, `..`, NUL, backslash separators, and Windows-unsafe segments are rejected. + +Every `ls-tree` entry must: + +- stay beneath the selected prefix after POSIX normalization; +- be a regular blob, never a symlink or submodule; +- map to a unique case-folded portable relative path; +- keep the bundle at or below 1,000 files and 10 MiB total bytes. + +The extracted bundle must contain `schema.yaml`, a real `templates` directory, and every parser-declared template inside that directory. The parsed schema name must equal the source-map key. + +Schema parsing and issue collection are shared, but validation has two explicit entry points. Legacy local validation preserves the pre-feature template lookup and symlink/path behavior. Remote-bundle validation adds the portable path, real-file, real-directory, containment, and declared-name rules. Remote validation returns every structured issue so sync and JSON diagnostics do not hide later failures behind the first error. + +Local/cache integrity traversal checks each regular file's stat size against the remaining byte budget before reading it, then rechecks the actual buffer length to remain correct if the file changes between stat and read. + +### 6. Activate cache and lock atomically + +Extraction occurs in a `mkdtemp` directory under the cache parent, followed by structural validation and digest calculation. If the final content-addressed directory is absent, it is installed with `rename`; if present, it is re-verified and reused. Temporary directories are removed on every exit path. + +The lock is serialized deterministically with source names sorted. It is written to a uniquely named temporary sibling, flushed by close, then renamed over the known lock filename. Sync never deletes the previous digest directory. A lock write failure therefore leaves ordinary resolution on the previous lock. Atomic replacement prevents partial files; the project-scoped sync lock separately prevents lost updates between processes. + +### 7. Remote declarations fail closed in the resolver + +With `projectRoot`, an undeclared name retains the existing project-local → user → package precedence. Once `schemaSources.` is declared, that declaration owns the name: + +1. a same-named project-local bundle is a `schema_name_conflict`; +2. a matching lock and verified cache resolve as `remote`; +3. a missing/stale lock, absent cache, malformed metadata, or integrity mismatch is an unavailable remote error; and +4. resolution never falls through to a same-named user or package copy. + +Without `projectRoot`, existing behavior stays exactly user then package and no config, lock, or remote cache is read. + +`SchemaInfo.source` and schema-command resolution types add `remote`. A remote location may include requested ref, commit, bundle path, and integrity. Diagnostic listing can describe an unsynchronized declaration, while APIs that must return a loadable directory fail with an actionable error. + +All-schema inspection uses a discriminated available/unavailable result per name. An unusable remote contributes its own structured status and does not abort inspection of healthy project, remote, user, or package schemas. Single-schema inspection uses the same result but exits non-zero when the requested entry is unavailable. + +### 8. Keep JSON output singular and stable + +Successful sync JSON has this top-level shape: + +```json +{ + "mode": "update", + "lockfile": "/project/openspec/schemas.lock.yaml", + "schemas": [ + { + "name": "qeda-sdd", + "git": "https://github.com/example/QEDASDD.git", + "requestedRef": "v1.0.0", + "resolvedCommit": "0123456789abcdef0123456789abcdef01234567", + "bundlePath": "schemas/qeda-sdd", + "integrity": "sha256:...", + "cachePath": "/user-data/openspec/schema-cache/v1/sha256/..." + } + ], + "status": [] +} +``` + +Failures emit the same null-safe fields with one or more structured statuses and exit one. JSON mode creates no spinner and writes no non-JSON stdout. Human mode reports the schema, requested ref to commit transition, cache action, and lockfile path. + +### 9. Implementation units + +- `src/core/remote-schema/types.ts`: public config/lock/result types and constants. +- `src/core/remote-schema/config.ts`: source grammar, URL credential checks, and portable Git path validation. +- `src/core/remote-schema/lockfile.ts`: strict lock parsing and atomic deterministic writes. +- `src/core/remote-schema/bundle.ts`: Git-tree entry validation, extraction, size limits, canonical digest, and cache verification. +- `src/core/remote-schema/git.ts`: bounded, credential-safe system Git calls. +- `src/core/remote-schema/sync.ts`: multi-source transaction orchestration. +- `src/core/remote-schema/sync-lock.ts`: project-scoped cross-process synchronization lock. +- `src/core/artifact-graph/schema-directory.ts`: separate legacy-local and strict-remote parser/template validation entry points. +- `src/core/artifact-graph/resolver.ts`: remote tier integration and metadata. +- `src/commands/schema.ts`: `sync`, enhanced `which`, validation reuse, and human/JSON rendering. + +Focused modules keep network-capable code out of ordinary resolution and make the boundary independently testable. + +## Risks / Trade-offs + +- **Integrity verification reads every cached file during resolution** → Bundles are capped at 10 MiB/1,000 files; correctness is preferred over an unverified stamp in the MVP. +- **Locked restoration needs history reachable from the advertised ref** → Fetch the requested ref without shallow depth in locked mode, verify the locked commit is its ancestor, and fail without advancing when the commit is no longer reachable. +- **Global cache can accumulate unused digests** → Document manual removal; automatic garbage collection is deferred until lock discovery semantics exist. +- **Case-fold collision checks are stricter than a Linux checkout** → This intentionally guarantees that one committed lock is usable on Windows and case-insensitive macOS filesystems. +- **Remote source syntax and lock shape may evolve after maintainer feedback** → Mark the command group experimental, version the lock, and keep parsing strict so migrations can be explicit. +- **A crashed sync can leave interprocess coordination state** → Publish participant records atomically, require ticket numbers, self-ignore the runtime directory, reclaim dead same-host owners immediately, and reclaim malformed records only after the acquisition timeout. +- **`StrictHostKeyChecking=accept-new` trusts a host on first contact** → Use it only as the default when the user supplied no policy; preserve explicit stricter policies unchanged. + +## Migration Plan + +1. Existing projects continue unchanged because `schemaSources` and the lockfile are optional. +2. A team adds a source declaration and runs `openspec schema sync `. +3. The team reviews and commits `openspec/config.yaml` plus `openspec/schemas.lock.yaml`. +4. Developers run update sync intentionally or locked sync to restore missing cache. +5. A same-named project-local bundle must be renamed or the remote declaration removed; OpenSpec does not choose one silently. +6. Removing a source declaration makes its stale lock/cache inert; no automatic deletion occurs. + +Rollback is removal of the declaration and lock entry (or reverting the feature commit). Once the declaration is removed, existing project/user/package schemas resume their prior precedence. + +## Open Questions + +None block the MVP. Maintainer review may rename provisional configuration keys, lockfile fields, or cache paths before upstream merge; behavior is specified independently so those representation changes remain localized. diff --git a/openspec/changes/add-remote-schema-sources/proposal.md b/openspec/changes/add-remote-schema-sources/proposal.md new file mode 100644 index 0000000000..f83bc1a771 --- /dev/null +++ b/openspec/changes/add-remote-schema-sources/proposal.md @@ -0,0 +1,38 @@ +## Why + +Teams that share custom OpenSpec workflows across multiple repositories currently have to copy schema bundles or maintain external automation. Those copies drift, and ordinary commands cannot prove which immutable schema revision they used in CI or offline environments. + +## What Changes + +- Allow projects to declare named Git-backed schema sources alongside the existing `schema: ` setting. +- Add explicit schema synchronization that resolves a requested Git ref to a commit, validates the selected bundle, installs it in a local cache, and writes a commit-friendly lockfile. +- Keep ordinary OpenSpec commands network-free by resolving remote schemas only from the project lockfile and verified local cache. +- Add deterministic locked-cache restoration for CI without advancing the requested ref. +- Resolve remote schema configuration and locks from the consumer repository root even when commands run from nested directories or the repository selects a planning store. +- Keep generated schema configuration at the consumer root while planning artifacts remain in a selected store. +- Extend schema discovery, template reporting, and diagnostics to distinguish project-local, remote, user-level, and package schemas without one unavailable remote aborting an all-schema inspection. +- Treat a declared remote source as the authority for its schema name and report a same-named project-local bundle as a configuration conflict. +- Serialize synchronization per consumer project so concurrent named syncs cannot lose lockfile updates, and recover safely from abandoned or malformed coordination files. +- Reject unsafe or incomplete bundles, credential-bearing declarations, path escapes, symlinks, oversized content, and lock/cache integrity mismatches. +- Preserve existing local-schema validation behavior while applying portable fail-closed checks only to remote bundles. +- Force Git and SSH transport to fail non-interactively while preserving existing SSH command configuration and any explicit host-key policy. + +## Capabilities + +### New Capabilities + +- `remote-schema-sources`: Declaring, synchronizing, locking, caching, validating, and safely restoring Git-backed schema bundles. + +### Modified Capabilities + +- `config-loading`: Parse and validate remote schema source declarations without invalid fields breaking unrelated project configuration. +- `schema-resolution`: Resolve locked remote schemas between project-local and user-level schemas, with deterministic offline errors and source reporting. +- `schema-which-command`: Report remote schema resolution and shadowing details without accessing the network. + +## Impact + +- Project contract: `openspec/config.yaml` gains provisional Git schema source declarations and projects may commit `openspec/schemas.lock.yaml`. +- CLI: `openspec schema sync [name]` gains human and JSON output plus a locked restoration mode. +- Runtime: schema resolution reads a content-addressed cache under OpenSpec's global data directory. +- Security: system Git handles transport and credentials; OpenSpec stores no tokens and validates bundle boundaries before activation. +- Documentation and release tracking: customization/CLI guides and a minor changeset are updated. diff --git a/openspec/changes/add-remote-schema-sources/specs/config-loading/spec.md b/openspec/changes/add-remote-schema-sources/specs/config-loading/spec.md new file mode 100644 index 0000000000..cbb78e911f --- /dev/null +++ b/openspec/changes/add-remote-schema-sources/specs/config-loading/spec.md @@ -0,0 +1,29 @@ +## ADDED Requirements + +### Requirement: Parse remote schema source declarations independently + +The system SHALL parse `schemaSources` as a map from valid schema names to Git source declarations while preserving valid unrelated project configuration when individual declarations are invalid. + +#### Scenario: Valid source map +- **WHEN** config contains `schemaSources.qeda-sdd` with non-empty `git`, `ref`, and `path` strings +- **THEN** the returned project configuration SHALL include the normalized `qeda-sdd` source declaration + +#### Scenario: Invalid schema source name +- **WHEN** a `schemaSources` key is not a valid kebab-case schema name +- **THEN** that source SHALL be omitted with a warning +- **AND** valid schema, context, rules, references, store, and other valid source fields SHALL remain available + +#### Scenario: Invalid source member +- **WHEN** one source has a missing or non-string `git`, `ref`, or `path` member +- **THEN** that source SHALL be omitted with a warning naming the source and invalid member +- **AND** other valid sources SHALL remain available + +#### Scenario: Credential-bearing HTTPS URL +- **WHEN** a source contains an HTTPS URL with user information +- **THEN** that source SHALL be omitted with a credential-safe warning +- **AND** the warning SHALL not repeat the credential-bearing URL + +#### Scenario: Source name prototype key +- **WHEN** `schemaSources` contains a key capable of mutating object prototypes +- **THEN** the key SHALL be rejected through explicit key comparison +- **AND** parsing SHALL not change any object prototype diff --git a/openspec/changes/add-remote-schema-sources/specs/remote-schema-sources/spec.md b/openspec/changes/add-remote-schema-sources/specs/remote-schema-sources/spec.md new file mode 100644 index 0000000000..50e5c51c51 --- /dev/null +++ b/openspec/changes/add-remote-schema-sources/specs/remote-schema-sources/spec.md @@ -0,0 +1,257 @@ +## ADDED Requirements + +### Requirement: Projects can declare Git-backed schema sources + +The system SHALL allow a project to associate a valid schema name with a Git repository, requested ref, and repository-relative bundle path while retaining the existing `schema: ` configuration. + +#### Scenario: HTTPS source declaration +- **WHEN** `openspec/config.yaml` declares a schema source with an HTTPS Git URL, non-empty ref, and repository-relative bundle path +- **THEN** the source SHALL be available to explicit schema synchronization + +#### Scenario: SSH source declaration +- **WHEN** `openspec/config.yaml` declares a schema source with an `ssh://` or scp-style Git URL +- **THEN** synchronization SHALL invoke the system Git client so its SSH configuration and credential mechanisms remain available + +#### Scenario: Existing SSH command configuration +- **WHEN** synchronization inherits a `GIT_SSH_COMMAND` containing identity, proxy, or other user options +- **THEN** the system SHALL preserve those options +- **AND** it SHALL enforce `BatchMode=yes` +- **AND** it SHALL preserve an explicit user `StrictHostKeyChecking` policy +- **AND** it SHALL add `StrictHostKeyChecking=accept-new` only when the inherited command has no host-key policy +- **AND** authentication, passphrase, and host-key questions SHALL NOT block the command + +#### Scenario: Credential-bearing HTTPS source +- **WHEN** an HTTPS source URL contains username, password, or token user information +- **THEN** the system SHALL reject the declaration +- **AND** neither project configuration generated by OpenSpec nor the lockfile SHALL contain that credential + +### Requirement: Schema synchronization produces an immutable lock + +The CLI SHALL provide `openspec schema sync [name]` to resolve configured Git refs, validate schema bundles, populate the local cache, and atomically write `openspec/schemas.lock.yaml`. + +#### Scenario: Synchronize from a nested directory +- **WHEN** a user runs schema synchronization beneath a consumer repository root +- **THEN** the system SHALL search ancestor directories for the nearest repository containing `openspec/` +- **AND** it SHALL read source declarations and write the lock beneath that consumer root +- **AND** it SHALL NOT create a lockfile relative to the raw current directory + +#### Scenario: Consumer repository selects a planning store +- **WHEN** the consumer repository configuration contains a `store:` pointer +- **THEN** remote schema declarations and `schemas.lock.yaml` SHALL remain owned by the consumer repository +- **AND** synchronization SHALL NOT read or write remote schema state in the selected planning store + +#### Scenario: Workflow planning uses a selected store +- **WHEN** a workflow command writes planning artifacts to a selected store +- **THEN** schema selection, metadata validation, template loading, status, instructions, apply guidance, validation, archive checks, and discovery SHALL resolve schemas from the consumer repository +- **AND** the planning store SHALL NOT become the schema authority + +#### Scenario: Store-backed change creation initializes schema configuration +- **WHEN** change creation writes planning artifacts beneath a selected planning root that differs from the consumer schema root +- **THEN** any generated `openspec/config.yaml` SHALL be written beneath the consumer schema root +- **AND** change directories, specs, and archive directories SHALL remain beneath the planning root +- **AND** change creation SHALL NOT generate schema configuration beneath the planning root + +#### Scenario: Synchronize one source +- **WHEN** a user runs `openspec schema sync qeda-sdd` +- **THEN** the system SHALL synchronize only the declared `qeda-sdd` source +- **AND** its lock entry SHALL record the schema name, Git source, requested ref, resolved 40-character commit SHA, bundle path, and SHA-256 integrity digest + +#### Scenario: Synchronize every source +- **WHEN** a user runs `openspec schema sync` without a name +- **THEN** the system SHALL synchronize every valid declared schema source +- **AND** it SHALL replace the lockfile only after every selected source succeeds + +#### Scenario: Branch and tag resolution +- **WHEN** a configured ref names a branch or tag +- **THEN** synchronization SHALL resolve it to the commit containing the selected schema bundle +- **AND** the immutable commit SHA SHALL be recorded in the lockfile + +#### Scenario: Explicit upgrade after branch moves +- **WHEN** a branch-backed source has already been synchronized and the remote branch later advances +- **AND** a user runs `openspec schema sync` for that source again +- **THEN** the lockfile and active cache SHALL advance to the newly resolved commit after validation succeeds + +#### Scenario: Concurrent named synchronization +- **WHEN** two processes synchronize different declared schema names in the same consumer repository +- **THEN** synchronization SHALL serialize the complete lock read, fetch/cache operation, merge, and lock write per project +- **AND** the final lockfile SHALL contain both successful updates + +#### Scenario: Live synchronization lock +- **WHEN** a second process cannot acquire the project synchronization lock within the bounded wait period +- **THEN** it SHALL fail with a `schema_sync_locked` status +- **AND** it SHALL NOT modify the lockfile + +#### Scenario: Abandoned same-host synchronization lock +- **WHEN** a unique lock claim or ticket names a same-host process that is no longer alive +- **THEN** a later synchronization SHALL remove only that abandoned participant's files +- **AND** concurrent reclaimers SHALL preserve mutual exclusion +- **AND** releasing an older owner SHALL NOT delete a successor's ticket + +#### Scenario: Partially written or malformed synchronization participant +- **WHEN** a choosing or ticket participant file is unparseable or a ticket omits its positive bakery number +- **THEN** the malformed participant SHALL NOT be admitted to bakery ordering +- **AND** it SHALL be reclaimed only after its filesystem age exceeds the bounded acquisition timeout +- **AND** subsequent synchronization SHALL recover without manual deletion + +#### Scenario: Coordination state stays out of Git +- **WHEN** synchronization creates choosing, ticket, or staging files beneath `openspec/.schemas.lock` +- **THEN** the coordination directory SHALL ignore all of its runtime contents +- **AND** ordinary Git status and staging SHALL NOT select those files + +### Requirement: Locked synchronization restores without upgrading + +The CLI SHALL support `openspec schema sync [name] --locked` to restore or verify cache entries from the existing lock without changing locked revisions. + +#### Scenario: Restore missing cache in CI +- **WHEN** a valid lock entry exists but its local cache is missing +- **AND** a user runs `openspec schema sync --locked` +- **THEN** the system SHALL fetch and install the exact locked commit +- **AND** the lockfile bytes SHALL remain unchanged + +#### Scenario: Locked source metadata drift +- **WHEN** project configuration no longer matches the source URL, requested ref, or bundle path in the lock entry +- **AND** a user runs `openspec schema sync --locked` +- **THEN** the command SHALL fail without changing the lockfile or active cache +- **AND** the error SHALL instruct the user to run an updating synchronization after reviewing the configuration change + +### Requirement: Normal schema use is network-free and deterministic + +Ordinary OpenSpec commands SHALL resolve a declared remote schema only from matching lock data and verified local cache content. + +#### Scenario: Offline resolution +- **WHEN** a declared schema has a matching lock entry and verified cache +- **AND** the network and source repository are unavailable +- **THEN** ordinary OpenSpec commands SHALL load the cached schema successfully + +#### Scenario: Remote branch changes without synchronization +- **WHEN** the remote branch advances after a successful synchronization +- **AND** no later schema synchronization occurs +- **THEN** ordinary OpenSpec commands SHALL continue using the previously locked commit + +#### Scenario: Missing lock entry +- **WHEN** project configuration declares a remote schema but no matching lock entry exists +- **THEN** schema resolution SHALL fail without falling back to a same-named user or package schema +- **AND** the error SHALL instruct the user to run `openspec schema sync ` + +#### Scenario: Missing cache entry +- **WHEN** a matching lock entry exists but its content-addressed cache entry is absent +- **THEN** schema resolution SHALL fail without accessing the network +- **AND** the error SHALL instruct the user to run `openspec schema sync --locked` + +### Requirement: Cached bundles are integrity checked + +The system SHALL derive a deterministic SHA-256 digest from the selected bundle's sorted relative file paths and bytes and SHALL verify cached content against the lock before use. + +#### Scenario: Cache content matches lock +- **WHEN** every cached bundle file matches the locked integrity digest +- **THEN** the schema SHALL be eligible for normal resolution + +#### Scenario: Cached file is modified +- **WHEN** a cached bundle file is added, removed, or modified after synchronization +- **THEN** normal resolution SHALL reject the bundle +- **AND** the error SHALL identify an integrity mismatch and provide the locked synchronization fix + +#### Scenario: Lock metadata is incomplete +- **WHEN** a lock entry omits required metadata or contains a malformed commit or integrity value +- **THEN** normal resolution SHALL reject the entry without using any corresponding cache directory + +### Requirement: Remote bundles remain inside a safe boundary + +The system SHALL accept only repository-relative source paths and regular tracked files that can be materialized safely and consistently on macOS, Linux, and Windows. + +#### Scenario: Parent path traversal +- **WHEN** a declared bundle path contains a `..` segment +- **THEN** synchronization SHALL reject it before activating downloaded content + +#### Scenario: Absolute or drive-qualified path +- **WHEN** a declared bundle path is POSIX-absolute, Windows-absolute, UNC, or drive-qualified +- **THEN** synchronization SHALL reject it + +#### Scenario: Symlink in selected bundle +- **WHEN** the selected Git tree contains a symbolic link anywhere under the bundle path +- **THEN** synchronization SHALL reject the bundle +- **AND** no linked target SHALL be read or copied + +#### Scenario: Git submodule in selected bundle +- **WHEN** the selected Git tree contains a submodule anywhere under the bundle path +- **THEN** synchronization SHALL reject the bundle without fetching the submodule + +#### Scenario: Cross-platform path collision +- **WHEN** two tracked bundle entries would resolve to the same path on a case-insensitive filesystem or use a path segment unsafe on Windows +- **THEN** synchronization SHALL reject the bundle on every platform + +### Requirement: Remote bundles are complete and bounded + +Before activation, the system SHALL validate the selected bundle with the existing schema parser plus remote-bundle boundary checks and SHALL enforce a maximum of 1,000 regular files and 10 MiB total content. + +#### Scenario: Complete schema bundle +- **WHEN** the selected path contains a valid `schema.yaml`, a `templates` directory, and every referenced template inside that directory +- **AND** the schema name matches the declared source name +- **THEN** the bundle SHALL pass structural validation + +#### Scenario: Missing schema or templates +- **WHEN** `schema.yaml`, the `templates` directory, or a referenced template is missing +- **THEN** synchronization SHALL fail before changing the active cache or lockfile + +#### Scenario: Oversized bundle +- **WHEN** the selected bundle exceeds either the file-count or total-byte limit +- **THEN** synchronization SHALL fail with the applicable limit +- **AND** downloaded content SHALL not become active + +#### Scenario: Oversized individual file +- **WHEN** a local extraction or cache entry contains one regular file larger than the remaining byte budget +- **THEN** integrity validation SHALL reject it using file metadata before reading the complete file +- **AND** it SHALL recheck the actual bytes read before accepting the bundle + +#### Scenario: Multiple remote validation failures +- **WHEN** a remote bundle violates more than one structural or boundary rule +- **THEN** validation SHALL preserve every discovered issue with its path +- **AND** human and JSON error surfaces SHALL NOT report only the first issue + +### Requirement: Remote strict validation does not change legacy local schemas + +The system SHALL apply portable path, real-file, real-directory, containment, and declared-name rules only to remote bundles while preserving pre-feature local schema validation behavior. + +#### Scenario: Legacy local template layout +- **WHEN** a project-local schema uses a template layout accepted before remote schema support +- **THEN** `schema validate` SHALL continue to apply the legacy local lookup behavior +- **AND** remote-only portable bundle checks SHALL NOT reject it + +#### Scenario: Same layout arrives from a remote source +- **WHEN** an equivalent layout violates the remote bundle boundary +- **THEN** synchronization and cache verification SHALL reject it before activation + +### Requirement: Synchronization is failure-atomic + +The system SHALL prepare and validate new cache content in a temporary directory and SHALL preserve the last valid lock and cache whenever synchronization fails. + +#### Scenario: Validation fails during upgrade +- **WHEN** a source already has a valid locked cache +- **AND** a later synchronization fetches an invalid bundle +- **THEN** the command SHALL fail +- **AND** the previous lock entry and cache SHALL remain usable + +#### Scenario: Lockfile replacement fails +- **WHEN** a new bundle is valid but the lockfile cannot be atomically replaced +- **THEN** the prior lockfile SHALL remain readable +- **AND** ordinary resolution SHALL continue selecting only content named by that prior lock + +### Requirement: Synchronization output supports humans and automation + +The schema synchronization command SHALL provide actionable human output, a single structured JSON document under `--json`, and exit status consistent with other schema commands. + +#### Scenario: Successful JSON synchronization +- **WHEN** a user runs `openspec schema sync --json` successfully +- **THEN** stdout SHALL contain one JSON document with the mode, lockfile path, synchronized schema records, and an empty status array +- **AND** the process SHALL exit zero + +#### Scenario: Failed JSON synchronization +- **WHEN** synchronization fails under `--json` +- **THEN** stdout SHALL contain one JSON document with no successful activation claim and a structured error status +- **AND** progress indicators SHALL not corrupt stdout +- **AND** the process SHALL exit non-zero + +#### Scenario: Authentication failure +- **WHEN** system Git cannot authenticate to a private source +- **THEN** the command SHALL report an authentication-safe synchronization error +- **AND** captured output SHALL not include passwords, tokens, authorization headers, or credential-helper secrets diff --git a/openspec/changes/add-remote-schema-sources/specs/schema-resolution/spec.md b/openspec/changes/add-remote-schema-sources/specs/schema-resolution/spec.md new file mode 100644 index 0000000000..d21318917f --- /dev/null +++ b/openspec/changes/add-remote-schema-sources/specs/schema-resolution/spec.md @@ -0,0 +1,92 @@ +## MODIFIED Requirements + +### Requirement: Project-local schema resolution + +The system SHALL preserve project-local, user override, then package built-in precedence for names without a remote declaration. When a project declares a remote source name, the declaration SHALL own that name and SHALL either resolve its verified locked cache or report an actionable remote-schema error. + +#### Scenario: Project-local schema conflicts with remote source +- **WHEN** a schema named "my-workflow" exists at `./openspec/schemas/my-workflow/schema.yaml` +- **AND** project configuration declares a remote source named "my-workflow" +- **AND** `getSchemaDir("my-workflow", projectRoot)` is called +- **THEN** resolution SHALL fail with a `schema_name_conflict` diagnostic +- **AND** the system SHALL NOT silently select either the local or remote bundle + +#### Scenario: Project-local schema takes precedence over user override +- **WHEN** a schema named "my-workflow" exists at `./openspec/schemas/my-workflow/schema.yaml` +- **AND** a schema named "my-workflow" exists at `~/.local/share/openspec/schemas/my-workflow/schema.yaml` +- **AND** `getSchemaDir("my-workflow", projectRoot)` is called +- **THEN** the system SHALL return the project-local path + +#### Scenario: Project-local schema takes precedence over package built-in +- **WHEN** a schema named "spec-driven" exists at `./openspec/schemas/spec-driven/schema.yaml` +- **AND** "spec-driven" is a package built-in schema +- **AND** `getSchemaDir("spec-driven", projectRoot)` is called +- **THEN** the system SHALL return the project-local path + +#### Scenario: Locked remote owns the name over user override +- **WHEN** project configuration declares a synchronized remote source named "my-workflow" +- **AND** a same-named user schema exists +- **THEN** the system SHALL return the verified remote cache path + +#### Scenario: Falls back to user override when no project-local schema +- **WHEN** no schema named "my-workflow" exists at `./openspec/schemas/my-workflow/` +- **AND** project configuration does not declare a remote source named "my-workflow" +- **AND** a schema named "my-workflow" exists at `~/.local/share/openspec/schemas/my-workflow/schema.yaml` +- **AND** `getSchemaDir("my-workflow", projectRoot)` is called +- **THEN** the system SHALL return the user override path + +#### Scenario: Falls back to package built-in when no project-local or user schema +- **WHEN** no project-local, declared remote, or user schema named "spec-driven" exists +- **AND** "spec-driven" is a package built-in schema +- **AND** `getSchemaDir("spec-driven", projectRoot)` is called +- **THEN** the system SHALL return the package built-in path + +#### Scenario: Backward compatibility when projectRoot not provided +- **WHEN** `getSchemaDir("my-workflow")` is called without a `projectRoot` parameter +- **THEN** the system SHALL only check user override and package built-in locations +- **AND** the system SHALL NOT read project configuration, lockfiles, project-local schemas, or remote caches + +## ADDED Requirements + +### Requirement: Declared remote schemas fail closed + +Once a project declares a remote source name, the resolver SHALL either return the matching verified locked cache or report an actionable remote-schema error before considering lower-priority locations. + +#### Scenario: Lock source differs from config +- **WHEN** the lock entry's Git URL, requested ref, or bundle path differs from the project declaration +- **THEN** resolution SHALL fail with a stale-lock diagnostic +- **AND** the diagnostic SHALL instruct the user to run `openspec schema sync ` + +#### Scenario: Schema name differs inside bundle +- **WHEN** cached `schema.yaml` has a name different from the declared source name +- **THEN** resolution SHALL reject the bundle rather than load it under an alias + +### Requirement: Remote schemas participate in discovery + +Schema listing APIs and commands SHALL include valid locked remote schemas with `source: "remote"` while preserving one visible entry per schema name according to resolution priority. + +#### Scenario: Remote schema appears in list +- **WHEN** a declared remote schema has matching lock and cache content +- **THEN** `listSchemas(projectRoot)` SHALL include its name +- **AND** `listSchemasWithInfo(projectRoot)` SHALL report source `remote` + +#### Scenario: Remote schema templates report their source +- **WHEN** a user requests template paths for a declared remote schema with matching lock and cache content +- **THEN** human and JSON template output SHALL report source `remote` +- **AND** the cache location SHALL NOT be mislabeled as a package schema + +#### Scenario: Project schema conflicts with remote in list +- **WHEN** project-local and remote schemas share a name +- **THEN** diagnostic schema listings SHALL contain one unavailable remote entry with `schema_name_conflict` +- **AND** loadable-schema APIs SHALL fail instead of reporting either bundle as active + +#### Scenario: Unsynchronized declaration is listed as unavailable +- **WHEN** project configuration declares a remote schema without usable locked cache content +- **THEN** diagnostic schema surfaces SHALL identify the declaration as requiring synchronization +- **AND** ordinary schema loading SHALL continue to fail closed + +#### Scenario: Same-named lower tiers do not mask an unavailable remote +- **WHEN** a declared remote schema is unavailable +- **AND** a same-named user or package schema exists +- **THEN** diagnostic schema surfaces SHALL report the unavailable remote +- **AND** ordinary schema loading SHALL NOT fall back to the lower-tier schema diff --git a/openspec/changes/add-remote-schema-sources/specs/schema-which-command/spec.md b/openspec/changes/add-remote-schema-sources/specs/schema-which-command/spec.md new file mode 100644 index 0000000000..75b77fc638 --- /dev/null +++ b/openspec/changes/add-remote-schema-sources/specs/schema-which-command/spec.md @@ -0,0 +1,34 @@ +## ADDED Requirements + +### Requirement: Schema which reports remote resolution without network access + +The CLI SHALL report a locked remote schema as source `remote`, include its cache path and immutable lock metadata in JSON output, and perform no Git or network operation while inspecting it. + +#### Scenario: Remote schema is active +- **WHEN** a user runs `openspec schema which qeda-sdd` for a valid locked remote schema +- **THEN** human output SHALL report source `remote`, cache path, requested ref, and resolved commit + +#### Scenario: Remote schema JSON +- **WHEN** a user runs `openspec schema which qeda-sdd --json` +- **THEN** JSON SHALL include `name`, `source`, `path`, `requestedRef`, `resolvedCommit`, `bundlePath`, `integrity`, and `shadows` + +#### Scenario: Project schema conflicts with remote +- **WHEN** a project-local schema and declared remote source share a name +- **THEN** the command SHALL report `schema_name_conflict` +- **AND** it SHALL NOT report either schema as active or shadowed + +#### Scenario: Declared remote cache is unavailable +- **WHEN** a user inspects a declared remote schema whose lock or cache is unusable +- **THEN** the command SHALL exit non-zero with the applicable synchronization fix +- **AND** it SHALL not silently report a same-named user or package schema as active + +#### Scenario: All-schema inspection contains one unavailable remote +- **WHEN** a user runs `openspec schema which --all` +- **AND** one declared remote schema is unavailable while other schemas are usable +- **THEN** the command SHALL return one unavailable entry with a structured status for that remote +- **AND** it SHALL continue returning every usable schema + +#### Scenario: All-schema JSON diagnostics +- **WHEN** a user runs `openspec schema which --all --json` with an unavailable remote +- **THEN** stdout SHALL contain one JSON document with available and unavailable entries +- **AND** each unavailable entry SHALL include a stable error code and actionable message diff --git a/openspec/changes/add-remote-schema-sources/tasks.md b/openspec/changes/add-remote-schema-sources/tasks.md new file mode 100644 index 0000000000..4317760dda --- /dev/null +++ b/openspec/changes/add-remote-schema-sources/tasks.md @@ -0,0 +1,84 @@ +## 1. Configuration and Lock Contracts + +- [x] 1.1 Add failing project-config tests for valid `schemaSources`, invalid names/members, prototype keys, and credential-bearing HTTPS URLs while preserving unrelated fields +- [x] 1.2 Implement normalized Git schema source types and resilient `schemaSources` parsing until the focused config tests pass +- [x] 1.3 Add failing lockfile tests for strict versioned parsing, malformed commits/digests, source mismatch, deterministic ordering, and atomic write preservation +- [x] 1.4 Implement lockfile constants, parser, serializer, and atomic sibling-file replacement until the lockfile tests pass + +## 2. Portable Bundle Validation and Integrity + +- [x] 2.1 Add failing tests for repository-relative bundle paths across POSIX and Windows forms, including traversal, drive, UNC, backslash, reserved-name, and case-fold collision cases +- [x] 2.2 Implement portable Git tree path and entry validation until all cross-platform path cases pass +- [x] 2.3 Add failing tests for canonical digest changes on added/removed/modified files, deterministic ordering, 1,000-file and 10 MiB limits, symlinks, submodules, and non-blob entries +- [x] 2.4 Implement bounded Git-tree extraction, canonical SHA-256 digesting, and content-addressed cache verification until the bundle tests pass +- [x] 2.5 Add failing tests proving missing `schema.yaml`, missing `templates`, escaped template paths, missing template files, and schema-name mismatch are rejected +- [x] 2.6 Extract a reusable core schema-directory validator from the schema CLI and apply stricter remote-bundle validation until existing and new validation tests pass + +## 3. Git Synchronization + +- [x] 3.1 Add failing Git-adapter integration tests using temporary local repositories for branch, lightweight tag, annotated tag, exact commit fetch, bounded failure, and credential-safe errors +- [x] 3.2 Implement the system-Git adapter with argument arrays, tracked-object enumeration, output/time bounds, and sanitized error mapping until adapter tests pass +- [x] 3.3 Add failing synchronization tests for single/all update mode, `--locked` restoration, byte-identical locked operation, and config/lock drift +- [x] 3.4 Implement update and locked synchronization orchestration with temporary extraction, cache rename, and one final atomic lock write +- [x] 3.5 Add failing rollback tests proving invalid upgrades, partial multi-source failures, cache installation failures, and lock replacement failures preserve the previous active state +- [x] 3.6 Complete failure-atomic cleanup and rollback behavior until all synchronization tests pass + +## 4. Resolver Integration + +- [x] 4.1 Add failing resolver tests for project → remote → user → package priority, projectRoot-free compatibility, and one-entry discovery/source reporting +- [x] 4.2 Add failing resolver tests for missing/stale lock, absent cache, digest mismatch, schema-name mismatch, and same-named lower-tier fail-closed behavior +- [x] 4.3 Integrate verified locked remote resolution and `remote` schema metadata into resolver/listing APIs until focused resolver tests pass +- [x] 4.4 Add an offline test that makes Git unavailable after synchronization and proves ordinary resolution still succeeds without spawning any process + +## 5. CLI Behavior + +- [x] 5.1 Add failing CLI tests for `schema sync [name]`, all-source sync, `--locked`, unknown names, no declarations, exit codes, human output, and one-document `--json` output +- [x] 5.2 Register and implement `openspec schema sync` rendering with no spinner or non-JSON stdout under `--json` +- [x] 5.3 Add failing `schema which` and `schemas` tests for active remote metadata, project shadowing, unsynchronized diagnostics, and no-network inspection +- [x] 5.4 Extend schema inspection/listing output and shadow metadata until existing and new schema command tests pass + +## 6. End-to-End Git Journeys and Security + +- [x] 6.1 Add a local-Git end-to-end journey covering initial branch sync, offline use, remote branch advancement with old-lock use, explicit upgrade, and locked cache restoration +- [x] 6.2 Add local-Git rejection journeys for `..`, absolute/drive/UNC paths, tracked symlinks, submodules, incomplete bundles, oversized bundles, and name conflicts +- [x] 6.3 Add a private-auth failure fixture proving human and JSON output do not expose injected credentials or untrusted Git stderr +- [x] 6.4 Run all new path and Git integration tests on Windows-compatible code paths and confirm the repository Windows CI matrix exercises them + +## 7. Documentation and Release Tracking + +- [x] 7.1 Update `docs/customization.md` to distinguish project-local, remote, user-level, and package schemas and document precedence, offline behavior, upgrades, security boundaries, and cache/lock commit policy +- [x] 7.2 Update `docs/cli.md` with sync/update/locked syntax, human/JSON examples, public HTTPS and private SSH examples, CI cache restoration, and failure guidance +- [x] 7.3 Add a minor changeset describing Git-backed schema sources, explicit synchronization, deterministic locks, and network-free normal resolution + +## 8. Verification and Consistency Review + +- [x] 8.1 Run focused config, lock, bundle, Git, resolver, schema CLI, and end-to-end tests and record zero failures +- [x] 8.2 Run `pnpm run build`, `pnpm exec tsc --noEmit`, and `pnpm lint` with zero errors +- [x] 8.3 Run the complete `pnpm test` suite and confirm the Linux/macOS/Windows-sensitive path cases remain covered +- [x] 8.4 Validate `add-remote-schema-sources` strictly and compare proposal, design, specs, tasks, docs, and implementation for unresolved contradictions or scope drift + +## 9. Maintainer Review Blockers + +- [x] 9.1 Add failing nested-directory and store-pointer regression tests, then resolve schema source configuration and locks from the consumer repository root +- [x] 9.2 Add failing same-name project/remote tests, then replace project shadowing with a stable `schema_name_conflict` across resolver, discovery, sync, fork, and `schema which` +- [x] 9.3 Add failing mixed-availability `schema which --all` tests, then return per-entry available/unavailable results without aborting healthy schema inspection +- [x] 9.4 Add failing legacy-local compatibility tests, then split local schema inspection from strict remote-bundle validation +- [x] 9.5 Add failing multi-issue remote validation tests, then preserve structured issue paths through sync and JSON errors +- [x] 9.6 Add a failing real-child-process concurrency test, then add a token-owned project sync lock covering lock read through final write +- [x] 9.7 Add failing live-lock, abandoned-lock, and release-ownership tests, then implement bounded acquisition and safe same-host stale-lock recovery +- [x] 9.8 Add failing SSH environment tests, then preserve existing `GIT_SSH_COMMAND` while enforcing `BatchMode=yes` and a default `StrictHostKeyChecking=accept-new` +- [x] 9.9 Add failing oversized-single-file and stat/read-change tests, then preflight file size before reading and recheck actual bytes +- [x] 9.10 Add failing direct locked-commit ancestry tests, then extract and verify present ancestor, present non-ancestor, and missing-commit paths +- [x] 9.11 Update CLI/customization documentation for consumer-root authority, name conflicts, per-entry diagnostics, sync serialization, local/remote validation, and SSH host-key policy +- [x] 9.12 Run focused tests after each fix, then run strict change validation, build, typecheck, lint, and the complete test suite + +## 10. CodeRabbit Follow-up Hardening + +- [x] 10.1 Add failing sync-lock tests for partial JSON, missing ticket numbers, atomic participant publication, delayed corrupt-state reclamation, and self-ignored runtime files +- [x] 10.2 Implement atomic participant publication, ticket-specific validation, timeout-aged malformed-file reclamation, and the persistent self-ignore coordination directory +- [x] 10.3 Add failing SSH environment tests proving explicit host-key policies are preserved while missing policies default to `accept-new` +- [x] 10.4 Preserve explicit `StrictHostKeyChecking` values, continue normalizing `BatchMode=yes`, and update CLI/customization documentation +- [x] 10.5 Add failing store-backed change tests proving generated configuration remains at the consumer schema root, then separate planning-root scaffolding from schema-root configuration initialization +- [x] 10.6 Add failing remote-template reporting tests, then classify declared remote template paths as source `remote` +- [x] 10.7 Harden affected tests with canonical path comparisons, hermetic Git configuration, actionable child-process stderr, and explicit concurrency timeouts +- [x] 10.8 Run focused tests after each fix, then strict change validation, build, typecheck, lint, and the complete test suite diff --git a/src/cli/index.ts b/src/cli/index.ts index d1bb282998..cdaab53635 100644 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -376,6 +376,7 @@ program await listCommand.execute(root.path, mode, { sort, json: options?.json, + schemaRoot: root.schemaRoot, ...(options?.json ? { root: toRootOutput(root) } : {}), }); } catch (error) { diff --git a/src/commands/schema.ts b/src/commands/schema.ts index 5ec8172be7..04b14526df 100644 --- a/src/commands/schema.ts +++ b/src/commands/schema.ts @@ -6,20 +6,42 @@ import ora from 'ora'; import { stringify as stringifyYaml, parseDocument } from 'yaml'; import { getSchemaDir, + getRemoteSchemaDir, getProjectSchemasDir, getUserSchemasDir, getPackageSchemasDir, isSchemaDir, listSchemas, } from '../core/artifact-graph/resolver.js'; -import { parseSchema, SchemaValidationError } from '../core/artifact-graph/schema.js'; +import { parseSchema } from '../core/artifact-graph/schema.js'; +import { + inspectLocalSchemaDirectory, + SchemaDirectoryValidationError, +} from '../core/artifact-graph/schema-directory.js'; import type { SchemaYaml, Artifact } from '../core/artifact-graph/types.js'; +import { syncRemoteSchemas } from '../core/remote-schema/sync.js'; +import { getSchemaLockPath } from '../core/remote-schema/lockfile.js'; +import { readSchemaLock } from '../core/remote-schema/lockfile.js'; +import type { RemoteSchemaLockEntry } from '../core/remote-schema/types.js'; +import { + assertProjectSchemaNameUnclaimed, + RemoteSchemaResolutionError, +} from '../core/remote-schema/authority.js'; +import { resolveSchemaConsumerRoot } from '../core/remote-schema/consumer-root.js'; +import { SchemaSyncLockError } from '../core/remote-schema/sync-lock.js'; import { FileSystemUtils } from '../utils/file-system.js'; /** * Schema source location type */ -type SchemaSource = 'project' | 'user' | 'package'; +type SchemaSource = 'project' | 'remote' | 'user' | 'package'; + +const SCHEMA_SOURCE_PRECEDENCE: Record = { + project: 0, + remote: 1, + user: 2, + package: 3, +}; /** * Result of checking a schema location @@ -28,6 +50,7 @@ interface SchemaLocation { source: SchemaSource; path: string; exists: boolean; + remote?: RemoteSchemaLockEntry; } /** @@ -35,9 +58,22 @@ interface SchemaLocation { */ interface SchemaResolution { name: string; + available: boolean; source: SchemaSource; - path: string; - shadows: Array<{ source: SchemaSource; path: string }>; + path: string | null; + shadows: Array< + { source: SchemaSource; path: string } & Partial + >; + status: Array<{ + level: 'error'; + code: string; + message: string; + }>; + git?: string; + requestedRef?: string; + resolvedCommit?: string; + bundlePath?: string; + integrity?: string; } /** @@ -49,8 +85,35 @@ interface ValidationIssue { message: string; } +function schemaSyncFailureStatus(error: unknown): Array<{ + level: 'error'; + code: string; + message: string; + path?: string; +}> { + if (error instanceof SchemaDirectoryValidationError) { + return error.issues.map((issue) => ({ + level: 'error', + code: 'remote_schema_invalid', + path: issue.path, + message: issue.message, + })); + } + const code = + error instanceof RemoteSchemaResolutionError + ? error.code + : error instanceof SchemaSyncLockError + ? error.code + : 'schema_sync_failed'; + return [{ + level: 'error', + code, + message: error instanceof Error ? error.message : String(error), + }]; +} + /** - * Check all three locations for a schema and return which ones exist. + * Check all schema locations and return which ones exist. */ function checkAllLocations( name: string, @@ -67,6 +130,27 @@ function checkAllLocations( exists: fs.existsSync(projectSchemaPath), }); + // Locked remote location. A declared remote owns its name; conflicts and + // unavailable cache state are reported by getSchemaResolution(). + let remoteDir: string | null = null; + let remoteEntry: RemoteSchemaLockEntry | undefined; + try { + remoteEntry = readSchemaLock(projectRoot)?.schemas[name]; + } catch { + remoteEntry = undefined; + } + try { + remoteDir = getRemoteSchemaDir(name, projectRoot); + } catch { + remoteDir = null; + } + locations.push({ + source: 'remote', + path: remoteDir ?? path.join(getUserSchemasDir(), '..', 'schema-cache'), + exists: remoteDir !== null, + ...(remoteEntry ? { remote: remoteEntry } : {}), + }); + // User location const userDir = path.join(getUserSchemasDir(), name); const userSchemaPath = path.join(userDir, 'schema.yaml'); @@ -95,24 +179,69 @@ function getSchemaResolution( name: string, projectRoot: string ): SchemaResolution | null { - const locations = checkAllLocations(name, projectRoot); - const existingLocations = locations.filter((loc) => loc.exists); - - if (existingLocations.length === 0) { + let resolvedDir: string | null; + try { + resolvedDir = getSchemaDir(name, projectRoot); + } catch (error) { + if (error instanceof RemoteSchemaResolutionError) { + let remote: RemoteSchemaLockEntry | undefined; + try { + remote = readSchemaLock(projectRoot)?.schemas[name]; + } catch { + remote = undefined; + } + return { + name, + available: false, + source: 'remote', + path: null, + shadows: [], + status: [ + { + level: 'error', + code: error.code, + message: error.message, + }, + ], + ...(remote ?? {}), + }; + } + throw error; + } + if (!resolvedDir) { return null; } - - const active = existingLocations[0]; - const shadows = existingLocations.slice(1).map((loc) => ({ - source: loc.source, - path: loc.path, - })); - + const locations = checkAllLocations(name, projectRoot); + const existingLocations = locations.filter((loc) => loc.exists); + const normalizedResolvedDir = path.resolve(resolvedDir); + const active = existingLocations.find( + (location) => path.resolve(location.path) === normalizedResolvedDir + ) ?? { + source: 'remote' as const, + path: resolvedDir, + exists: true, + remote: locations.find((location) => location.source === 'remote')?.remote, + }; + const activeRank = SCHEMA_SOURCE_PRECEDENCE[active.source]; + const shadows = existingLocations + .filter( + (location) => + SCHEMA_SOURCE_PRECEDENCE[location.source] > activeRank && + path.resolve(location.path) !== normalizedResolvedDir + ) + .map((loc) => ({ + source: loc.source, + path: loc.path, + ...(loc.remote ?? {}), + })); return { name, + available: true, source: active.source, path: active.path, shadows, + status: [], + ...(active.remote ?? {}), }; } @@ -142,96 +271,36 @@ function validateSchema( schemaDir: string, verbose: boolean = false ): { valid: boolean; issues: ValidationIssue[] } { - const issues: ValidationIssue[] = []; - const schemaPath = path.join(schemaDir, 'schema.yaml'); - - // Check schema.yaml exists if (verbose) { console.log(' Checking schema.yaml exists...'); - } - if (!fs.existsSync(schemaPath)) { - issues.push({ - level: 'error', - path: 'schema.yaml', - message: 'schema.yaml not found', - }); - return { valid: false, issues }; - } - - // Parse YAML - if (verbose) { console.log(' Parsing YAML...'); - } - let content: string; - try { - content = fs.readFileSync(schemaPath, 'utf-8'); - } catch (err) { - issues.push({ - level: 'error', - path: 'schema.yaml', - message: `Failed to read file: ${(err as Error).message}`, - }); - return { valid: false, issues }; - } - - // Validate against Zod schema - if (verbose) { console.log(' Validating schema structure...'); + console.log(' Checking template files...'); } - let schema: SchemaYaml; + let inspection; try { - schema = parseSchema(content); - } catch (err) { - if (err instanceof SchemaValidationError) { - issues.push({ - level: 'error', - path: 'schema.yaml', - message: err.message, - }); - } else { - issues.push({ + inspection = inspectLocalSchemaDirectory(schemaDir); + } catch (error) { + return { + valid: false, + issues: [{ level: 'error', path: 'schema.yaml', - message: `Parse error: ${(err as Error).message}`, - }); - } - return { valid: false, issues }; - } - - // Check template files exist in the same directory used at runtime. - if (verbose) { - console.log(' Checking template files...'); - } - for (const artifact of schema.artifacts) { - const templatesDir = path.join(schemaDir, 'templates'); - const existingTemplatePath = path.join(templatesDir, artifact.template); - - if (!fs.existsSync(existingTemplatePath)) { - issues.push({ - level: 'error', - path: `artifacts.${artifact.id}.template`, - message: `Template file '${artifact.template}' not found for artifact '${artifact.id}'`, - }); - continue; - } - - try { - FileSystemUtils.assertPathWithin(templatesDir, existingTemplatePath); - } catch { - issues.push({ - level: 'error', - path: `artifacts.${artifact.id}.template`, - message: `Template file '${artifact.template}' points outside the schema templates directory`, - }); - } + message: error instanceof Error ? error.message : String(error), + }], + }; } + const issues: ValidationIssue[] = inspection.issues.map((issue) => ({ + level: 'error', + path: issue.path, + message: issue.message, + })); // Dependency graph validation is already done by parseSchema // (it throws on cycles and invalid references) if (verbose) { console.log(' Dependency graph validation passed (via parseSchema)'); } - return { valid: issues.length === 0, issues }; } @@ -419,6 +488,75 @@ export function registerSchemaCommand(program: Command): void { console.error('Note: Schema commands are experimental and may change.'); }); + // schema sync + schemaCmd + .command('sync [name]') + .description('Synchronize project-declared Git schema sources') + .option('--locked', 'Restore exactly the commits and content in the lockfile') + .option('--json', 'Output as JSON') + .action( + async ( + name?: string, + options?: { locked?: boolean; json?: boolean } + ) => { + let projectRoot: string | null = null; + try { + projectRoot = resolveSchemaConsumerRoot(process.cwd()); + if (!projectRoot) { + throw new Error( + `No consumer OpenSpec project found from '${process.cwd()}' or its ancestors` + ); + } + const result = await syncRemoteSchemas(projectRoot, { + name, + locked: options?.locked, + }); + if (options?.json) { + console.log(JSON.stringify({ synced: true, ...result }, null, 2)); + } else { + for (const schema of result.schemas) { + const action = schema.restored + ? 'Restored' + : result.locked + ? 'Verified' + : 'Synced'; + console.log( + `${action} '${schema.name}': ${schema.requestedRef} → ${schema.resolvedCommit}` + ); + console.log(` Cache: ${schema.cachePath} (${schema.integrity})`); + } + console.log( + `${result.locked ? 'Verified' : 'Updated'} ${result.lockfile}` + ); + } + } catch (error) { + const status = schemaSyncFailureStatus(error); + if (options?.json) { + console.log( + JSON.stringify( + { + synced: false, + mode: options?.locked ? 'locked' : 'update', + lockfile: projectRoot ? getSchemaLockPath(projectRoot) : null, + schemas: [], + status, + error: (error as Error).message, + }, + null, + 2 + ) + ); + } else { + for (const diagnostic of status) { + const location = diagnostic.path ? `${diagnostic.path}: ` : ''; + console.error(`Error: ${location}${diagnostic.message}`); + } + } + process.exitCode = 1; + } + } + ); + // schema which schemaCmd .command('which [name]') @@ -427,7 +565,8 @@ export function registerSchemaCommand(program: Command): void { .option('--all', 'List all schemas with their resolution sources') .action(async (name?: string, options?: { json?: boolean; all?: boolean }) => { try { - const projectRoot = process.cwd(); + const projectRoot = + resolveSchemaConsumerRoot(process.cwd()) ?? process.cwd(); if (options?.all) { // List all schemas @@ -443,9 +582,11 @@ export function registerSchemaCommand(program: Command): void { // Group by source const bySource = { - project: schemas.filter((s) => s.source === 'project'), - user: schemas.filter((s) => s.source === 'user'), - package: schemas.filter((s) => s.source === 'package'), + project: schemas.filter((s) => s.available && s.source === 'project'), + remote: schemas.filter((s) => s.available && s.source === 'remote'), + user: schemas.filter((s) => s.available && s.source === 'user'), + package: schemas.filter((s) => s.available && s.source === 'package'), + unavailable: schemas.filter((s) => !s.available), }; if (bySource.project.length > 0) { @@ -458,6 +599,16 @@ export function registerSchemaCommand(program: Command): void { } } + if (bySource.remote.length > 0) { + console.log('\nRemote schemas:'); + for (const schema of bySource.remote) { + const shadowInfo = schema.shadows.length > 0 + ? ` (shadows: ${schema.shadows.map((s) => s.source).join(', ')})` + : ''; + console.log(` ${schema.name}${shadowInfo}`); + } + } + if (bySource.user.length > 0) { console.log('\nUser schemas:'); for (const schema of bySource.user) { @@ -474,12 +625,42 @@ export function registerSchemaCommand(program: Command): void { console.log(` ${schema.name}`); } } + + if (bySource.unavailable.length > 0) { + console.log('\nUnavailable schemas:'); + for (const schema of bySource.unavailable) { + console.log(` ${schema.name}: ${schema.status[0]?.message ?? 'Unavailable'}`); + } + } } return; } if (!name) { - console.error('Error: Schema name is required (or use --all to list all schemas)'); + const message = 'Schema name is required (or use --all to list all schemas)'; + if (options?.json) { + console.log( + JSON.stringify( + { + name: null, + source: null, + path: null, + shadows: [], + status: [ + { + level: 'error', + code: 'schema_name_required', + message, + }, + ], + }, + null, + 2 + ) + ); + } else { + console.error(`Error: ${message}`); + } process.exitCode = 1; return; } @@ -501,12 +682,28 @@ export function registerSchemaCommand(program: Command): void { return; } + if (!resolution.available) { + if (options?.json) { + console.log(JSON.stringify(resolution, null, 2)); + } else { + console.error(`Error: ${resolution.status[0]?.message ?? `Schema '${name}' is unavailable`}`); + } + process.exitCode = 1; + return; + } + if (options?.json) { console.log(JSON.stringify(resolution, null, 2)); } else { console.log(`Schema: ${resolution.name}`); console.log(`Source: ${resolution.source}`); console.log(`Path: ${resolution.path}`); + if (resolution.requestedRef && resolution.resolvedCommit) { + console.log( + `Locked: ${resolution.requestedRef} → ${resolution.resolvedCommit}` + ); + console.log(`Integrity: ${resolution.integrity}`); + } if (resolution.shadows.length > 0) { console.log('\nShadows:'); @@ -516,7 +713,29 @@ export function registerSchemaCommand(program: Command): void { } } } catch (error) { - console.error(`Error: ${(error as Error).message}`); + if (options?.json) { + console.log( + JSON.stringify( + { + name: name ?? null, + source: null, + path: null, + shadows: [], + status: [ + { + level: 'error', + code: 'schema_resolution_failed', + message: (error as Error).message, + }, + ], + }, + null, + 2 + ) + ); + } else { + console.error(`Error: ${(error as Error).message}`); + } process.exitCode = 1; } }); @@ -529,7 +748,8 @@ export function registerSchemaCommand(program: Command): void { .option('--verbose', 'Show detailed validation steps') .action(async (name?: string, options?: { json?: boolean; verbose?: boolean }) => { try { - const projectRoot = process.cwd(); + const projectRoot = + resolveSchemaConsumerRoot(process.cwd()) ?? process.cwd(); if (!name) { // Validate all project schemas @@ -678,7 +898,8 @@ export function registerSchemaCommand(program: Command): void { const spinner = options?.json ? null : ora(); try { - const projectRoot = process.cwd(); + const projectRoot = + resolveSchemaConsumerRoot(process.cwd()) ?? process.cwd(); const destinationName = name || `${source}-custom`; // Validate destination name @@ -696,6 +917,8 @@ export function registerSchemaCommand(program: Command): void { return; } + assertProjectSchemaNameUnclaimed(projectRoot, destinationName); + // Find source schema const sourceDir = getSchemaDir(source, projectRoot); if (!sourceDir) { @@ -909,6 +1132,9 @@ export function registerSchemaCommand(program: Command): void { if (options?.json) { console.log(JSON.stringify({ forked: false, + ...(error instanceof RemoteSchemaResolutionError + ? { code: error.code } + : {}), error: (error as Error).message, }, null, 2)); } else { @@ -941,7 +1167,8 @@ export function registerSchemaCommand(program: Command): void { const spinner = options?.json ? null : ora(); try { - const projectRoot = process.cwd(); + const projectRoot = + resolveSchemaConsumerRoot(process.cwd()) ?? process.cwd(); // Validate name if (!isValidSchemaName(name)) { @@ -958,6 +1185,8 @@ export function registerSchemaCommand(program: Command): void { return; } + assertProjectSchemaNameUnclaimed(projectRoot, name); + const schemaDir = path.join(getProjectSchemasDir(projectRoot), name); // Check overwrite permission without mutating the destination @@ -1179,6 +1408,9 @@ export function registerSchemaCommand(program: Command): void { if (options?.json) { console.log(JSON.stringify({ created: false, + ...(error instanceof RemoteSchemaResolutionError + ? { code: error.code } + : {}), error: (error as Error).message, }, null, 2)); } else { diff --git a/src/commands/validate.ts b/src/commands/validate.ts index 8f7428e647..1bc244628d 100644 --- a/src/commands/validate.ts +++ b/src/commands/validate.ts @@ -213,7 +213,7 @@ export class ValidateCommand { } private async validateByType(root: ResolvedOpenSpecRoot, type: ItemType, id: string, opts: { strict: boolean; json: boolean }): Promise { - const validator = new Validator(opts.strict); + const validator = new Validator(opts.strict, root.schemaRoot); if (type === 'change') { const changeDir = path.join(root.changesDir, id); const start = Date.now(); @@ -295,7 +295,7 @@ export class ValidateCommand { const DEFAULT_CONCURRENCY = 6; const maxSuggestions = 5; // used by nearestMatches const concurrency = normalizeConcurrency(opts.concurrency) ?? normalizeConcurrency(process.env.OPENSPEC_CONCURRENCY) ?? DEFAULT_CONCURRENCY; - const validator = new Validator(opts.strict); + const validator = new Validator(opts.strict, root.schemaRoot); const queue: Array<() => Promise> = []; for (const id of changeIds) { @@ -468,7 +468,13 @@ export class ValidateCommand { // The explicit root.path override is load-bearing: an archived change // lives one directory deeper (changes/archive/), so the default // "../../.." projectRoot derivation would be wrong without it. - const progress = await getTaskProgressDetailForChange(root.archiveDir, id, root.path, schemaGlobCache); + const progress = await getTaskProgressDetailForChange( + root.archiveDir, + id, + root.path, + schemaGlobCache, + root.schemaRoot + ); // A tasks file that exists but cannot be read must fail loudly, not be // silently counted as "no tasks" and pass. Report one issue per file, // pathed like every other validate issue (POSIX, root-relative). diff --git a/src/commands/workflow/instructions.ts b/src/commands/workflow/instructions.ts index 1ae6fac7c0..0c32fce866 100644 --- a/src/commands/workflow/instructions.ts +++ b/src/commands/workflow/instructions.ts @@ -82,10 +82,13 @@ export type ArchiveInstructionsOptions = ApplyInstructionsOptions; */ async function loadRootConfigContext(root: ResolvedOpenSpecRoot): Promise<{ projectConfig: ProjectConfig | null; + schemaConfig: ProjectConfig | null; references: ReferenceIndexEntry[] | undefined; }> { // readProjectConfig never throws: missing/unparseable configs are null. const projectConfig = readProjectConfig(root.path); + const schemaConfig = + root.schemaRoot === root.path ? projectConfig : readProjectConfig(root.schemaRoot); // One registry read serves every relationship consumer in this // output so it never carries a torn snapshot. @@ -102,6 +105,7 @@ async function loadRootConfigContext(root: ResolvedOpenSpecRoot): Promise<{ // look identical to an undeclared one in JSON. return { projectConfig, + schemaConfig, references: index.length > 0 ? index : undefined, }; } @@ -121,6 +125,7 @@ export async function instructionsCommand( try { const planningHome = toPlanningHome(root); const projectRoot = root.path; + const schemaRoot = root.schemaRoot; const changeName = await validateChangeExists( options.change, projectRoot, @@ -130,16 +135,18 @@ export async function instructionsCommand( // Validate schema if explicitly provided if (options.schema) { - validateSchemaExists(options.schema, projectRoot); + validateSchemaExists(options.schema, schemaRoot); } - const { projectConfig, references } = await loadRootConfigContext(root); + const { projectConfig, schemaConfig, references } = await loadRootConfigContext(root); // loadChangeContext will auto-detect schema from metadata if not provided const context = loadChangeContext(projectRoot, changeName, options.schema, { changeDir: getChangeDir(planningHome, changeName), planningHome, projectConfig, + schemaConfig, + schemaRoot, }); if (!artifactId) { @@ -162,6 +169,7 @@ export async function instructionsCommand( const instructions = generateInstructions(context, artifactId, projectRoot, { projectConfig, + schemaConfig, references, }); const isBlocked = instructions.dependencies.some((d) => !d.done); @@ -354,6 +362,8 @@ export interface GenerateApplyInstructionsOptions { planningHome?: PlanningHome; references?: ReferenceIndexEntry[]; projectConfig?: ProjectConfig | null; + schemaConfig?: ProjectConfig | null; + schemaRoot?: string; } /** @@ -375,11 +385,17 @@ export async function generateApplyInstructions( changeDir: getChangeDir(planningHome, changeName), planningHome, projectConfig: options.projectConfig, + schemaConfig: options.schemaConfig, + schemaRoot: options.schemaRoot, }); const changeDir = context.changeDir; // Get the full schema to access the apply phase configuration - const schema = resolveSchema(context.schemaName, projectRoot); + const schema = resolveSchema( + context.schemaName, + context.schemaRoot, + options.schemaConfig !== undefined ? options.schemaConfig : options.projectConfig + ); const applyConfig = schema.apply; // Determine required artifacts and tracking file from schema @@ -488,6 +504,7 @@ export async function applyInstructionsCommand(options: ApplyInstructionsOptions try { const planningHome = toPlanningHome(root); const projectRoot = root.path; + const schemaRoot = root.schemaRoot; const changeName = await validateChangeExists( options.change, projectRoot, @@ -497,16 +514,18 @@ export async function applyInstructionsCommand(options: ApplyInstructionsOptions // Validate schema if explicitly provided if (options.schema) { - validateSchemaExists(options.schema, projectRoot); + validateSchemaExists(options.schema, schemaRoot); } - // One parsed config snapshot supplies schema fallback, references, context, - // and operation guidance for this command. - const { projectConfig, references } = await loadRootConfigContext(root); + // Planning-root config supplies references, context, and operation guidance; + // consumer-root config independently owns schema resolution. + const { projectConfig, schemaConfig, references } = await loadRootConfigContext(root); const instructions = await generateApplyInstructions(projectRoot, changeName, options.schema, { planningHome, references, projectConfig, + schemaConfig, + schemaRoot, }); spinner?.stop(); diff --git a/src/commands/workflow/new-change.ts b/src/commands/workflow/new-change.ts index 3e059242dc..cf9760e1aa 100644 --- a/src/commands/workflow/new-change.ts +++ b/src/commands/workflow/new-change.ts @@ -109,10 +109,11 @@ export async function newChangeCommand(name: string | undefined, options: NewCha } const projectRoot = root.path; + const schemaRoot = root.schemaRoot; // Validate schema if provided if (options.schema) { - validateSchemaExists(options.schema, projectRoot); + validateSchemaExists(options.schema, schemaRoot); } const resolvedSchema = options.schema ?? root.defaultSchema; @@ -124,6 +125,7 @@ export async function newChangeCommand(name: string | undefined, options: NewCha schema: options.schema, defaultSchema: root.defaultSchema, changesDir: root.changesDir, + schemaRoot, metadata: { ...(options.goal ? { goal: options.goal } : {}), }, diff --git a/src/commands/workflow/schemas.ts b/src/commands/workflow/schemas.ts index 0880e20c9a..22a9c1eb31 100644 --- a/src/commands/workflow/schemas.ts +++ b/src/commands/workflow/schemas.ts @@ -31,6 +31,9 @@ export async function schemasCommand(options: SchemasOptions): Promise { return; } + // `schemas --store` is an explicit inspection of the selected root. Schema + // ownership for workflow execution remains on `root.schemaRoot`, but this + // discovery command must preserve the selected-store listing contract. const schemas = listSchemasWithInfo(root.path); if (options.json) { @@ -45,10 +48,23 @@ export async function schemasCommand(options: SchemasOptions): Promise { let sourceLabel = ''; if (schema.source === 'project') { sourceLabel = chalk.cyan(' (project)'); + } else if (schema.source === 'remote') { + sourceLabel = schema.available === false + ? chalk.yellow(' (remote, unavailable)') + : chalk.cyan(' (remote)'); } else if (schema.source === 'user') { sourceLabel = chalk.dim(' (user override)'); } console.log(` ${chalk.bold(schema.name)}${sourceLabel}`); + if (schema.available === false) { + const diagnostic = schema.status?.[0]; + const message = diagnostic + ? `${diagnostic.code}: ${diagnostic.message}` + : schema.error ?? 'Remote schema is unavailable'; + console.log(` ${chalk.yellow(message)}`); + console.log(); + continue; + } console.log(` ${schema.description}`); console.log(` Artifacts: ${schema.artifacts.join(' → ')}`); console.log(); diff --git a/src/commands/workflow/status.ts b/src/commands/workflow/status.ts index 32f5950716..1456f92f31 100644 --- a/src/commands/workflow/status.ts +++ b/src/commands/workflow/status.ts @@ -56,6 +56,7 @@ export async function statusCommand(options: StatusOptions): Promise { try { const planningHome = toPlanningHome(root); const projectRoot = root.path; + const schemaRoot = root.schemaRoot; const rootOutput = toRootOutput(root); const newChangeHint = withStoreFlag(root, 'openspec new change '); @@ -94,13 +95,14 @@ export async function statusCommand(options: StatusOptions): Promise { // Validate schema if explicitly provided if (options.schema) { - validateSchemaExists(options.schema, projectRoot); + validateSchemaExists(options.schema, schemaRoot); } // loadChangeContext will auto-detect schema from metadata if not provided const context = loadChangeContext(projectRoot, changeName, options.schema, { changeDir: getChangeDir(planningHome, changeName), planningHome, + schemaRoot, }); const status = formatChangeStatus( context, diff --git a/src/commands/workflow/templates.ts b/src/commands/workflow/templates.ts index 02d2c5a01d..b32b7eb4f7 100644 --- a/src/commands/workflow/templates.ts +++ b/src/commands/workflow/templates.ts @@ -9,8 +9,10 @@ import path from 'path'; import { resolveSchema, getSchemaDir, + listSchemasWithInfo, ArtifactGraph, } from '../../core/artifact-graph/index.js'; +import { resolveSchemaConsumerRoot } from '../../core/remote-schema/consumer-root.js'; import { FileSystemUtils } from '../../utils/file-system.js'; import { validateSchemaExists, DEFAULT_SCHEMA } from './shared.js'; @@ -26,7 +28,7 @@ export interface TemplatesOptions { export interface TemplateInfo { artifactId: string; templatePath: string; - source: 'project' | 'user' | 'package'; + source: 'project' | 'remote' | 'user' | 'package'; } // ----------------------------------------------------------------------------- @@ -37,35 +39,16 @@ export async function templatesCommand(options: TemplatesOptions): Promise const spinner = options.json ? undefined : ora('Loading templates...').start(); try { - const projectRoot = process.cwd(); + const projectRoot = resolveSchemaConsumerRoot(process.cwd()) ?? process.cwd(); const schemaName = validateSchemaExists(options.schema ?? DEFAULT_SCHEMA, projectRoot); const schema = resolveSchema(schemaName, projectRoot); const graph = ArtifactGraph.fromSchema(schema); const schemaDir = getSchemaDir(schemaName, projectRoot)!; - // Determine the source (project, user, or package) - const { - getUserSchemasDir, - getProjectSchemasDir, - } = await import('../../core/artifact-graph/resolver.js'); - const projectSchemasDir = getProjectSchemasDir(projectRoot); - const userSchemasDir = getUserSchemasDir(); - - // Determine source by checking if schemaDir is inside each base directory - // Using path.relative is more robust than startsWith for path comparisons - const isInsideDir = (child: string, parent: string): boolean => { - const relative = path.relative(parent, child); - return !relative.startsWith('..') && !path.isAbsolute(relative); - }; - - let source: 'project' | 'user' | 'package'; - if (isInsideDir(schemaDir, projectSchemasDir)) { - source = 'project'; - } else if (isInsideDir(schemaDir, userSchemasDir)) { - source = 'user'; - } else { - source = 'package'; - } + const source = + listSchemasWithInfo(projectRoot).find( + (schemaInfo) => schemaInfo.name === schemaName + )?.source ?? 'package'; const templatesDir = path.join(schemaDir, 'templates'); const templates: TemplateInfo[] = graph.getAllArtifacts().map((artifact) => { diff --git a/src/core/archive.ts b/src/core/archive.ts index 888a6135a6..1e9f81bc59 100644 --- a/src/core/archive.ts +++ b/src/core/archive.ts @@ -794,12 +794,13 @@ async function fingerprintPortableContent(filePath: string): Promise { async function assertRetirementAuthorization( changeDir: string, expectedFingerprint: string, - options: { verifyMarker?: boolean } = {} + options: { verifyMarker?: boolean; schemaRoot?: string } = {} ): Promise { const metadataPath = path.join(changeDir, METADATA_FILENAME); const before = await fingerprintPortableContent(metadataPath); const markerStillDeclared = - options.verifyMarker === false || readRetireCapabilitiesMarker(changeDir).declared; + options.verifyMarker === false || + readRetireCapabilitiesMarker(changeDir, options.schemaRoot).declared; const after = await fingerprintPortableContent(metadataPath); if ( before !== expectedFingerprint || @@ -1181,7 +1182,7 @@ export class ArchiveCommand { // Validate specs and change before archiving if (!skipValidation) { - const validator = new Validator(); + const validator = new Validator(false, root.schemaRoot); let hasValidationErrors = false; // Validate proposal.md (informative only; human mode prints warnings) @@ -1237,7 +1238,7 @@ export class ArchiveCommand { // proposal warnings — a gap that predates the marker and is left // unchanged here.) if (!hasDeltaSpecs) { - const marker = readSkipSpecsMarker(changeDir); + const marker = readSkipSpecsMarker(changeDir, root.schemaRoot); if (marker.invalidReason) { hasDeltaSpecs = true; } else if (marker.declared) { @@ -1332,7 +1333,12 @@ export class ArchiveCommand { } // Show progress and check for incomplete tasks - const progress = await getTaskProgressForChange(changesDir, changeName, path.resolve(changesDir, '..', '..')); + const progress = await getTaskProgressForChange( + changesDir, + changeName, + path.resolve(changesDir, '..', '..'), + root.schemaRoot + ); if (!json) { const status = formatTaskStatus(progress); console.log(`Task status: ${status}`); @@ -1387,7 +1393,10 @@ export class ArchiveCommand { // retire a capability at all. An unhonorable marker counts as undeclared, // exactly as skip_specs treats one, so metadata the rest of the CLI rejects // can never authorise a deletion. - const retirementMarker = readRetireCapabilitiesMarker(changeDir); + const retirementMarker = readRetireCapabilitiesMarker( + changeDir, + root.schemaRoot + ); const retirementDeclared = retirementMarker.declared; const retirementAuthorizationFingerprint = retirementDeclared ? await fingerprintPortableContent(path.join(changeDir, METADATA_FILENAME)) @@ -1516,7 +1525,10 @@ export class ArchiveCommand { // delete a requirement added while the prompt was waiting. if (prepareError === undefined) { try { - const currentRetirementMarker = readRetireCapabilitiesMarker(changeDir); + const currentRetirementMarker = readRetireCapabilitiesMarker( + changeDir, + root.schemaRoot + ); if ( currentRetirementMarker.declared !== retirementMarker.declared || currentRetirementMarker.invalidReason !== retirementMarker.invalidReason @@ -1779,7 +1791,8 @@ export class ArchiveCommand { } await assertRetirementAuthorization( changeDir, - retirementAuthorizationFingerprint + retirementAuthorizationFingerprint, + { schemaRoot: root.schemaRoot } ); if ( (await fingerprintSpecInputs(p.update)) !== @@ -1794,7 +1807,8 @@ export class ArchiveCommand { verifyDisplaced: async (displacedPath) => { await assertRetirementAuthorization( changeDir, - retirementAuthorizationFingerprint! + retirementAuthorizationFingerprint!, + { schemaRoot: root.schemaRoot } ); if ( (await fingerprintMovablePath(displacedPath)) !== @@ -1915,7 +1929,8 @@ export class ArchiveCommand { if (hasRetirements) { await assertRetirementAuthorization( changeDir, - retirementAuthorizationFingerprint! + retirementAuthorizationFingerprint!, + { schemaRoot: root.schemaRoot } ); } const verifyArchivedDeltas = async ( @@ -1934,7 +1949,8 @@ export class ArchiveCommand { if (stagedSource) { await assertRetirementAuthorization( stagedSource, - retirementAuthorizationFingerprint! + retirementAuthorizationFingerprint!, + { schemaRoot: root.schemaRoot } ); } } @@ -2088,7 +2104,12 @@ export class ArchiveCommand { try { const progressList: Array<{ id: string; status: string }> = []; for (const id of changeDirs) { - const progress = await getTaskProgressForChange(changesDir, id, path.resolve(changesDir, '..', '..')); + const progress = await getTaskProgressForChange( + changesDir, + id, + root.path, + root.schemaRoot + ); const status = formatTaskStatus(progress); progressList.push({ id, status }); } diff --git a/src/core/artifact-graph/index.ts b/src/core/artifact-graph/index.ts index 2a2d346d00..05f5d4f01e 100644 --- a/src/core/artifact-graph/index.ts +++ b/src/core/artifact-graph/index.ts @@ -29,6 +29,7 @@ export { listSchemas, listSchemasWithInfo, getSchemaDir, + getRemoteSchemaDir, getPackageSchemasDir, getUserSchemasDir, SchemaLoadError, diff --git a/src/core/artifact-graph/instruction-loader.ts b/src/core/artifact-graph/instruction-loader.ts index e1363daaab..ac563ca351 100644 --- a/src/core/artifact-graph/instruction-loader.ts +++ b/src/core/artifact-graph/instruction-loader.ts @@ -55,6 +55,8 @@ export interface ChangeContext { changeDir: string; /** Project root directory */ projectRoot: string; + /** Consumer repository that owns schema configuration and caches. */ + schemaRoot: string; /** Resolved planning home for this change */ planningHome?: PlanningHome; /** Parsed change metadata, when present */ @@ -70,8 +72,11 @@ export interface ChangeContext { export interface LoadChangeContextOptions { changeDir?: string; planningHome?: PlanningHome; - /** Pre-read project config; suppresses schema resolution's fallback config read. */ + /** Pre-read planning-root config used for instruction context and rules. */ projectConfig?: ProjectConfig | null; + /** Pre-read consumer-root config used for schema selection and resolution. */ + schemaConfig?: ProjectConfig | null; + schemaRoot?: string; } /** @@ -207,9 +212,10 @@ export interface ArtifactPathSummary { export function loadTemplate( schemaName: string, templatePath: string, - projectRoot?: string + projectRoot?: string, + projectConfig?: ProjectConfig | null ): string { - const schemaDir = getSchemaDir(schemaName, projectRoot); + const schemaDir = getSchemaDir(schemaName, projectRoot, projectConfig); if (!schemaDir) { throw new TemplateLoadError( `Schema '${schemaName}' not found`, @@ -268,17 +274,20 @@ export function loadChangeContext( schemaName?: string, options: LoadChangeContextOptions = {} ): ChangeContext { + const schemaRoot = options.schemaRoot ?? projectRoot; + const schemaConfig = + options.schemaConfig !== undefined ? options.schemaConfig : options.projectConfig; const changeDir = FileSystemUtils.canonicalizeExistingPath( options.changeDir ?? path.join(projectRoot, 'openspec', 'changes', changeName) ); - const metadata = readChangeMetadata(changeDir, projectRoot) ?? undefined; - const resolvedSchemaName = resolveSchemaForChange(changeDir, schemaName, projectRoot, { + const metadata = readChangeMetadata(changeDir, schemaRoot, schemaConfig) ?? undefined; + const resolvedSchemaName = resolveSchemaForChange(changeDir, schemaName, schemaRoot, { metadata: metadata ?? null, - projectConfig: options.projectConfig, + projectConfig: schemaConfig, }); - const schema = resolveSchema(resolvedSchemaName, projectRoot); + const schema = resolveSchema(resolvedSchemaName, schemaRoot, schemaConfig); const graph = ArtifactGraph.fromSchema(schema); const completed = detectCompleted(graph, changeDir); @@ -303,6 +312,7 @@ export function loadChangeContext( changeName, changeDir, projectRoot, + schemaRoot, ...(options.planningHome ? { planningHome: options.planningHome } : {}), ...(metadata ? { metadata } : {}), ...(skippedArtifacts.size > 0 ? { skippedArtifacts } : {}), @@ -324,8 +334,10 @@ export function loadChangeContext( * @throws Error if artifact not found */ export interface GenerateInstructionsOptions { - /** Pre-read project config; suppresses the internal read (no double read). */ + /** Pre-read planning-root config for context, rules, and operation guidance. */ projectConfig?: ProjectConfig | null; + /** Pre-read consumer-root config for schema and template resolution. */ + schemaConfig?: ProjectConfig | null; /** Referenced-store index assembled at the command boundary. */ references?: ReferenceIndexEntry[]; } @@ -341,7 +353,14 @@ export function generateInstructions( throw new Error(`Artifact '${artifactId}' not found in schema '${context.schemaName}'`); } - const templateContent = loadTemplate(context.schemaName, artifact.template, context.projectRoot); + const schemaConfig = + options.schemaConfig !== undefined ? options.schemaConfig : options.projectConfig; + const templateContent = loadTemplate( + context.schemaName, + artifact.template, + context.schemaRoot, + schemaConfig + ); const dependencies = getDependencyInfo(artifact, context.graph, context.completed, context.skippedArtifacts); const unlocks = getUnlockedArtifacts(context.graph, artifactId); @@ -363,7 +382,7 @@ export function generateInstructions( // key is only "unknown" when it matches no artifact in ANY available schema. if (projectConfig?.rules) { const validArtifactIds = new Set( - listSchemasWithInfo(effectiveProjectRoot ?? undefined).flatMap((s) => s.artifacts) + listSchemasWithInfo(context.schemaRoot, schemaConfig).flatMap((s) => s.artifacts) ); const warnings = validateConfigRules(projectConfig.rules, validArtifactIds); @@ -458,7 +477,7 @@ export function formatChangeStatus( options: { storeId?: string } = {} ): ChangeStatus { // Load schema to get apply phase configuration - const schema = resolveSchema(context.schemaName, context.projectRoot); + const schema = resolveSchema(context.schemaName, context.schemaRoot); const applyRequires = schema.apply?.requires ?? schema.artifacts.map(a => a.id); const artifacts = context.graph.getAllArtifacts(); diff --git a/src/core/artifact-graph/resolver.ts b/src/core/artifact-graph/resolver.ts index f7d84d140c..310d3740a7 100644 --- a/src/core/artifact-graph/resolver.ts +++ b/src/core/artifact-graph/resolver.ts @@ -2,6 +2,14 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; import { fileURLToPath } from 'node:url'; import { getGlobalDataDir } from '../global-config.js'; +import { readProjectConfig, type ProjectConfig } from '../project-config.js'; +import { verifyRemoteSchemaCache } from '../remote-schema/cache.js'; +import { readSchemaLock } from '../remote-schema/lockfile.js'; +import { + assertNoProjectSchemaConflict, + RemoteSchemaResolutionError, +} from '../remote-schema/authority.js'; +import { validateRemoteSchemaDirectory } from './schema-directory.js'; import { FileSystemUtils } from '../../utils/file-system.js'; import { parseSchema, SchemaValidationError } from './schema.js'; import type { SchemaYaml } from './types.js'; @@ -46,6 +54,70 @@ export function getProjectSchemasDir(projectRoot: string): string { return path.join(projectRoot, 'openspec', 'schemas'); } +export function getRemoteSchemaDir( + name: string, + projectRoot: string, + projectConfig?: ProjectConfig | null +): string | null { + const config = + projectConfig === undefined ? readProjectConfig(projectRoot) : projectConfig; + const source = config?.schemaSources?.[name]; + if (!source) { + return null; + } + assertNoProjectSchemaConflict(projectRoot, name); + + let lock; + try { + lock = readSchemaLock(projectRoot); + } catch (error) { + throw new RemoteSchemaResolutionError( + 'remote_lock_invalid', + `${error instanceof Error ? error.message : String(error)}; run 'openspec schema sync'` + ); + } + const entry = lock?.schemas[name]; + if (!entry) { + throw new RemoteSchemaResolutionError( + 'remote_not_locked', + `Remote schema '${name}' is not locked; run 'openspec schema sync ${name}'` + ); + } + if ( + entry.git !== source.git || + entry.requestedRef !== source.ref || + entry.bundlePath !== source.path + ) { + throw new RemoteSchemaResolutionError( + 'remote_lock_mismatch', + `Remote schema '${name}' lock does not match openspec/config.yaml; run 'openspec schema sync ${name}'` + ); + } + + let cacheDir: string; + try { + cacheDir = verifyRemoteSchemaCache(entry.integrity); + } catch (error) { + throw new RemoteSchemaResolutionError( + 'remote_cache_invalid', + `Remote schema '${name}' cache is unavailable or corrupt: ${ + error instanceof Error ? error.message : String(error) + }; run 'openspec schema sync ${name} --locked'` + ); + } + try { + validateRemoteSchemaDirectory(cacheDir, name); + } catch (error) { + throw new RemoteSchemaResolutionError( + 'remote_cache_invalid', + `Remote schema '${name}' cache is invalid: ${ + error instanceof Error ? error.message : String(error) + }; run 'openspec schema sync ${name} --locked'` + ); + } + return cacheDir; +} + /** * Determines whether a directory entry represents a schema directory candidate. * @@ -113,9 +185,10 @@ function getSchemaCandidateDir(schemasDir: string, name: string): string | null * Resolves a schema name to its directory path. * * Resolution order (when projectRoot is provided): - * 1. Project-local: /openspec/schemas//schema.yaml - * 2. User override: ${XDG_DATA_HOME}/openspec/schemas//schema.yaml - * 3. Package built-in: /schemas//schema.yaml + * 1. A declared remote owns its name and conflicts with a same-named project schema + * 2. Otherwise, project-local: /openspec/schemas//schema.yaml + * 3. User override: ${XDG_DATA_HOME}/openspec/schemas//schema.yaml + * 4. Package built-in: /schemas//schema.yaml * * When projectRoot is not provided, only user override and package built-in are checked * (backward compatible behavior). @@ -126,7 +199,8 @@ function getSchemaCandidateDir(schemasDir: string, name: string): string | null */ export function getSchemaDir( name: string, - projectRoot?: string + projectRoot?: string, + projectConfig?: ProjectConfig | null ): string | null { if ( name.length === 0 || @@ -142,19 +216,31 @@ export function getSchemaDir( // 1. Check project-local directory (if projectRoot provided) if (projectRoot) { + const config = + projectConfig === undefined ? readProjectConfig(projectRoot) : projectConfig; + const declaredRemote = config?.schemaSources?.[name]; + if (declaredRemote) { + return getRemoteSchemaDir(name, projectRoot, config); + } + const projectDir = getSchemaCandidateDir(getProjectSchemasDir(projectRoot), name); if (projectDir) { return projectDir; } + + const remoteDir = getRemoteSchemaDir(name, projectRoot, config); + if (remoteDir) { + return remoteDir; + } } - // 2. Check user override directory + // 3. Check user override directory const userDir = getSchemaCandidateDir(getUserSchemasDir(), name); if (userDir) { return userDir; } - // 3. Check package built-in directory + // 4. Check package built-in directory const packageDir = getSchemaCandidateDir(getPackageSchemasDir(), name); if (packageDir) { return packageDir; @@ -167,9 +253,10 @@ export function getSchemaDir( * Resolves a schema name to a SchemaYaml object. * * Resolution order (when projectRoot is provided): - * 1. Project-local: /openspec/schemas//schema.yaml - * 2. User override: ${XDG_DATA_HOME}/openspec/schemas//schema.yaml - * 3. Package built-in: /schemas//schema.yaml + * 1. A declared remote owns its name and conflicts with a same-named project schema + * 2. Otherwise, project-local: /openspec/schemas//schema.yaml + * 3. User override: ${XDG_DATA_HOME}/openspec/schemas//schema.yaml + * 4. Package built-in: /schemas//schema.yaml * * When projectRoot is not provided, only user override and package built-in are checked * (backward compatible behavior). @@ -179,13 +266,17 @@ export function getSchemaDir( * @returns The resolved schema object * @throws Error if schema is not found in any location */ -export function resolveSchema(name: string, projectRoot?: string): SchemaYaml { +export function resolveSchema( + name: string, + projectRoot?: string, + projectConfig?: ProjectConfig | null +): SchemaYaml { // Normalize name (remove .yaml extension if provided) const normalizedName = name.replace(/\.ya?ml$/, ''); - const schemaDir = getSchemaDir(normalizedName, projectRoot); + const schemaDir = getSchemaDir(normalizedName, projectRoot, projectConfig); if (!schemaDir) { - const availableSchemas = listSchemas(projectRoot); + const availableSchemas = listSchemas(projectRoot, projectConfig); throw new Error( `Schema '${normalizedName}' not found. Available schemas: ${availableSchemas.join(', ')}` ); @@ -231,7 +322,10 @@ export function resolveSchema(name: string, projectRoot?: string): SchemaYaml { * * @param projectRoot - Optional project root directory for project-local schema resolution */ -export function listSchemas(projectRoot?: string): string[] { +export function listSchemas( + projectRoot?: string, + projectConfig?: ProjectConfig | null +): string[] { const schemas = new Set(); // Add package built-in schemas @@ -262,6 +356,15 @@ export function listSchemas(projectRoot?: string): string[] { // Add project-local schemas (if projectRoot provided) if (projectRoot) { + const config = + projectConfig === undefined ? readProjectConfig(projectRoot) : projectConfig; + const remoteNames = Object.keys( + config?.schemaSources ?? {} + ); + for (const name of remoteNames) { + schemas.add(name); + } + const projectDir = getProjectSchemasDir(projectRoot); if (fs.existsSync(projectDir)) { for (const entry of fs.readdirSync(projectDir, { withFileTypes: true })) { @@ -285,7 +388,14 @@ export interface SchemaInfo { name: string; description: string; artifacts: string[]; - source: 'project' | 'user' | 'package'; + source: 'project' | 'remote' | 'user' | 'package'; + available?: boolean; + error?: string; + status?: Array<{ + level: 'error'; + code: string; + message: string; + }>; } /** @@ -294,18 +404,30 @@ export interface SchemaInfo { * * @param projectRoot - Optional project root directory for project-local schema resolution */ -export function listSchemasWithInfo(projectRoot?: string): SchemaInfo[] { +export function listSchemasWithInfo( + projectRoot?: string, + projectConfig?: ProjectConfig | null +): SchemaInfo[] { const schemas: SchemaInfo[] = []; const seenNames = new Set(); + const config = projectRoot + ? projectConfig === undefined + ? readProjectConfig(projectRoot) + : projectConfig + : null; + const remoteNames = projectRoot + ? Object.keys(config?.schemaSources ?? {}).sort() + : []; + const remoteNameSet = new Set(remoteNames); - // Add project-local schemas first (highest priority, if projectRoot provided) + // Add unclaimed project-local schemas first (if projectRoot provided). if (projectRoot) { const projectDir = getProjectSchemasDir(projectRoot); if (fs.existsSync(projectDir)) { for (const entry of fs.readdirSync(projectDir, { withFileTypes: true })) { if (isSchemaDir(projectDir, entry)) { const schemaPath = path.join(projectDir, entry.name, 'schema.yaml'); - if (fs.existsSync(schemaPath)) { + if (fs.existsSync(schemaPath) && !remoteNameSet.has(entry.name)) { try { const schema = parseSchema(fs.readFileSync(schemaPath, 'utf-8')); schemas.push({ @@ -322,6 +444,40 @@ export function listSchemasWithInfo(projectRoot?: string): SchemaInfo[] { } } } + + for (const name of remoteNames) { + if (seenNames.has(name)) continue; + try { + const remoteDir = getRemoteSchemaDir(name, projectRoot, config); + if (!remoteDir) continue; + const schema = parseSchema( + fs.readFileSync(path.join(remoteDir, 'schema.yaml'), 'utf8') + ); + schemas.push({ + name, + description: schema.description || '', + artifacts: schema.artifacts.map((artifact) => artifact.id), + source: 'remote', + }); + seenNames.add(name); + } catch (error) { + const code = + error instanceof RemoteSchemaResolutionError + ? error.code + : 'remote_schema_unavailable'; + const message = error instanceof Error ? error.message : String(error); + schemas.push({ + name, + description: '', + artifacts: [], + source: 'remote', + available: false, + error: message, + status: [{ level: 'error', code, message }], + }); + seenNames.add(name); + } + } } // Add user override schemas (if not overridden by project) diff --git a/src/core/artifact-graph/schema-directory.ts b/src/core/artifact-graph/schema-directory.ts new file mode 100644 index 0000000000..a85e7e9e92 --- /dev/null +++ b/src/core/artifact-graph/schema-directory.ts @@ -0,0 +1,204 @@ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { normalizeBundlePath } from '../remote-schema/bundle.js'; +import { FileSystemUtils } from '../../utils/file-system.js'; +import { inspectSchema } from './schema.js'; +import type { SchemaYaml } from './types.js'; + +export interface SchemaDirectoryOptions { + expectedName?: string; + requireTemplatesDirectory?: boolean; +} + +export interface ValidatedSchemaDirectory { + schema: SchemaYaml; + templatePaths: Record; +} + +export interface SchemaDirectoryIssue { + path: string; + message: string; +} + +export interface InspectedSchemaDirectory { + schema?: SchemaYaml; + templatePaths: Record; + issues: SchemaDirectoryIssue[]; +} + +export class SchemaDirectoryValidationError extends Error { + constructor(public readonly issues: SchemaDirectoryIssue[]) { + super( + issues.map((issue) => issue.message).join('; ') || 'Schema validation failed' + ); + this.name = 'SchemaDirectoryValidationError'; + } +} + +export function validateSchemaDirectory( + schemaDir: string, + options: SchemaDirectoryOptions = {} +): ValidatedSchemaDirectory { + return options.expectedName !== undefined || options.requireTemplatesDirectory + ? validateRemoteSchemaDirectory( + schemaDir, + options.expectedName + ) + : validateLocalSchemaDirectory(schemaDir); +} + +function validatedResult( + inspection: InspectedSchemaDirectory +): ValidatedSchemaDirectory { + if (!inspection.schema || inspection.issues.length > 0) { + throw new SchemaDirectoryValidationError(inspection.issues); + } + return { + schema: inspection.schema, + templatePaths: inspection.templatePaths, + }; +} + +export function validateLocalSchemaDirectory( + schemaDir: string +): ValidatedSchemaDirectory { + return validatedResult(inspectLocalSchemaDirectory(schemaDir)); +} + +export function validateRemoteSchemaDirectory( + schemaDir: string, + expectedName?: string +): ValidatedSchemaDirectory { + return validatedResult(inspectRemoteSchemaDirectory(schemaDir, expectedName)); +} + +export function inspectSchemaDirectory( + schemaDir: string, + options: SchemaDirectoryOptions = {} +): InspectedSchemaDirectory { + return options.expectedName !== undefined || options.requireTemplatesDirectory + ? inspectRemoteSchemaDirectory(schemaDir, options.expectedName) + : inspectLocalSchemaDirectory(schemaDir); +} + +export function inspectLocalSchemaDirectory( + schemaDir: string +): InspectedSchemaDirectory { + return inspectSchemaDirectoryWithMode(schemaDir, 'legacy-local'); +} + +export function inspectRemoteSchemaDirectory( + schemaDir: string, + expectedName?: string +): InspectedSchemaDirectory { + return inspectSchemaDirectoryWithMode(schemaDir, 'remote-bundle', expectedName); +} + +function inspectSchemaDirectoryWithMode( + schemaDir: string, + mode: 'legacy-local' | 'remote-bundle', + expectedName?: string +): InspectedSchemaDirectory { + const issues: SchemaDirectoryIssue[] = []; + const templatePaths: Record = {}; + const schemaPath = path.join(schemaDir, 'schema.yaml'); + if (!fs.existsSync(schemaPath)) { + return { + templatePaths, + issues: [{ path: 'schema.yaml', message: `schema.yaml not found in '${schemaDir}'` }], + }; + } + if (mode === 'remote-bundle') { + const schemaStat = fs.lstatSync(schemaPath); + if (schemaStat.isSymbolicLink() || !schemaStat.isFile()) { + return { + templatePaths, + issues: [{ + path: 'schema.yaml', + message: `schema.yaml must be a regular file in '${schemaDir}'`, + }], + }; + } + } + + const schemaInspection = inspectSchema(fs.readFileSync(schemaPath, 'utf8')); + issues.push(...schemaInspection.issues); + const schema = schemaInspection.schema; + if (!schema) { + return { templatePaths, issues }; + } + if (expectedName !== undefined && schema.name !== expectedName) { + issues.push({ + path: 'schema.yaml', + message: `Remote schema bundle was declared as '${expectedName}' but schema.yaml name is '${schema.name}'`, + }); + } + + const templatesDir = path.join(schemaDir, 'templates'); + if (mode === 'remote-bundle') { + if (!fs.existsSync(templatesDir)) { + issues.push({ + path: 'templates', + message: `templates directory not found in '${schemaDir}'`, + }); + } else { + const templatesStat = fs.lstatSync(templatesDir); + if (templatesStat.isSymbolicLink() || !templatesStat.isDirectory()) { + issues.push({ + path: 'templates', + message: `templates must be a real directory in '${schemaDir}'`, + }); + } + } + } + + for (const artifact of schema.artifacts) { + let normalizedTemplate = artifact.template; + if (mode === 'remote-bundle') { + try { + normalizedTemplate = normalizeBundlePath(artifact.template); + } catch { + issues.push({ + path: `schema.yaml:artifacts.${artifact.id}.template`, + message: `Artifact '${artifact.id}' has unsafe template path '${artifact.template}'`, + }); + continue; + } + } + + const templateSegments = + mode === 'remote-bundle' ? normalizedTemplate.split('/') : [artifact.template]; + const templatePath = path.join(templatesDir, ...templateSegments); + if (!fs.existsSync(templatePath)) { + issues.push({ + path: `templates/${normalizedTemplate}`, + message: `Template file '${artifact.template}' not found for artifact '${artifact.id}'`, + }); + continue; + } + + if (mode === 'legacy-local') { + try { + FileSystemUtils.assertPathWithin(templatesDir, templatePath); + } catch { + issues.push({ + path: `artifacts.${artifact.id}.template`, + message: `Template '${artifact.template}' for artifact '${artifact.id}' resolves outside the schema templates directory`, + }); + continue; + } + } else { + const stat = fs.lstatSync(templatePath); + if (!stat.isFile() || stat.isSymbolicLink()) { + issues.push({ + path: `templates/${normalizedTemplate}`, + message: `Template file '${artifact.template}' not found for artifact '${artifact.id}'`, + }); + continue; + } + } + templatePaths[artifact.id] = templatePath; + } + + return { schema, templatePaths, issues }; +} diff --git a/src/core/artifact-graph/schema.ts b/src/core/artifact-graph/schema.ts index 6371745e28..1644012804 100644 --- a/src/core/artifact-graph/schema.ts +++ b/src/core/artifact-graph/schema.ts @@ -9,6 +9,17 @@ export class SchemaValidationError extends Error { } } +export interface SchemaIssue { + path: string; + message: string; +} + +export interface SchemaInspection { + schema?: SchemaYaml; + issues: SchemaIssue[]; + parseError?: Error; +} + /** * Loads and validates an artifact schema from a YAML file. */ @@ -21,64 +32,118 @@ export function loadSchema(filePath: string): SchemaYaml { * Parses and validates an artifact schema from YAML content. */ export function parseSchema(yamlContent: string): SchemaYaml { - const parsed = parseYaml(yamlContent); + const inspection = inspectSchema(yamlContent); + if (inspection.parseError) { + throw inspection.parseError; + } + if (!inspection.schema || inspection.issues.length > 0) { + throw new SchemaValidationError( + inspection.issues.map((issue) => issue.message).join(', ') + ); + } + return inspection.schema; +} + +/** + * Parse a schema and collect independent validation failures for diagnostic + * commands. Callers that need fail-fast behavior should use parseSchema(). + */ +export function inspectSchema(yamlContent: string): SchemaInspection { + let parsed: unknown; + try { + parsed = parseYaml(yamlContent); + } catch (error) { + const parseError = error instanceof Error ? error : new Error(String(error)); + return { + issues: [{ + path: 'schema.yaml', + message: `Invalid schema YAML: ${parseError.message}`, + }], + parseError, + }; + } - // Validate with Zod const result = SchemaYamlSchema.safeParse(parsed); if (!result.success) { - const errors = result.error.issues.map(e => `${e.path.join('.')}: ${e.message}`).join(', '); - throw new SchemaValidationError(`Invalid schema: ${errors}`); + return { + issues: result.error.issues.map((issue) => ({ + path: issue.path.length > 0 ? `schema.yaml:${issue.path.join('.')}` : 'schema.yaml', + message: `Invalid schema: ${issue.path.join('.')}: ${issue.message}`, + })), + }; } const schema = result.data; + const issues: SchemaIssue[] = []; + + const duplicateIds = findDuplicateIds(schema.artifacts); + for (const id of duplicateIds) { + issues.push({ + path: 'schema.yaml', + message: `Duplicate artifact ID: ${id}`, + }); + } - // Check for duplicate artifact IDs - validateNoDuplicateIds(schema.artifacts); - - // Check that all requires references are valid - validateRequiresReferences(schema.artifacts); + const invalidReferences = findInvalidRequiresReferences(schema.artifacts); + for (const { artifactId, requiredId } of invalidReferences) { + issues.push({ + path: 'schema.yaml', + message: `Invalid dependency reference in artifact '${artifactId}': '${requiredId}' does not exist`, + }); + } - // Check for cycles - validateNoCycles(schema.artifacts); + if (duplicateIds.length === 0 && invalidReferences.length === 0) { + const cycle = findCycle(schema.artifacts); + if (cycle) { + issues.push({ + path: 'schema.yaml', + message: `Cyclic dependency detected: ${cycle}`, + }); + } + } - return schema; + return { schema, issues }; } /** * Validates that there are no duplicate artifact IDs. */ -function validateNoDuplicateIds(artifacts: Artifact[]): void { +function findDuplicateIds(artifacts: Artifact[]): string[] { const seen = new Set(); + const duplicates = new Set(); for (const artifact of artifacts) { if (seen.has(artifact.id)) { - throw new SchemaValidationError(`Duplicate artifact ID: ${artifact.id}`); + duplicates.add(artifact.id); } seen.add(artifact.id); } + return [...duplicates]; } /** * Validates that all `requires` references point to valid artifact IDs. */ -function validateRequiresReferences(artifacts: Artifact[]): void { +function findInvalidRequiresReferences( + artifacts: Artifact[] +): Array<{ artifactId: string; requiredId: string }> { const validIds = new Set(artifacts.map(a => a.id)); + const invalid: Array<{ artifactId: string; requiredId: string }> = []; for (const artifact of artifacts) { for (const req of artifact.requires) { if (!validIds.has(req)) { - throw new SchemaValidationError( - `Invalid dependency reference in artifact '${artifact.id}': '${req}' does not exist` - ); + invalid.push({ artifactId: artifact.id, requiredId: req }); } } } + return invalid; } /** * Validates that there are no cyclic dependencies. * Uses DFS to detect cycles and reports the full cycle path. */ -function validateNoCycles(artifacts: Artifact[]): void { +function findCycle(artifacts: Artifact[]): string | null { const artifactMap = new Map(artifacts.map(a => [a.id, a])); const visited = new Set(); const inStack = new Set(); @@ -117,8 +182,9 @@ function validateNoCycles(artifacts: Artifact[]): void { if (!visited.has(artifact.id)) { const cycle = dfs(artifact.id); if (cycle) { - throw new SchemaValidationError(`Cyclic dependency detected: ${cycle}`); + return cycle; } } } + return null; } diff --git a/src/core/completions/command-registry.ts b/src/core/completions/command-registry.ts index 67cd8d8172..23157cd56b 100644 --- a/src/core/completions/command-registry.ts +++ b/src/core/completions/command-registry.ts @@ -705,6 +705,20 @@ export const COMMAND_REGISTRY: CommandDefinition[] = [ description: 'Manage workflow schemas', flags: [], subcommands: [ + { + name: 'sync', + description: 'Synchronize project-declared Git schema sources', + acceptsPositional: true, + positionalType: 'schema-name', + positionals: [{ name: 'name', type: 'schema-name', optional: true }], + flags: [ + COMMON_FLAGS.json, + { + name: 'locked', + description: 'Restore exactly the commits and content in the lockfile', + }, + ], + }, { name: 'which', description: 'Show where a schema resolves from', diff --git a/src/core/completions/completion-provider.ts b/src/core/completions/completion-provider.ts index 0159131486..9c497ed082 100644 --- a/src/core/completions/completion-provider.ts +++ b/src/core/completions/completion-provider.ts @@ -1,5 +1,6 @@ import { getActiveChangeIds, getSpecIds } from '../../utils/item-discovery.js'; import { listSchemas } from '../artifact-graph/index.js'; +import { resolveSchemaConsumerRoot } from '../remote-schema/consumer-root.js'; /** * Cache entry for completion data @@ -97,7 +98,9 @@ export class CompletionProvider { } // Fetch fresh data - const schemaNames = listSchemas(this.projectRoot); + const schemaNames = listSchemas( + resolveSchemaConsumerRoot(this.projectRoot) ?? this.projectRoot + ); // Update cache this.schemaCache = { diff --git a/src/core/list.ts b/src/core/list.ts index f6b6faf2f8..afd07544f4 100644 --- a/src/core/list.ts +++ b/src/core/list.ts @@ -17,6 +17,7 @@ interface ListOptions { sort?: 'recent' | 'name'; json?: boolean; root?: RootOutput; + schemaRoot?: string; } function isMissingPathError(error: unknown): boolean { @@ -96,7 +97,7 @@ function formatRelativeTime(date: Date): string { export class ListCommand { async execute(targetPath: string = '.', mode: 'changes' | 'specs' = 'changes', options: ListOptions = {}): Promise { - const { sort = 'recent', json = false, root } = options; + const { sort = 'recent', json = false, root, schemaRoot = targetPath } = options; if (mode === 'changes') { const changesDir = path.join(targetPath, 'openspec', 'changes'); @@ -120,7 +121,12 @@ export class ListCommand { const changes: ChangeInfo[] = []; for (const changeDir of changeDirs) { - const progress = await getTaskProgressForChange(changesDir, changeDir, targetPath); + const progress = await getTaskProgressForChange( + changesDir, + changeDir, + targetPath, + schemaRoot + ); const changePath = path.join(changesDir, changeDir); const lastModified = await getLastModified(changePath); changes.push({ diff --git a/src/core/project-config.ts b/src/core/project-config.ts index 922e31505b..3506c0e1df 100644 --- a/src/core/project-config.ts +++ b/src/core/project-config.ts @@ -2,6 +2,8 @@ import { existsSync, readFileSync, statSync } from 'fs'; import path from 'path'; import { parse as parseYaml } from 'yaml'; import { z } from 'zod'; +import { parseSchemaSources } from './remote-schema/config.js'; +import type { GitSchemaSource } from './remote-schema/types.js'; export const OPERATION_IDS = ['apply', 'archive'] as const; export type OperationId = (typeof OPERATION_IDS)[number]; @@ -94,6 +96,7 @@ export interface DeclarationEntry { export type ProjectConfig = z.infer & { references?: DeclarationEntry[]; + schemaSources?: Record; }; export interface OperationInputs { @@ -363,6 +366,11 @@ export function readProjectConfig(projectRoot: string): ProjectConfig | null { config.references = references; } + const schemaSources = parseSchemaSources(raw.schemaSources, (message) => console.warn(message)); + if (schemaSources) { + config.schemaSources = schemaSources; + } + // Parse store pointer field: a string, or dropped with a warning. // (Root resolution does NOT use this parse — it uses readStorePointer // below, which errors on malformed pointers instead of dropping.) diff --git a/src/core/remote-schema/authority.ts b/src/core/remote-schema/authority.ts new file mode 100644 index 0000000000..de68e5dd37 --- /dev/null +++ b/src/core/remote-schema/authority.ts @@ -0,0 +1,53 @@ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { readProjectConfig } from '../project-config.js'; + +export type RemoteSchemaResolutionErrorCode = + | 'schema_name_conflict' + | 'remote_lock_invalid' + | 'remote_not_locked' + | 'remote_lock_mismatch' + | 'remote_cache_invalid'; + +export class RemoteSchemaResolutionError extends Error { + constructor( + public readonly code: RemoteSchemaResolutionErrorCode, + message: string + ) { + super(message); + this.name = 'RemoteSchemaResolutionError'; + } +} + +export function getProjectSchemaConflictPath( + projectRoot: string, + name: string +): string | null { + const schemaDir = path.join(projectRoot, 'openspec', 'schemas', name); + return fs.existsSync(path.join(schemaDir, 'schema.yaml')) ? schemaDir : null; +} + +export function assertNoProjectSchemaConflict( + projectRoot: string, + name: string +): void { + const conflictPath = getProjectSchemaConflictPath(projectRoot, name); + if (conflictPath) { + throw new RemoteSchemaResolutionError( + 'schema_name_conflict', + `Project-local schema '${name}' at '${conflictPath}' conflicts with declared remote schema '${name}'; rename the local schema or remove the remote declaration` + ); + } +} + +export function assertProjectSchemaNameUnclaimed( + projectRoot: string, + name: string +): void { + if (readProjectConfig(projectRoot)?.schemaSources?.[name]) { + throw new RemoteSchemaResolutionError( + 'schema_name_conflict', + `Cannot create project-local schema '${name}' because that name is declared by a remote schema source; choose a different name or remove the remote declaration` + ); + } +} diff --git a/src/core/remote-schema/bundle.ts b/src/core/remote-schema/bundle.ts new file mode 100644 index 0000000000..887a0464b2 --- /dev/null +++ b/src/core/remote-schema/bundle.ts @@ -0,0 +1,155 @@ +import { createHash } from 'node:crypto'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +export interface BundleLimits { + maxFiles: number; + maxBytes: number; +} + +export interface BundleIntegrity { + integrity: string; + fileCount: number; + totalBytes: number; +} + +export const MAX_SCHEMA_BUNDLE_FILES = 1_000; +export const MAX_SCHEMA_BUNDLE_BYTES = 10 * 1024 * 1024; + +const WINDOWS_RESERVED_SEGMENT = + /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\..*)?$/i; + +function invalidBundlePath(value: string, reason: string): never { + throw new Error(`Invalid schema bundle path '${value}': ${reason}`); +} + +export function normalizeBundlePath(value: string): string { + if ( + value.length === 0 || + /[\u0000-\u001f\u007f]/.test(value) || + value.includes('\\') || + path.posix.isAbsolute(value) || + path.win32.isAbsolute(value) || + /^[a-zA-Z]:/.test(value) + ) { + return invalidBundlePath(value, 'must be a repository-relative portable Git path'); + } + + const segments = value.split('/'); + for (const segment of segments) { + if ( + segment.length === 0 || + segment === '.' || + segment === '..' || + segment.endsWith('.') || + segment.endsWith(' ') || + segment.includes(':') || + /[<>"|?*]/.test(segment) || + WINDOWS_RESERVED_SEGMENT.test(segment) + ) { + return invalidBundlePath(value, `unsafe path segment '${segment}'`); + } + } + return segments.join('/'); +} + +export function assertPortableBundleEntries(entries: string[]): void { + const seen = new Map(); + for (const entry of entries) { + const normalized = normalizeBundlePath(entry); + const segments = normalized.split('/'); + for (let index = 0; index < segments.length; index++) { + const originalPrefix = segments.slice(0, index + 1).join('/'); + const portableKey = originalPrefix.normalize('NFC').toLocaleLowerCase('en-US'); + const previous = seen.get(portableKey); + if (previous !== undefined && previous !== originalPrefix) { + throw new Error( + `Remote schema bundle has a portable path collision between '${previous}' and '${originalPrefix}'` + ); + } + seen.set(portableKey, originalPrefix); + } + } +} + +export function computeBundleIntegrity( + bundleDir: string, + limits: BundleLimits = { + maxFiles: MAX_SCHEMA_BUNDLE_FILES, + maxBytes: MAX_SCHEMA_BUNDLE_BYTES, + } +): BundleIntegrity { + const rootStat = fs.lstatSync(bundleDir); + if (rootStat.isSymbolicLink()) { + throw new Error('Remote schema bundle root must not be a symbolic link'); + } + if (!rootStat.isDirectory()) { + throw new Error('Remote schema bundle root must be a directory'); + } + + const files: Array<{ relativePath: string; content: Buffer }> = []; + let totalBytes = 0; + const visit = (currentDir: string, relativeDir: string): void => { + const entries = fs + .readdirSync(currentDir, { withFileTypes: true }) + .sort((left, right) => + Buffer.compare(Buffer.from(left.name, 'utf8'), Buffer.from(right.name, 'utf8')) + ); + for (const entry of entries) { + const absolutePath = path.join(currentDir, entry.name); + const relativePath = relativeDir + ? `${relativeDir}/${entry.name}` + : entry.name; + if (entry.isSymbolicLink()) { + throw new Error(`Remote schema bundle contains symbolic link '${relativePath}'`); + } + if (entry.isDirectory()) { + visit(absolutePath, relativePath); + continue; + } + if (!entry.isFile()) { + throw new Error(`Remote schema bundle contains non-regular file '${relativePath}'`); + } + if (files.length + 1 > limits.maxFiles) { + throw new Error(`Remote schema bundle contains more than ${limits.maxFiles} files`); + } + const size = fs.statSync(absolutePath).size; + if (totalBytes + size > limits.maxBytes) { + throw new Error(`Remote schema bundle contains more than ${limits.maxBytes} bytes`); + } + const content = fs.readFileSync(absolutePath); + if (totalBytes + content.length > limits.maxBytes) { + throw new Error(`Remote schema bundle contains more than ${limits.maxBytes} bytes`); + } + files.push({ relativePath, content }); + totalBytes += content.length; + } + }; + visit(bundleDir, ''); + + const relativePaths = files.map((file) => file.relativePath); + assertPortableBundleEntries(relativePaths); + const hash = createHash('sha256'); + for (const file of files.sort((left, right) => + Buffer.compare( + Buffer.from(left.relativePath, 'utf8'), + Buffer.from(right.relativePath, 'utf8') + ) + )) { + const pathBytes = Buffer.from(file.relativePath, 'utf8'); + const pathLength = Buffer.allocUnsafe(4); + pathLength.writeUInt32BE(pathBytes.length); + const contentLength = Buffer.allocUnsafe(8); + contentLength.writeBigUInt64BE(BigInt(file.content.length)); + hash.update(pathLength); + hash.update(pathBytes); + hash.update(contentLength); + hash.update(file.content); + } + + return { + integrity: `sha256:${hash.digest('hex')}`, + fileCount: files.length, + totalBytes, + }; +} diff --git a/src/core/remote-schema/cache.ts b/src/core/remote-schema/cache.ts new file mode 100644 index 0000000000..a6ce7caf9a --- /dev/null +++ b/src/core/remote-schema/cache.ts @@ -0,0 +1,114 @@ +import { randomUUID } from 'node:crypto'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { getGlobalDataDir } from '../global-config.js'; +import { computeBundleIntegrity } from './bundle.js'; + +const INTEGRITY_PATTERN = /^sha256:([0-9a-f]{64})$/; + +function integrityHash(integrity: string): string { + const match = INTEGRITY_PATTERN.exec(integrity); + if (!match) { + throw new Error('Remote schema cache requires a valid SHA-256 integrity value'); + } + return match[1]; +} + +export function getRemoteSchemaCacheDir( + integrity: string, + globalDataDir = getGlobalDataDir() +): string { + return path.join( + globalDataDir, + 'schema-cache', + 'v1', + 'sha256', + integrityHash(integrity) + ); +} + +export function verifyRemoteSchemaCache( + integrity: string, + globalDataDir = getGlobalDataDir() +): string { + const cacheDir = getRemoteSchemaCacheDir(integrity, globalDataDir); + if (!fs.existsSync(cacheDir)) { + throw new Error( + `Remote schema cache is missing for ${integrity}; run 'openspec schema sync'` + ); + } + const actual = computeBundleIntegrity(cacheDir).integrity; + if (actual !== integrity) { + throw new Error( + `Remote schema cache at '${cacheDir}' does not match its locked integrity; run 'openspec schema sync --locked'` + ); + } + return cacheDir; +} + +export function installRemoteSchemaCache( + sourceDir: string, + integrity: string, + globalDataDir = getGlobalDataDir() +): string { + const actual = computeBundleIntegrity(sourceDir).integrity; + if (actual !== integrity) { + throw new Error('Remote schema bundle does not match the expected integrity'); + } + + const cacheDir = getRemoteSchemaCacheDir(integrity, globalDataDir); + if (fs.existsSync(cacheDir)) { + try { + return verifyRemoteSchemaCache(integrity, globalDataDir); + } catch { + // A verified extraction can repair the content-addressed entry below. + } + } + + const parentDir = path.dirname(cacheDir); + fs.mkdirSync(parentDir, { recursive: true }); + const tempDir = path.join(parentDir, `.install-${process.pid}-${randomUUID()}`); + const displacedDir = path.join( + parentDir, + `.displaced-${process.pid}-${randomUUID()}` + ); + let displaced = false; + try { + fs.cpSync(sourceDir, tempDir, { + recursive: true, + errorOnExist: true, + force: false, + verbatimSymlinks: true, + }); + if (computeBundleIntegrity(tempDir).integrity !== integrity) { + throw new Error('Remote schema cache copy failed integrity verification'); + } + if (fs.existsSync(cacheDir)) { + fs.renameSync(cacheDir, displacedDir); + displaced = true; + } + try { + fs.renameSync(tempDir, cacheDir); + } catch (error) { + if (!displaced && fs.existsSync(cacheDir)) { + return verifyRemoteSchemaCache(integrity, globalDataDir); + } + throw error; + } + const verified = verifyRemoteSchemaCache(integrity, globalDataDir); + if (displaced) { + fs.rmSync(displacedDir, { recursive: true, force: true }); + displaced = false; + } + return verified; + } catch (error) { + if (displaced) { + fs.rmSync(cacheDir, { recursive: true, force: true }); + fs.renameSync(displacedDir, cacheDir); + displaced = false; + } + throw error; + } finally { + fs.rmSync(tempDir, { recursive: true, force: true }); + } +} diff --git a/src/core/remote-schema/config.ts b/src/core/remote-schema/config.ts new file mode 100644 index 0000000000..016ccced95 --- /dev/null +++ b/src/core/remote-schema/config.ts @@ -0,0 +1,102 @@ +import type { GitSchemaSource } from './types.js'; + +const SCHEMA_NAME_PATTERN = /^[a-z][a-z0-9]*(-[a-z0-9]+)*$/; + +export function isValidSchemaSourceName(name: string): boolean { + return ( + name !== '__proto__' && + name !== 'prototype' && + name !== 'constructor' && + SCHEMA_NAME_PATTERN.test(name) + ); +} + +export type GitSourceValidation = 'valid' | 'credentials' | 'transport'; + +export function validateGitSource(value: string): GitSourceValidation { + if ( + value.length === 0 || + value.startsWith('-') || + /[\u0000-\u001f\u007f]/.test(value) || + /^[a-z]:[\\/]/i.test(value) || + /^[a-z][a-z0-9+.-]*::/i.test(value) + ) { + return 'transport'; + } + if ( + !value.includes('://') && + /^(?:[^/@:\s]+@)?[^/:\s]+:.+$/.test(value) + ) { + return 'valid'; + } + try { + const url = new URL(value); + if (!['https:', 'ssh:', 'file:'].includes(url.protocol)) { + return 'transport'; + } + if ( + (url.protocol === 'https:' && (url.username.length > 0 || url.password.length > 0)) || + url.password.length > 0 + ) { + return 'credentials'; + } + return 'valid'; + } catch { + return 'transport'; + } +} + +export function parseSchemaSources( + raw: unknown, + warn: (message: string) => void +): Record | undefined { + if (raw === undefined) { + return undefined; + } + if (!raw || typeof raw !== 'object' || Array.isArray(raw)) { + warn("Invalid 'schemaSources' field in config (must be an object)"); + return undefined; + } + + const sources: Record = {}; + for (const [name, value] of Object.entries(raw)) { + if (!isValidSchemaSourceName(name)) { + warn(`Invalid schema source name '${name}' (must be kebab-case)`); + continue; + } + if (!value || typeof value !== 'object' || Array.isArray(value)) { + warn(`Invalid schema source '${name}' (must be an object)`); + continue; + } + const candidate = value as Record; + let valid = true; + for (const field of ['git', 'ref', 'path'] as const) { + if (typeof candidate[field] !== 'string' || candidate[field].length === 0) { + warn(`Invalid '${field}' for schema source '${name}' (must be a non-empty string)`); + valid = false; + } + } + if (!valid) { + continue; + } + const git = candidate.git as string; + const gitValidation = validateGitSource(git); + if (gitValidation === 'credentials') { + warn(`Credentials are not allowed in Git URL for schema source '${name}'`); + continue; + } + if (gitValidation === 'transport') { + warn( + `Unsupported Git source for schema source '${name}' (use HTTPS, SSH, scp-style SSH, or file URL)` + ); + continue; + } + sources[name] = { + git, + ref: candidate.ref as string, + path: candidate.path as string, + }; + } + + return Object.keys(sources).length > 0 ? sources : undefined; +} diff --git a/src/core/remote-schema/consumer-root.ts b/src/core/remote-schema/consumer-root.ts new file mode 100644 index 0000000000..d3d5ea8371 --- /dev/null +++ b/src/core/remote-schema/consumer-root.ts @@ -0,0 +1,5 @@ +import { findRepoPlanningRootSync } from '../planning-home.js'; + +export function resolveSchemaConsumerRoot(startPath: string): string | null { + return findRepoPlanningRootSync(startPath); +} diff --git a/src/core/remote-schema/git.ts b/src/core/remote-schema/git.ts new file mode 100644 index 0000000000..60ac5219d1 --- /dev/null +++ b/src/core/remote-schema/git.ts @@ -0,0 +1,350 @@ +import { execFile } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { + assertPortableBundleEntries, + computeBundleIntegrity, + MAX_SCHEMA_BUNDLE_BYTES, + MAX_SCHEMA_BUNDLE_FILES, + normalizeBundlePath, + type BundleIntegrity, +} from './bundle.js'; +import { validateGitSource } from './config.js'; + +export interface FetchSchemaBundleOptions { + git: string; + requestedRef: string; + lockedCommit?: string; + bundlePath: string; + destinationDir: string; + timeoutMs?: number; +} + +export interface FetchSchemaBundleResult extends BundleIntegrity { + resolvedCommit: string; +} + +interface GitTreeEntry { + mode: string; + type: string; + objectId: string; + relativePath: string; +} + +const DEFAULT_GIT_TIMEOUT_MS = 120_000; +const GIT_OUTPUT_LIMIT = MAX_SCHEMA_BUNDLE_BYTES + 1024 * 1024; + +function removeSshOption(command: string, option: string): string { + const assignment = `${option}(?:\\s*=\\s*|\\s+)`; + return command + .replace( + new RegExp( + `(^|\\s)-o\\s*(?:"${assignment}[^"]*"|'${assignment}[^']*'|${assignment}(?:"[^"]*"|'[^']*'|\\S+))`, + 'gi' + ), + '$1' + ) + .trim(); +} + +function hasSshOption(command: string, option: string): boolean { + const assignment = `${option}(?:\\s*=\\s*|\\s+)`; + return new RegExp( + `(^|\\s)-o\\s*(?:"${assignment}[^"]*"|'${assignment}[^']*'|${assignment}(?:"[^"]*"|'[^']*'|\\S+))`, + 'i' + ).test(command); +} + +export function buildNonInteractiveGitEnvironment( + environment: NodeJS.ProcessEnv = process.env +): NodeJS.ProcessEnv { + const originalSshCommand = + environment.GIT_SSH_COMMAND?.trim() || 'ssh'; + const hasHostKeyPolicy = hasSshOption( + originalSshCommand, + 'StrictHostKeyChecking' + ); + let sshCommand = originalSshCommand; + sshCommand = removeSshOption(sshCommand, 'BatchMode'); + const defaultHostKeyPolicy = hasHostKeyPolicy + ? '' + : ' -o StrictHostKeyChecking=accept-new'; + return { + ...environment, + GIT_TERMINAL_PROMPT: '0', + GIT_SSH_COMMAND: + `${sshCommand} -o BatchMode=yes${defaultHostKeyPolicy}`, + }; +} + +function runGit( + cwd: string, + args: string[], + operation: string, + timeoutMs: number, + input?: Buffer +): Promise { + return new Promise((resolve, reject) => { + const child = execFile( + 'git', + args, + { + cwd, + env: buildNonInteractiveGitEnvironment(), + encoding: 'buffer', + maxBuffer: GIT_OUTPUT_LIMIT, + timeout: timeoutMs, + windowsHide: true, + }, + (error, stdout) => { + if (error) { + const credentialHint = + operation === 'fetch' + ? '; check the ref and system Git SSH/credential-helper access' + : ''; + reject( + new Error( + `Git ${operation} failed while synchronizing remote schema${credentialHint}` + ) + ); + return; + } + resolve(Buffer.isBuffer(stdout) ? stdout : Buffer.from(stdout)); + } + ); + if (input !== undefined) { + child.stdin?.end(input); + } + }); +} + +export async function verifyLockedCommitIsAncestor( + repositoryDir: string, + lockedCommit: string, + fetchedRef: string, + timeoutMs: number +): Promise { + let verifiedCommit: string; + try { + verifiedCommit = ( + await runGit( + repositoryDir, + ['rev-parse', `${lockedCommit}^{commit}`], + 'locked commit presence verification', + timeoutMs + ) + ).toString('utf8').trim(); + } catch { + throw new Error( + 'Locked commit verification failed: the commit is not present after fetching the requested ref' + ); + } + + try { + await runGit( + repositoryDir, + ['merge-base', '--is-ancestor', verifiedCommit, fetchedRef], + 'locked commit ancestry verification', + timeoutMs + ); + } catch { + throw new Error( + 'Locked commit verification failed: the commit is not reachable from the requested ref' + ); + } + return verifiedCommit; +} + +function parseTreeEntries(output: Buffer, bundlePath: string): GitTreeEntry[] { + const decoded = output.toString('utf8'); + if (!Buffer.from(decoded, 'utf8').equals(output)) { + throw new Error('Remote schema bundle contains a non-UTF-8 Git path'); + } + const prefix = `${bundlePath}/`; + const records = decoded.split('\0').filter(Boolean); + const entries: GitTreeEntry[] = []; + for (const record of records) { + const tabIndex = record.indexOf('\t'); + if (tabIndex < 0) { + throw new Error('Git returned an invalid tree entry for remote schema bundle'); + } + const [mode, type, objectId] = record.slice(0, tabIndex).split(' '); + const repositoryPath = record.slice(tabIndex + 1); + if (!mode || !type || !objectId || !repositoryPath.startsWith(prefix)) { + throw new Error('Git returned a tree entry outside the selected remote schema bundle'); + } + const relativePath = repositoryPath.slice(prefix.length); + if (mode === '120000') { + throw new Error(`Remote schema bundle contains symbolic link '${relativePath}'`); + } + if (mode === '160000' || type === 'commit') { + throw new Error(`Remote schema bundle contains Git submodule '${relativePath}'`); + } + if (!mode.startsWith('100') || type !== 'blob') { + throw new Error(`Remote schema bundle contains unsupported entry '${relativePath}'`); + } + entries.push({ mode, type, objectId, relativePath }); + } + if (entries.length === 0) { + throw new Error(`Remote schema bundle path '${bundlePath}' contains no tracked files`); + } + if (entries.length > MAX_SCHEMA_BUNDLE_FILES) { + throw new Error(`Remote schema bundle contains more than ${MAX_SCHEMA_BUNDLE_FILES} files`); + } + assertPortableBundleEntries(entries.map((entry) => entry.relativePath)); + return entries; +} + +function parseBatchBlobs( + output: Buffer, + entries: GitTreeEntry[] +): Array { + const blobs: Array = []; + let offset = 0; + for (const entry of entries) { + const headerEnd = output.indexOf(0x0a, offset); + if (headerEnd < 0) { + throw new Error('Git returned an incomplete blob batch for remote schema bundle'); + } + const header = output.subarray(offset, headerEnd).toString('ascii'); + const [objectId, type, sizeText] = header.split(' '); + const size = Number(sizeText); + if ( + objectId !== entry.objectId || + type !== 'blob' || + !Number.isSafeInteger(size) || + size < 0 + ) { + throw new Error('Git returned an invalid blob batch for remote schema bundle'); + } + const contentStart = headerEnd + 1; + const contentEnd = contentStart + size; + if (contentEnd >= output.length || output[contentEnd] !== 0x0a) { + throw new Error('Git returned an incomplete blob batch for remote schema bundle'); + } + blobs.push({ ...entry, content: output.subarray(contentStart, contentEnd) }); + offset = contentEnd + 1; + } + if (offset !== output.length) { + throw new Error('Git returned unexpected data after the remote schema blob batch'); + } + return blobs; +} + +export async function fetchSchemaBundleFromGit( + options: FetchSchemaBundleOptions +): Promise { + const bundlePath = normalizeBundlePath(options.bundlePath); + const timeoutMs = options.timeoutMs ?? DEFAULT_GIT_TIMEOUT_MS; + if (validateGitSource(options.git) !== 'valid') { + throw new Error( + 'Invalid remote schema Git source; use a credential-free HTTPS, SSH, scp-style SSH, or file URL' + ); + } + if ( + options.requestedRef.length === 0 || + options.requestedRef.startsWith('-') || + /[\u0000-\u001f\u007f]/.test(options.requestedRef) + ) { + throw new Error('Invalid remote schema ref'); + } + if ( + options.lockedCommit !== undefined && + !/^[0-9a-f]{40}$/.test(options.lockedCommit) + ) { + throw new Error('Locked remote schema commit must be a 40-character hexadecimal SHA'); + } + + const repositoryDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-schema-fetch-')); + try { + await runGit(repositoryDir, ['init', '--quiet'], 'initialization', timeoutMs); + await runGit( + repositoryDir, + ['remote', 'add', 'origin', options.git], + 'remote setup', + timeoutMs + ); + await runGit( + repositoryDir, + options.lockedCommit + ? ['fetch', '--quiet', '--no-tags', 'origin', options.requestedRef] + : ['fetch', '--quiet', '--depth=1', '--no-tags', 'origin', options.requestedRef], + 'fetch', + timeoutMs + ); + let resolvedCommit: string; + if (options.lockedCommit) { + resolvedCommit = await verifyLockedCommitIsAncestor( + repositoryDir, + options.lockedCommit, + 'FETCH_HEAD^{commit}', + timeoutMs + ); + } else { + resolvedCommit = ( + await runGit( + repositoryDir, + ['rev-parse', 'FETCH_HEAD^{commit}'], + 'commit resolution', + timeoutMs + ) + ).toString('utf8').trim(); + } + if (!/^[0-9a-f]{40}$/.test(resolvedCommit)) { + throw new Error('Git did not resolve the remote schema ref to a commit SHA'); + } + if (options.lockedCommit !== undefined && resolvedCommit !== options.lockedCommit) { + throw new Error('Fetched remote schema commit does not match the lockfile'); + } + + const treeOutput = await runGit( + repositoryDir, + ['ls-tree', '-r', '-z', resolvedCommit, '--', bundlePath], + 'tree inspection', + timeoutMs + ); + const entries = parseTreeEntries(treeOutput, bundlePath); + + const batchInput = Buffer.from( + `${entries.map((entry) => entry.objectId).join('\n')}\n`, + 'ascii' + ); + const blobs = parseBatchBlobs( + await runGit( + repositoryDir, + ['cat-file', '--batch'], + 'blob extraction', + timeoutMs, + batchInput + ), + entries + ); + let totalBytes = 0; + for (const blob of blobs) { + totalBytes += blob.content.length; + if (totalBytes > MAX_SCHEMA_BUNDLE_BYTES) { + throw new Error( + `Remote schema bundle contains more than ${MAX_SCHEMA_BUNDLE_BYTES} bytes` + ); + } + } + + fs.mkdirSync(options.destinationDir, { recursive: true }); + for (const blob of blobs) { + const destinationPath = path.join( + options.destinationDir, + ...blob.relativePath.split('/') + ); + fs.mkdirSync(path.dirname(destinationPath), { recursive: true }); + fs.writeFileSync(destinationPath, blob.content, { flag: 'wx' }); + } + + return { + resolvedCommit, + ...computeBundleIntegrity(options.destinationDir), + }; + } finally { + fs.rmSync(repositoryDir, { recursive: true, force: true }); + } +} diff --git a/src/core/remote-schema/lockfile.ts b/src/core/remote-schema/lockfile.ts new file mode 100644 index 0000000000..9f60b0bd19 --- /dev/null +++ b/src/core/remote-schema/lockfile.ts @@ -0,0 +1,101 @@ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { parse as parseYaml, stringify as stringifyYaml } from 'yaml'; +import { z } from 'zod'; +import { isValidSchemaSourceName, validateGitSource } from './config.js'; +import { normalizeBundlePath } from './bundle.js'; +import type { RemoteSchemaLock } from './types.js'; + +export const SCHEMA_LOCK_FILE_NAME = 'schemas.lock.yaml'; + +export function getSchemaLockPath(projectRoot: string): string { + return path.join(projectRoot, 'openspec', SCHEMA_LOCK_FILE_NAME); +} + +const LockEntrySchema = z.strictObject({ + git: z + .string() + .min(1) + .refine( + (value) => validateGitSource(value) === 'valid', + 'must be a credential-free HTTPS, SSH, scp-style SSH, or file URL' + ), + requestedRef: z.string().min(1), + resolvedCommit: z.string().regex(/^[0-9a-f]{40}$/), + bundlePath: z.string().min(1).refine((value) => { + try { + normalizeBundlePath(value); + return true; + } catch { + return false; + } + }, 'must be a repository-relative portable Git path'), + integrity: z.string().regex(/^sha256:[0-9a-f]{64}$/), + }); + +const LockSchema = z.strictObject({ + version: z.literal(1), + schemas: z.record( + z.string().refine(isValidSchemaSourceName, 'must be a valid schema name'), + LockEntrySchema + ), + }); + +export function readSchemaLock(projectRoot: string): RemoteSchemaLock | null { + const lockPath = getSchemaLockPath(projectRoot); + if (!fs.existsSync(lockPath)) { + return null; + } + + try { + const parsed = parseYaml(fs.readFileSync(lockPath, 'utf8')); + const result = LockSchema.safeParse(parsed); + if (!result.success) { + const details = result.error.issues + .map((issue) => `${issue.path.join('.')}: ${issue.message}`) + .join(', '); + throw new Error(details); + } + return result.data; + } catch (error) { + throw new Error( + `Invalid remote schema lockfile at '${lockPath}': ${ + error instanceof Error ? error.message.split('\n')[0] : String(error) + }` + ); + } +} + +function sortedLock(lock: RemoteSchemaLock): RemoteSchemaLock { + const schemas = Object.fromEntries( + Object.entries(lock.schemas).sort(([left], [right]) => left.localeCompare(right)) + ); + return { version: 1, schemas }; +} + +export function writeSchemaLock(projectRoot: string, lock: RemoteSchemaLock): void { + const result = LockSchema.safeParse(lock); + if (!result.success) { + throw new Error(`Invalid remote schema lockfile data: ${result.error.message}`); + } + + const lockPath = getSchemaLockPath(projectRoot); + const lockDir = path.dirname(lockPath); + fs.mkdirSync(lockDir, { recursive: true }); + const tempPath = path.join( + lockDir, + `.${SCHEMA_LOCK_FILE_NAME}.${process.pid}.${randomUUID()}.tmp` + ); + + try { + fs.writeFileSync(tempPath, stringifyYaml(sortedLock(result.data)), { + encoding: 'utf8', + flag: 'wx', + }); + fs.renameSync(tempPath, lockPath); + } catch (error) { + fs.rmSync(tempPath, { force: true }); + throw error; + } +} diff --git a/src/core/remote-schema/sync-lock.ts b/src/core/remote-schema/sync-lock.ts new file mode 100644 index 0000000000..2f85f7ccfd --- /dev/null +++ b/src/core/remote-schema/sync-lock.ts @@ -0,0 +1,385 @@ +import { randomUUID } from 'node:crypto'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +interface SchemaSyncParticipant { + token: string; + pid: number; + hostname: string; + startedAt: string; + number?: number; +} + +export interface SchemaSyncLockOptions { + timeoutMs?: number; + retryDelayMs?: number; +} + +export class SchemaSyncLockError extends Error { + readonly code = 'schema_sync_locked'; + + constructor(message: string) { + super(message); + this.name = 'SchemaSyncLockError'; + } +} + +const DEFAULT_TIMEOUT_MS = 30_000; +const DEFAULT_RETRY_DELAY_MS = 50; +const CHOOSING_SUFFIX = '.choosing.json'; +const TICKET_SUFFIX = '.ticket.json'; +const SELF_IGNORE_FILE = '.gitignore'; +const SELF_IGNORE_CONTENT = '*\n'; + +export function getSchemaSyncLockPath(projectRoot: string): string { + return path.join(projectRoot, 'openspec', '.schemas.lock'); +} + +function readParticipant(filePath: string): SchemaSyncParticipant | null { + try { + const value = JSON.parse( + fs.readFileSync(filePath, 'utf8') + ) as Partial; + if ( + typeof value.token !== 'string' || + typeof value.pid !== 'number' || + typeof value.hostname !== 'string' || + typeof value.startedAt !== 'string' || + (value.number !== undefined && + (!Number.isSafeInteger(value.number) || value.number < 1)) + ) { + return null; + } + if ( + filePath.endsWith(TICKET_SUFFIX) && + value.number === undefined + ) { + return null; + } + return value as SchemaSyncParticipant; + } catch { + return null; + } +} + +function isProcessAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code !== 'ESRCH'; + } +} + +function isAbandonedSameHost( + participant: SchemaSyncParticipant | null +): boolean { + return ( + participant !== null && + participant.hostname === os.hostname() && + !isProcessAlive(participant.pid) + ); +} + +function removeOwnFile(filePath: string, token: string): void { + if (readParticipant(filePath)?.token === token) { + try { + fs.unlinkSync(filePath); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') { + throw error; + } + } + } +} + +function removeParticipantFile(filePath: string): void { + try { + fs.unlinkSync(filePath); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (code !== 'ENOENT') { + throw error; + } + } +} + +function malformedParticipantExpired( + filePath: string, + timeoutMs: number +): boolean { + try { + // Date.now() has integer-millisecond precision while filesystem mtimes may + // retain fractional milliseconds. Compare on the same precision so a file + // created just before acquisition expires at the bounded deadline instead + // of surviving it by a sub-millisecond rounding artifact. + const modifiedAtMs = Math.floor(fs.statSync(filePath).mtimeMs); + return Date.now() - modifiedAtMs >= timeoutMs; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') { + return false; + } + throw error; + } +} + +function listParticipantFiles( + lockPath: string, + suffix: string, + timeoutMs: number +): Array<{ filePath: string; participant: SchemaSyncParticipant | null }> { + let names: string[]; + try { + names = fs.readdirSync(lockPath); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') { + return []; + } + throw error; + } + + const participants: Array<{ + filePath: string; + participant: SchemaSyncParticipant | null; + }> = []; + for (const name of names.filter((candidate) => candidate.endsWith(suffix))) { + const filePath = path.join(lockPath, name); + const participant = readParticipant(filePath); + if (isAbandonedSameHost(participant)) { + removeOwnFile(filePath, participant!.token); + continue; + } + if ( + participant === null && + malformedParticipantExpired(filePath, timeoutMs) + ) { + removeParticipantFile(filePath); + continue; + } + participants.push({ filePath, participant }); + } + return participants; +} + +function ensureSelfIgnoredLockDirectory(lockPath: string): void { + fs.mkdirSync(lockPath, { recursive: true }); + const ignorePath = path.join(lockPath, SELF_IGNORE_FILE); + try { + fs.writeFileSync(ignorePath, SELF_IGNORE_CONTENT, { + encoding: 'utf8', + flag: 'wx', + }); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'EEXIST') { + throw error; + } + } + if (fs.readFileSync(ignorePath, 'utf8') === SELF_IGNORE_CONTENT) { + return; + } + + const stagingPath = path.join( + lockPath, + `.self-ignore-${randomUUID()}.tmp` + ); + try { + fs.writeFileSync(stagingPath, SELF_IGNORE_CONTENT, { + encoding: 'utf8', + flag: 'wx', + }); + fs.renameSync(stagingPath, ignorePath); + } finally { + fs.rmSync(stagingPath, { force: true }); + } +} + +function createParticipantFile( + lockPath: string, + fileName: string, + participant: SchemaSyncParticipant, + deadline: number +): string { + const filePath = path.join(lockPath, fileName); + while (true) { + let stagingPath: string | undefined; + try { + ensureSelfIgnoredLockDirectory(lockPath); + stagingPath = path.join( + lockPath, + `.participant-${randomUUID()}.tmp` + ); + fs.writeFileSync(stagingPath, JSON.stringify(participant), { + encoding: 'utf8', + flag: 'wx', + }); + try { + fs.linkSync(stagingPath, filePath); + } catch (linkError) { + const code = (linkError as NodeJS.ErrnoException).code; + if (code === 'ENOENT' || code === 'EEXIST') { + throw linkError; + } + // Some filesystems do not support hard links. The participant name + // contains a fresh UUID, so rename remains an atomic no-overwrite + // publication in practice while preserving cleanup of the staging file. + fs.renameSync(stagingPath, filePath); + } + return filePath; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') { + if (Date.now() >= deadline) { + throw new SchemaSyncLockError( + `schema_sync_locked: timed out creating a participant in '${lockPath}'` + ); + } + continue; + } + throw error; + } finally { + if (stagingPath) { + fs.rmSync(stagingPath, { force: true }); + } + } + } +} + +function delay(milliseconds: number): Promise { + return new Promise((resolve) => setTimeout(resolve, milliseconds)); +} + +function precedes( + left: SchemaSyncParticipant, + right: SchemaSyncParticipant +): boolean { + const numberDifference = left.number! - right.number!; + return numberDifference < 0 || + (numberDifference === 0 && left.token < right.token); +} + +async function acquireSchemaSyncLock( + projectRoot: string, + options: SchemaSyncLockOptions +): Promise<{ lockPath: string; token: string; ticketPath: string }> { + const lockPath = getSchemaSyncLockPath(projectRoot); + const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS; + const retryDelayMs = options.retryDelayMs ?? DEFAULT_RETRY_DELAY_MS; + const deadline = Date.now() + timeoutMs; + const token = randomUUID(); + const baseParticipant = { + token, + pid: process.pid, + hostname: os.hostname(), + startedAt: new Date().toISOString(), + }; + const choosingPath = createParticipantFile( + lockPath, + `claim-${token}${CHOOSING_SUFFIX}`, + baseParticipant, + deadline + ); + let ticketPath: string | null = null; + + try { + const highestNumber = listParticipantFiles( + lockPath, + TICKET_SUFFIX, + timeoutMs + ) + .reduce( + (highest, entry) => + Math.max(highest, entry.participant?.number ?? 0), + 0 + ); + const participant = { + ...baseParticipant, + number: highestNumber + 1, + }; + ticketPath = createParticipantFile( + lockPath, + `claim-${token}${TICKET_SUFFIX}`, + participant, + deadline + ); + removeOwnFile(choosingPath, token); + + while (true) { + const anotherIsChoosing = listParticipantFiles( + lockPath, + CHOOSING_SUFFIX, + timeoutMs + ).some((entry) => entry.participant?.token !== token); + const predecessorExists = listParticipantFiles( + lockPath, + TICKET_SUFFIX, + timeoutMs + ).some( + (entry) => + entry.participant === null || + (entry.participant.token !== token && + precedes(entry.participant, participant)) + ); + if (!anotherIsChoosing && !predecessorExists) { + return { lockPath, token, ticketPath }; + } + if (Date.now() >= deadline) { + // Time can cross the deadline between the scans above and this check. + // Give malformed participants one final expiry/reclamation sweep before + // reporting a live owner; otherwise a participant expiring exactly at + // the deadline is incorrectly treated as still holding the lock. + const choosingAfterDeadline = listParticipantFiles( + lockPath, + CHOOSING_SUFFIX, + timeoutMs + ).some((entry) => entry.participant?.token !== token); + const predecessorAfterDeadline = listParticipantFiles( + lockPath, + TICKET_SUFFIX, + timeoutMs + ).some( + (entry) => + entry.participant === null || + (entry.participant.token !== token && + precedes(entry.participant, participant)) + ); + if (!choosingAfterDeadline && !predecessorAfterDeadline) { + return { lockPath, token, ticketPath }; + } + throw new SchemaSyncLockError( + `schema_sync_locked: another schema synchronization owns '${lockPath}'` + ); + } + await delay(retryDelayMs); + } + } catch (error) { + removeOwnFile(choosingPath, token); + if (ticketPath) { + removeOwnFile(ticketPath, token); + } + throw error; + } +} + +function releaseSchemaSyncLock( + lockPath: string, + ticketPath: string, + token: string +): void { + removeOwnFile(ticketPath, token); +} + +export async function withSchemaSyncLock( + projectRoot: string, + callback: () => Promise, + options: SchemaSyncLockOptions = {} +): Promise { + const { lockPath, token, ticketPath } = await acquireSchemaSyncLock( + projectRoot, + options + ); + try { + return await callback(); + } finally { + releaseSchemaSyncLock(lockPath, ticketPath, token); + } +} diff --git a/src/core/remote-schema/sync.ts b/src/core/remote-schema/sync.ts new file mode 100644 index 0000000000..e58d39fde8 --- /dev/null +++ b/src/core/remote-schema/sync.ts @@ -0,0 +1,209 @@ +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { validateRemoteSchemaDirectory } from '../artifact-graph/schema-directory.js'; +import { getGlobalDataDir } from '../global-config.js'; +import { readProjectConfig } from '../project-config.js'; +import { installRemoteSchemaCache, verifyRemoteSchemaCache } from './cache.js'; +import { fetchSchemaBundleFromGit } from './git.js'; +import { getSchemaLockPath, readSchemaLock, writeSchemaLock } from './lockfile.js'; +import { assertNoProjectSchemaConflict } from './authority.js'; +import { withSchemaSyncLock } from './sync-lock.js'; +import type { + GitSchemaSource, + RemoteSchemaLock, + RemoteSchemaLockEntry, +} from './types.js'; + +export interface SyncRemoteSchemasOptions { + name?: string; + locked?: boolean; + globalDataDir?: string; +} + +export interface SyncedRemoteSchema { + name: string; + git: string; + requestedRef: string; + resolvedCommit: string; + bundlePath: string; + integrity: string; + cachePath: string; + restored: boolean; +} + +export interface SyncRemoteSchemasResult { + mode: 'update' | 'locked'; + lockfile: string; + locked: boolean; + schemas: SyncedRemoteSchema[]; + status: Array<{ + level: 'error' | 'warning'; + code: string; + message: string; + }>; +} + +function sourceMatchesLock( + source: GitSchemaSource, + entry: RemoteSchemaLockEntry +): boolean { + return ( + source.git === entry.git && + source.ref === entry.requestedRef && + source.path === entry.bundlePath + ); +} + +function selectedSources( + sources: Record, + name: string | undefined +): Array<[string, GitSchemaSource]> { + if (name !== undefined) { + const source = sources[name]; + if (!source) { + throw new Error(`Remote schema '${name}' is not declared in openspec/config.yaml`); + } + return [[name, source]]; + } + return Object.entries(sources).sort(([left], [right]) => left.localeCompare(right)); +} + +export async function syncRemoteSchemas( + projectRoot: string, + options: SyncRemoteSchemasOptions = {} +): Promise { + return withSchemaSyncLock(projectRoot, () => + syncRemoteSchemasUnlocked(projectRoot, options) + ); +} + +async function syncRemoteSchemasUnlocked( + projectRoot: string, + options: SyncRemoteSchemasOptions +): Promise { + const config = readProjectConfig(projectRoot); + const sources = config?.schemaSources; + if (!sources || Object.keys(sources).length === 0) { + throw new Error('No remote schema sources are declared in openspec/config.yaml'); + } + + const selected = selectedSources(sources, options.name); + for (const [name] of selected) { + assertNoProjectSchemaConflict(projectRoot, name); + } + let currentLock: RemoteSchemaLock | null; + try { + currentLock = readSchemaLock(projectRoot); + } catch (error) { + if (options.locked || options.name !== undefined) { + throw new Error( + `${ + error instanceof Error ? error.message : String(error) + }; run 'openspec schema sync' without a schema name to rebuild it` + ); + } + currentLock = null; + } + if (options.locked && !currentLock) { + throw new Error( + "Remote schema lockfile is missing; run 'openspec schema sync' to create it" + ); + } + + const globalDataDir = options.globalDataDir ?? getGlobalDataDir(); + const nextEntries: Record = + options.name === undefined + ? {} + : { ...(currentLock?.schemas ?? {}) }; + const results: SyncedRemoteSchema[] = []; + + for (const [name, source] of selected) { + const lockedEntry = currentLock?.schemas[name]; + if (options.locked) { + if (!lockedEntry) { + throw new Error(`Remote schema '${name}' is missing from the lockfile`); + } + if (!sourceMatchesLock(source, lockedEntry)) { + throw new Error( + `Remote schema '${name}' lock does not match the configured source; run 'openspec schema sync' to update it` + ); + } + try { + const cacheDir = verifyRemoteSchemaCache(lockedEntry.integrity, globalDataDir); + validateRemoteSchemaDirectory(cacheDir, name); + results.push({ + name, + git: source.git, + requestedRef: source.ref, + resolvedCommit: lockedEntry.resolvedCommit, + bundlePath: source.path, + integrity: lockedEntry.integrity, + cachePath: cacheDir, + restored: false, + }); + continue; + } catch { + // Explicit --locked sync restores a missing or corrupt entry from the exact commit. + } + } + + const extractionDir = fs.mkdtempSync( + path.join(os.tmpdir(), `openspec-schema-${name}-`) + ); + try { + const fetched = await fetchSchemaBundleFromGit({ + git: source.git, + requestedRef: source.ref, + lockedCommit: options.locked ? lockedEntry?.resolvedCommit : undefined, + bundlePath: source.path, + destinationDir: extractionDir, + }); + validateRemoteSchemaDirectory(extractionDir, name); + if (options.locked && fetched.integrity !== lockedEntry?.integrity) { + throw new Error( + `Remote schema '${name}' content does not match the lockfile integrity` + ); + } + const cacheDir = installRemoteSchemaCache( + extractionDir, + fetched.integrity, + globalDataDir + ); + results.push({ + name, + git: source.git, + requestedRef: source.ref, + resolvedCommit: fetched.resolvedCommit, + bundlePath: source.path, + integrity: fetched.integrity, + cachePath: cacheDir, + restored: Boolean(options.locked), + }); + if (!options.locked) { + nextEntries[name] = { + git: source.git, + requestedRef: source.ref, + resolvedCommit: fetched.resolvedCommit, + bundlePath: source.path, + integrity: fetched.integrity, + }; + } + } finally { + fs.rmSync(extractionDir, { recursive: true, force: true }); + } + } + + if (!options.locked) { + const nextLock: RemoteSchemaLock = { version: 1, schemas: nextEntries }; + writeSchemaLock(projectRoot, nextLock); + } + + return { + mode: options.locked ? 'locked' : 'update', + lockfile: getSchemaLockPath(projectRoot), + locked: Boolean(options.locked), + schemas: results, + status: [], + }; +} diff --git a/src/core/remote-schema/types.ts b/src/core/remote-schema/types.ts new file mode 100644 index 0000000000..711e4c99a9 --- /dev/null +++ b/src/core/remote-schema/types.ts @@ -0,0 +1,18 @@ +export interface GitSchemaSource { + git: string; + ref: string; + path: string; +} + +export interface RemoteSchemaLockEntry { + git: string; + requestedRef: string; + resolvedCommit: string; + bundlePath: string; + integrity: string; +} + +export interface RemoteSchemaLock { + version: 1; + schemas: Record; +} diff --git a/src/core/root-selection.ts b/src/core/root-selection.ts index 333d24f721..5a0abbe4fa 100644 --- a/src/core/root-selection.ts +++ b/src/core/root-selection.ts @@ -60,6 +60,8 @@ export interface ResolveOpenSpecRootOptions extends StoreSelectorOptions { export interface ResolvedOpenSpecRoot { path: string; + /** Consumer repository that owns schema config, locks, and local schemas. */ + schemaRoot: string; changesDir: string; specsDir: string; archiveDir: string; @@ -117,10 +119,12 @@ function doctorFix(id: string): string { function makeRoot( rootPath: string, source: OpenSpecRootSource, - storeId?: string + storeId?: string, + schemaRoot = rootPath ): ResolvedOpenSpecRoot { return { path: rootPath, + schemaRoot, changesDir: path.join(rootPath, 'openspec', 'changes'), specsDir: path.join(rootPath, 'openspec', 'specs'), archiveDir: path.join(rootPath, 'openspec', 'changes', 'archive'), @@ -145,7 +149,8 @@ function canonicalDirectory(startPath: string): string { async function resolveStoreRoot( id: string, globalDataDir?: string, - source: OpenSpecRootSource = 'store' + source: OpenSpecRootSource = 'store', + schemaRoot?: string ): Promise { try { validateStoreId(id); @@ -213,7 +218,12 @@ async function resolveStoreRoot( { target: 'openspec.root', fix: doctorFix(id) } ); case 'ok': - return makeRoot(inspection.canonicalRoot, source, id); + return makeRoot( + inspection.canonicalRoot, + source, + id, + schemaRoot ?? inspection.canonicalRoot + ); default: { // Exhaustiveness guard: a new inspection kind must be handled // here explicitly, not fall through to an undefined root. @@ -332,7 +342,12 @@ async function resolveNearestOrDeclaredRoot( } try { - return await resolveStoreRoot(pointer.value, globalDataDir, 'declared'); + return await resolveStoreRoot( + pointer.value, + globalDataDir, + 'declared', + nearestRoot + ); } catch (error) { if (error instanceof RootSelectionError) { // Rewrap with the declaration origin. The unknown-store fix is @@ -403,11 +418,18 @@ export async function resolveOpenSpecRoot( ); } + const startPath = options.startPath ?? process.cwd(); + const consumerRoot = findRepoPlanningRootSync(startPath); + if (options.store !== undefined) { - return resolveStoreRoot(options.store, options.globalDataDir); + return resolveStoreRoot( + options.store, + options.globalDataDir, + 'store', + consumerRoot ?? undefined + ); } - const startPath = options.startPath ?? process.cwd(); const nearestRoot = findQualifyingRootSync(startPath); if (nearestRoot) { return resolveNearestOrDeclaredRoot(nearestRoot, options.globalDataDir); diff --git a/src/core/validation/validator.ts b/src/core/validation/validator.ts index 56f771e1a8..94e2003e3f 100644 --- a/src/core/validation/validator.ts +++ b/src/core/validation/validator.ts @@ -37,9 +37,11 @@ import { getPackageSchemasDir, getSchemaDir } from '../artifact-graph/index.js'; export class Validator { private strictMode: boolean; + private schemaRoot?: string; - constructor(strictMode: boolean = false) { + constructor(strictMode: boolean = false, schemaRoot?: string) { this.strictMode = strictMode; + this.schemaRoot = schemaRoot; } async validateSpec(filePath: string): Promise { @@ -105,7 +107,7 @@ export class Validator { const result = ChangeSchema.safeParse(change); - const marker = readSkipSpecsMarker(changeDir); + const marker = readSkipSpecsMarker(changeDir, this.schemaRoot); if (marker.invalidReason) { issues.push({ level: 'ERROR', path: METADATA_FILENAME, message: this.formatInvalidMarkerMessage(marker.invalidReason) }); } @@ -421,7 +423,7 @@ export class Validator { }); } - const marker = readSkipSpecsMarker(changeDir); + const marker = readSkipSpecsMarker(changeDir, this.schemaRoot); if (marker.invalidReason) { issues.push({ level: 'ERROR', path: METADATA_FILENAME, message: this.formatInvalidMarkerMessage(marker.invalidReason) }); } @@ -470,11 +472,12 @@ export class Validator { projectRoot: string ): Promise { try { - const schemaName = resolveSchemaForChange(changeDir, undefined, projectRoot).replace( + const schemaRoot = this.schemaRoot ?? projectRoot; + const schemaName = resolveSchemaForChange(changeDir, undefined, schemaRoot).replace( /\.ya?ml$/, '' ); - const schemaDir = getSchemaDir(schemaName, projectRoot); + const schemaDir = getSchemaDir(schemaName, schemaRoot); const builtInSchemaDir = path.join(getPackageSchemasDir(), 'spec-driven'); if ( schemaName !== 'spec-driven' || @@ -490,7 +493,12 @@ export class Validator { let taskFiles: string[]; try { - taskFiles = resolveTaskFilesForChange(changeDir, projectRoot); + taskFiles = resolveTaskFilesForChange( + changeDir, + projectRoot, + undefined, + this.schemaRoot ?? projectRoot + ); } catch { return []; } diff --git a/src/utils/change-metadata.ts b/src/utils/change-metadata.ts index 1f31a44596..b5ce4a5f10 100644 --- a/src/utils/change-metadata.ts +++ b/src/utils/change-metadata.ts @@ -93,7 +93,8 @@ export function writeChangeMetadata( */ export function readChangeMetadata( changeDir: string, - projectRoot?: string + projectRoot?: string, + projectConfig?: ProjectConfig | null ): ChangeMetadata | null { const metaPath = path.join(changeDir, METADATA_FILENAME); @@ -135,7 +136,7 @@ export function readChangeMetadata( } // Validate that the schema exists - const availableSchemas = listSchemas(projectRoot); + const availableSchemas = listSchemas(projectRoot, projectConfig); if (!availableSchemas.includes(parseResult.data.schema)) { throw new ChangeMetadataError( `Unknown schema '${parseResult.data.schema}'. Available: ${availableSchemas.join(', ')}`, @@ -235,8 +236,11 @@ export type SkipSpecsMarker = MetadataMarker; * Missing metadata means "not declared"; a marker that cannot be honored * yields invalidReason so callers can say why. */ -export function readSkipSpecsMarker(changeDir: string): MetadataMarker { - return readBooleanMarker(changeDir, 'skip_specs'); +export function readSkipSpecsMarker( + changeDir: string, + schemaRoot?: string +): SkipSpecsMarker { + return readBooleanMarker(changeDir, 'skip_specs', schemaRoot); } /** @@ -249,8 +253,11 @@ export function readSkipSpecsMarker(changeDir: string): MetadataMarker { * (#1302). Declared rather than inferred because the delete is recoverable only * from git, so it is the author's call. */ -export function readRetireCapabilitiesMarker(changeDir: string): MetadataMarker { - return readBooleanMarker(changeDir, 'retire_capabilities'); +export function readRetireCapabilitiesMarker( + changeDir: string, + schemaRoot?: string +): MetadataMarker { + return readBooleanMarker(changeDir, 'retire_capabilities', schemaRoot); } /** @@ -273,7 +280,8 @@ function unhonorable(reason: string): MetadataMarker { function readBooleanMarker( changeDir: string, - key: 'skip_specs' | 'retire_capabilities' + key: 'skip_specs' | 'retire_capabilities', + schemaRoot?: string ): MetadataMarker { let raw: string; try { @@ -314,7 +322,7 @@ function readBooleanMarker( // resolveSchema alone would normalize and accept); resolveSchema then // proves the schema actually parses. Any failure fails closed. try { - const projectRoot = path.resolve(changeDir, '../../..'); + const projectRoot = schemaRoot ?? path.resolve(changeDir, '../../..'); if (!listSchemas(projectRoot).includes(result.data.schema)) { return unhonorable(`schema: unknown schema '${result.data.schema}'`); } diff --git a/src/utils/change-utils.ts b/src/utils/change-utils.ts index 803405953e..ea1b2fa418 100644 --- a/src/utils/change-utils.ts +++ b/src/utils/change-utils.ts @@ -20,6 +20,8 @@ export interface CreateChangeOptions { defaultSchema?: string; /** Directory that should contain the change directories */ changesDir?: string; + /** Consumer repository that owns schema configuration and caches. */ + schemaRoot?: string; /** Additional metadata to persist in the change's .openspec.yaml */ metadata?: Partial>; } @@ -140,6 +142,7 @@ export async function createChange( } const defaultSchema = options.defaultSchema ?? DEFAULT_SCHEMA; + const schemaRoot = options.schemaRoot ?? projectRoot; // Determine schema: explicit option → project config → supplied default let schemaName: string; @@ -148,7 +151,7 @@ export async function createChange( } else { // Try to read from project config try { - const config = readProjectConfig(projectRoot); + const config = readProjectConfig(schemaRoot); schemaName = config?.schema ?? defaultSchema; } catch { // If config read fails, use default @@ -157,7 +160,7 @@ export async function createChange( } // Validate the resolved schema - validateSchemaName(schemaName, projectRoot); + validateSchemaName(schemaName, schemaRoot); // Build the change directory path const changeDir = path.join(options.changesDir ?? path.join(projectRoot, 'openspec', 'changes'), name); @@ -167,7 +170,7 @@ export async function createChange( throw new Error(`Change '${name}' already exists at ${changeDir}`); } - const schema = resolveSchema(schemaName, projectRoot); + const schema = resolveSchema(schemaName, schemaRoot); const skipsSpecs = !schema.artifacts.some(artifact => isSpecsArtifactPath(artifact.generates) ); @@ -178,18 +181,22 @@ export async function createChange( // specs/ and changes/archive/ exist, and write a config only when // none exists. The config records the PROJECT default schema, never // a one-change --schema override. - const openspecDir = path.join(projectRoot, 'openspec'); + const planningOpenSpecDir = path.join(projectRoot, 'openspec'); + const configOpenSpecDir = path.join(schemaRoot, 'openspec'); // Create the directory (including parent directories if needed) await FileSystemUtils.createDirectory(changeDir); - await FileSystemUtils.createDirectory(path.join(openspecDir, 'specs')); - await FileSystemUtils.createDirectory(path.join(openspecDir, 'changes', 'archive')); - const configPath = path.join(openspecDir, 'config.yaml'); - const configYmlPath = path.join(openspecDir, 'config.yml'); + await FileSystemUtils.createDirectory(path.join(planningOpenSpecDir, 'specs')); + await FileSystemUtils.createDirectory( + path.join(planningOpenSpecDir, 'changes', 'archive') + ); + const configPath = path.join(configOpenSpecDir, 'config.yaml'); + const configYmlPath = path.join(configOpenSpecDir, 'config.yml'); if ( !(await FileSystemUtils.fileExists(configPath)) && !(await FileSystemUtils.fileExists(configYmlPath)) ) { + await FileSystemUtils.createDirectory(configOpenSpecDir); await FileSystemUtils.writeFile(configPath, `schema: ${defaultSchema}\n`); } @@ -199,7 +206,7 @@ export async function createChange( created: formatLocalDate(), ...(skipsSpecs ? { skip_specs: true } : {}), ...options.metadata, - }, projectRoot); + }, schemaRoot); return { schema: schemaName, changeDir }; } diff --git a/src/utils/task-progress.ts b/src/utils/task-progress.ts index e3ebf56ac6..f248480b4e 100644 --- a/src/utils/task-progress.ts +++ b/src/utils/task-progress.ts @@ -100,12 +100,13 @@ export type SchemaGlobCache = Map; function resolveTrackedTasksGlob( changeDir: string, projectRoot: string, - schemaGlobCache?: SchemaGlobCache + schemaGlobCache?: SchemaGlobCache, + schemaRoot = projectRoot ): string | undefined { try { - const schemaName = resolveSchemaForChange(changeDir, undefined, projectRoot); + const schemaName = resolveSchemaForChange(changeDir, undefined, schemaRoot); if (schemaGlobCache?.has(schemaName)) return schemaGlobCache.get(schemaName); - const schema = resolveSchema(schemaName, projectRoot); + const schema = resolveSchema(schemaName, schemaRoot); const generates = findTrackedTasksArtifact(schema)?.generates; schemaGlobCache?.set(schemaName, generates); return generates; @@ -118,9 +119,15 @@ function resolveTrackedTasksGlob( export function resolveTaskFilesForChange( changeDir: string, projectRoot: string, - schemaGlobCache?: SchemaGlobCache + schemaGlobCache?: SchemaGlobCache, + schemaRoot = projectRoot ): string[] { - const generates = resolveTrackedTasksGlob(changeDir, projectRoot, schemaGlobCache); + const generates = resolveTrackedTasksGlob( + changeDir, + projectRoot, + schemaGlobCache, + schemaRoot + ); return generates ? resolveArtifactOutputs(changeDir, generates) : []; } @@ -169,10 +176,16 @@ export async function getTaskProgressDetailForChange( changesDir: string, changeName: string, projectRoot: string, - schemaGlobCache?: SchemaGlobCache + schemaGlobCache?: SchemaGlobCache, + schemaRoot = projectRoot ): Promise { const changeDir = path.join(changesDir, changeName); - const files = resolveTaskFilesForChange(changeDir, projectRoot, schemaGlobCache); + const files = resolveTaskFilesForChange( + changeDir, + projectRoot, + schemaGlobCache, + schemaRoot + ); const targets = files.length > 0 ? files : [path.join(changeDir, 'tasks.md')]; const unreadable: string[] = []; let total = 0; @@ -195,12 +208,15 @@ export async function getTaskProgressDetailForChange( export async function getTaskProgressForChange( changesDir: string, changeName: string, - projectRoot: string + projectRoot: string, + schemaRoot = projectRoot ): Promise { const { total, completed } = await getTaskProgressDetailForChange( changesDir, changeName, - projectRoot + projectRoot, + undefined, + schemaRoot ); return { total, completed }; } @@ -210,5 +226,3 @@ export function formatTaskStatus(progress: TaskProgress): string { if (progress.completed === progress.total) return '✓ Complete'; return `${progress.completed}/${progress.total} tasks`; } - - diff --git a/test/cli-e2e/remote-schema-concurrency.test.ts b/test/cli-e2e/remote-schema-concurrency.test.ts new file mode 100644 index 0000000000..8074f77c10 --- /dev/null +++ b/test/cli-e2e/remote-schema-concurrency.test.ts @@ -0,0 +1,124 @@ +import { execFileSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { parse as parseYaml } from 'yaml'; +import { afterEach, describe, expect, it } from 'vitest'; +import { runCLI } from '../helpers/run-cli.js'; + +function git(cwd: string, ...args: string[]): void { + execFileSync('git', ['-c', 'commit.gpgsign=false', ...args], { + cwd, + env: { + ...process.env, + GIT_CONFIG_GLOBAL: path.join(cwd, '.missing-global-gitconfig'), + GIT_CONFIG_SYSTEM: path.join(cwd, '.missing-system-gitconfig'), + GIT_AUTHOR_NAME: 'OpenSpec Test', + GIT_AUTHOR_EMAIL: 'openspec@example.test', + GIT_COMMITTER_NAME: 'OpenSpec Test', + GIT_COMMITTER_EMAIL: 'openspec@example.test', + }, + stdio: ['ignore', 'pipe', 'pipe'], + }); +} + +function createRemote(root: string, name: string): string { + const repo = path.join(root, `${name}-remote`); + const schemaDir = path.join(repo, 'schemas', name); + fs.mkdirSync(path.join(schemaDir, 'templates'), { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: ${name} +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: proposal.md + requires: [] +` + ); + fs.writeFileSync(path.join(schemaDir, 'templates', 'proposal.md'), '# Proposal\n'); + git(repo, 'init', '-b', 'main'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', name); + return repo; +} + +describe('remote schema synchronization across CLI processes', () => { + let tempDir: string | undefined; + + afterEach(() => { + if (tempDir) { + fs.rmSync(tempDir, { recursive: true, force: true }); + } + }); + + it('reclaims an abandoned ticket while concurrent named sync processes preserve both entries', async () => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-sync-processes-')); + const projectRoot = path.join(tempDir, 'consumer'); + fs.mkdirSync(path.join(projectRoot, 'openspec'), { recursive: true }); + const alphaRepo = createRemote(tempDir, 'alpha-flow'); + const betaRepo = createRemote(tempDir, 'beta-flow'); + fs.writeFileSync( + path.join(projectRoot, 'openspec', 'config.yaml'), + `schema: spec-driven +schemaSources: + alpha-flow: + git: ${pathToFileURL(alphaRepo).href} + ref: main + path: schemas/alpha-flow + beta-flow: + git: ${pathToFileURL(betaRepo).href} + ref: main + path: schemas/beta-flow +` + ); + const env = { + XDG_DATA_HOME: path.join(tempDir, 'data'), + XDG_CONFIG_HOME: path.join(tempDir, 'config'), + OPENSPEC_TELEMETRY: '0', + }; + const staleLock = path.join( + projectRoot, + 'openspec', + '.schemas.lock' + ); + fs.mkdirSync(staleLock); + fs.writeFileSync( + path.join(staleLock, 'claim-abandoned.ticket.json'), + JSON.stringify({ + token: 'abandoned', + pid: 2_147_483_647, + hostname: os.hostname(), + startedAt: new Date().toISOString(), + number: 1, + }) + ); + + const [alpha, beta] = await Promise.all([ + runCLI(['schema', 'sync', 'alpha-flow', '--json'], { + cwd: projectRoot, + env, + }), + runCLI(['schema', 'sync', 'beta-flow', '--json'], { + cwd: projectRoot, + env, + }), + ]); + + expect(alpha.exitCode, alpha.stderr).toBe(0); + expect(beta.exitCode, beta.stderr).toBe(0); + const lock = parseYaml( + fs.readFileSync( + path.join(projectRoot, 'openspec', 'schemas.lock.yaml'), + 'utf8' + ) + ) as { schemas: Record }; + expect(Object.keys(lock.schemas).sort()).toEqual([ + 'alpha-flow', + 'beta-flow', + ]); + }, 60_000); +}); diff --git a/test/commands/schema-sync.test.ts b/test/commands/schema-sync.test.ts new file mode 100644 index 0000000000..a78270c078 --- /dev/null +++ b/test/commands/schema-sync.test.ts @@ -0,0 +1,673 @@ +import { execFileSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { Command } from 'commander'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { registerSchemaCommand } from '../../src/commands/schema.js'; +import { schemasCommand } from '../../src/commands/workflow/schemas.js'; +import { templatesCommand } from '../../src/commands/workflow/templates.js'; + +const gitEnv = { + ...process.env, + GIT_AUTHOR_NAME: 'OpenSpec Test', + GIT_AUTHOR_EMAIL: 'openspec@example.test', + GIT_COMMITTER_NAME: 'OpenSpec Test', + GIT_COMMITTER_EMAIL: 'openspec@example.test', +}; + +function git(cwd: string, ...args: string[]): void { + execFileSync('git', args, { + cwd, + env: gitEnv, + stdio: ['ignore', 'pipe', 'pipe'], + }); +} + +function createProgram(): Command { + const program = new Command(); + program.exitOverride(); + registerSchemaCommand(program); + return program; +} + +describe('schema sync command', () => { + let tempDir: string; + let originalCwd: string; + let originalEnv: NodeJS.ProcessEnv; + + beforeEach(() => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-schema-sync-cli-')); + originalCwd = process.cwd(); + originalEnv = { ...process.env }; + process.chdir(tempDir); + process.env.XDG_DATA_HOME = path.join(tempDir, 'data'); + process.exitCode = undefined; + fs.mkdirSync(path.join(tempDir, 'openspec'), { recursive: true }); + }); + + afterEach(() => { + vi.useRealTimers(); + process.chdir(originalCwd); + process.env = originalEnv; + process.exitCode = undefined; + vi.restoreAllMocks(); + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + it('emits exactly one JSON document and a failure exit code', async () => { + fs.writeFileSync(path.join(tempDir, 'openspec', 'config.yaml'), 'schema: spec-driven\n'); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + + await createProgram().parseAsync(['node', 'openspec', 'schema', 'sync', '--json']); + + expect(log).toHaveBeenCalledTimes(1); + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + synced: false, + error: expect.stringMatching(/No remote schema sources/), + }); + expect(process.exitCode).toBe(1); + }); + + it('reports a structured code when another process owns the sync lock', async () => { + const lockDir = path.join(tempDir, 'openspec', '.schemas.lock'); + fs.mkdirSync(lockDir); + fs.writeFileSync( + path.join(lockDir, 'claim-busy.ticket.json'), + JSON.stringify({ + token: 'busy', + pid: process.pid, + hostname: os.hostname(), + startedAt: new Date().toISOString(), + number: 1, + }) + ); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + vi.useFakeTimers(); + + const command = createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'sync', + '--json', + ]); + await vi.advanceTimersByTimeAsync(30_100); + await command; + + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + synced: false, + status: [{ code: 'schema_sync_locked' }], + }); + expect(process.exitCode).toBe(1); + vi.useRealTimers(); + }); + + it('emits JSON when schema which is missing its name', async () => { + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'which', + '--json', + ]); + + expect(log).toHaveBeenCalledTimes(1); + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + name: null, + status: [{ code: 'schema_name_required' }], + }); + expect(process.exitCode).toBe(1); + }); + + it('reports every semantic and template validation failure in JSON output', async () => { + const schemaDir = path.join(tempDir, 'openspec', 'schemas', 'broken-flow'); + fs.mkdirSync(schemaDir, { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: broken-flow +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: missing-proposal.md + requires: [missing-proposal-dependency] + - id: design + generates: design.md + description: Design + template: missing-design.md + requires: [missing-design-dependency] +` + ); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'validate', + 'broken-flow', + '--json', + ]); + + const output = JSON.parse(String(log.mock.calls[0][0])); + expect(output.valid).toBe(false); + expect(output.issues.map((issue: { message: string }) => issue.message)).toEqual( + expect.arrayContaining([ + expect.stringContaining('missing-proposal-dependency'), + expect.stringContaining('missing-design-dependency'), + expect.stringContaining('missing-proposal.md'), + expect.stringContaining('missing-design.md'), + ]) + ); + expect(output.issues).toHaveLength(4); + }); + + it('syncs a named local Git source with JSON output', async () => { + const repo = path.join(tempDir, 'remote'); + const schemaDir = path.join(repo, 'schemas', 'team-flow'); + fs.mkdirSync(path.join(schemaDir, 'templates'), { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: team-flow +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: proposal.md + requires: [] +` + ); + fs.writeFileSync(path.join(schemaDir, 'templates', 'proposal.md'), '# Proposal\n'); + git(repo, 'init', '-b', 'main'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'schema'); + fs.writeFileSync( + path.join(tempDir, 'openspec', 'config.yaml'), + `schema: team-flow +schemaSources: + team-flow: + git: ${pathToFileURL(repo).href} + ref: main + path: schemas/team-flow +` + ); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'sync', + 'team-flow', + '--json', + ]); + + expect(log).toHaveBeenCalledTimes(1); + const output = JSON.parse(String(log.mock.calls[0][0])); + expect(output).toMatchObject({ + synced: true, + locked: false, + schemas: [{ name: 'team-flow', resolvedCommit: expect.stringMatching(/^[0-9a-f]{40}$/) }], + }); + expect(process.exitCode).toBeUndefined(); + expect(fs.existsSync(path.join(tempDir, 'openspec', 'schemas.lock.yaml'))).toBe(true); + + log.mockClear(); + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'sync', + '--locked', + ]); + expect(String(log.mock.calls[0][0])).toMatch( + /Verified 'team-flow': main → [0-9a-f]{40}/ + ); + + log.mockClear(); + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'sync', + 'not-declared', + '--json', + ]); + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + synced: false, + error: expect.stringMatching(/not declared/), + }); + expect(process.exitCode).toBe(1); + process.exitCode = undefined; + + log.mockClear(); + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'which', + 'team-flow', + '--json', + ]); + expect(log).toHaveBeenCalledTimes(1); + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + name: 'team-flow', + source: 'remote', + requestedRef: 'main', + resolvedCommit: expect.stringMatching(/^[0-9a-f]{40}$/), + bundlePath: 'schemas/team-flow', + integrity: expect.stringMatching(/^sha256:/), + }); + + const localSchema = path.join(tempDir, 'openspec', 'schemas', 'team-flow'); + fs.cpSync(schemaDir, localSchema, { recursive: true }); + log.mockClear(); + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'which', + 'team-flow', + '--json', + ]); + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + name: 'team-flow', + available: false, + source: 'remote', + path: null, + status: [ + { + code: 'schema_name_conflict', + message: expect.stringMatching(/project-local schema.*conflicts/i), + }, + ], + }); + expect(process.exitCode).toBe(1); + process.exitCode = undefined; + + log.mockClear(); + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'sync', + 'team-flow', + '--json', + ]); + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + synced: false, + status: [ + expect.objectContaining({ + code: 'schema_name_conflict', + }), + ], + }); + expect(process.exitCode).toBe(1); + process.exitCode = undefined; + }); + + it('reports synchronized remote template paths as remote', async () => { + const repo = path.join(tempDir, 'remote-templates'); + const schemaDir = path.join(repo, 'schemas', 'template-flow'); + fs.mkdirSync(path.join(schemaDir, 'templates'), { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: template-flow +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: proposal.md + requires: [] +` + ); + fs.writeFileSync( + path.join(schemaDir, 'templates', 'proposal.md'), + '# Proposal\n' + ); + git(repo, 'init', '-b', 'main'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'schema'); + fs.writeFileSync( + path.join(tempDir, 'openspec', 'config.yaml'), + `schema: template-flow +schemaSources: + template-flow: + git: ${pathToFileURL(repo).href} + ref: main + path: schemas/template-flow +` + ); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'sync', + 'template-flow', + '--json', + ]); + expect(process.exitCode).toBeUndefined(); + + log.mockClear(); + await templatesCommand({ schema: 'template-flow', json: true }); + + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + proposal: { + path: expect.stringContaining('proposal.md'), + source: 'remote', + }, + }); + }); + + it('preserves every remote validation issue path in sync JSON output', async () => { + const repo = path.join(tempDir, 'invalid-remote'); + const schemaDir = path.join(repo, 'schemas', 'broken-flow'); + fs.mkdirSync(schemaDir, { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: wrong-name +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: missing-proposal.md + requires: [] + - id: design + generates: design.md + description: Design + template: missing-design.md + requires: [] +` + ); + git(repo, 'init', '-b', 'main'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'invalid schema'); + fs.writeFileSync( + path.join(tempDir, 'openspec', 'config.yaml'), + `schema: spec-driven +schemaSources: + broken-flow: + git: ${pathToFileURL(repo).href} + ref: main + path: schemas/broken-flow +` + ); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'sync', + 'broken-flow', + '--json', + ]); + + const output = JSON.parse(String(log.mock.calls[0][0])); + expect(output.synced).toBe(false); + expect(output.status).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + code: 'remote_schema_invalid', + path: 'schema.yaml', + }), + expect.objectContaining({ + code: 'remote_schema_invalid', + path: 'templates', + }), + expect.objectContaining({ + code: 'remote_schema_invalid', + path: 'templates/missing-proposal.md', + }), + expect.objectContaining({ + code: 'remote_schema_invalid', + path: 'templates/missing-design.md', + }), + ]) + ); + expect(process.exitCode).toBe(1); + }); + + it('resolves the consumer root when synchronizing from a nested directory', async () => { + const repo = path.join(tempDir, 'remote-nested'); + const schemaDir = path.join(repo, 'schemas', 'nested-flow'); + fs.mkdirSync(path.join(schemaDir, 'templates'), { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: nested-flow +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: proposal.md + requires: [] +` + ); + fs.writeFileSync(path.join(schemaDir, 'templates', 'proposal.md'), '# Proposal\n'); + git(repo, 'init', '-b', 'main'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'schema'); + fs.writeFileSync( + path.join(tempDir, 'openspec', 'config.yaml'), + `schema: nested-flow +store: team-context +schemaSources: + nested-flow: + git: ${pathToFileURL(repo).href} + ref: main + path: schemas/nested-flow +` + ); + const nestedDir = path.join(tempDir, 'packages', 'app', 'src'); + fs.mkdirSync(nestedDir, { recursive: true }); + process.chdir(nestedDir); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'sync', + 'nested-flow', + '--json', + ]); + + const consumerRoot = fs.realpathSync.native(tempDir); + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + synced: true, + lockfile: path.join(consumerRoot, 'openspec', 'schemas.lock.yaml'), + schemas: [{ name: 'nested-flow' }], + }); + expect(fs.existsSync(path.join(tempDir, 'openspec', 'schemas.lock.yaml'))).toBe(true); + expect(fs.existsSync(path.join(nestedDir, 'openspec', 'schemas.lock.yaml'))).toBe(false); + expect(process.exitCode).toBeUndefined(); + + log.mockClear(); + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'validate', + 'nested-flow', + '--json', + ]); + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + valid: true, + name: 'nested-flow', + }); + }); + + it('keeps healthy schemas visible when one declared remote is unsynchronized', async () => { + fs.writeFileSync( + path.join(tempDir, 'openspec', 'config.yaml'), + `schema: spec-driven +schemaSources: + unavailable-flow: + git: https://example.com/schemas.git + ref: main + path: schemas/unavailable-flow +` + ); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'which', + '--all', + '--json', + ]); + + const output = JSON.parse(String(log.mock.calls[0][0])); + expect(output).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + name: 'spec-driven', + available: true, + source: 'package', + }), + expect.objectContaining({ + name: 'unavailable-flow', + available: false, + source: 'remote', + path: null, + status: [ + expect.objectContaining({ + code: 'remote_not_locked', + message: expect.stringMatching(/schema sync unavailable-flow/), + }), + ], + }), + ]) + ); + expect(process.exitCode).toBeUndefined(); + }); + + it('rejects forking into a name declared by a remote source', async () => { + fs.writeFileSync( + path.join(tempDir, 'openspec', 'config.yaml'), + `schema: spec-driven +schemaSources: + claimed-flow: + git: https://example.com/schemas.git + ref: main + path: schemas/claimed-flow +` + ); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + + await createProgram().parseAsync([ + 'node', + 'openspec', + 'schema', + 'fork', + 'spec-driven', + 'claimed-flow', + '--json', + ]); + + expect(JSON.parse(String(log.mock.calls[0][0]))).toMatchObject({ + forked: false, + code: 'schema_name_conflict', + error: expect.stringMatching(/claimed-flow/), + }); + expect( + fs.existsSync(path.join(tempDir, 'openspec', 'schemas', 'claimed-flow')) + ).toBe(false); + expect(process.exitCode).toBe(1); + }); + + it('preserves conflict codes in schema discovery JSON', async () => { + const local = path.join( + tempDir, + 'openspec', + 'schemas', + 'claimed-flow' + ); + fs.mkdirSync(local, { recursive: true }); + fs.writeFileSync( + path.join(local, 'schema.yaml'), + 'name: claimed-flow\nversion: 1\nartifacts: []\n' + ); + fs.writeFileSync( + path.join(tempDir, 'openspec', 'config.yaml'), + `schema: spec-driven +schemaSources: + claimed-flow: + git: https://example.com/schemas.git + ref: main + path: schemas/claimed-flow +` + ); + const nested = path.join(tempDir, 'src', 'nested'); + fs.mkdirSync(nested, { recursive: true }); + process.chdir(nested); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + + await schemasCommand({ json: true }); + + expect(JSON.parse(String(log.mock.calls[0][0]))).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + name: 'claimed-flow', + available: false, + status: [ + expect.objectContaining({ + code: 'schema_name_conflict', + }), + ], + }), + ]) + ); + }); + + it('does not leak credentials from a rejected HTTPS declaration', async () => { + const secret = 'never-print-this-token'; + fs.writeFileSync( + path.join(tempDir, 'openspec', 'config.yaml'), + `schema: private-flow +schemaSources: + private-flow: + git: https://oauth2:${secret}@github.com/acme/private.git + ref: main + path: schemas/private-flow +` + ); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + const error = vi.spyOn(console, 'error').mockImplementation(() => {}); + const warn = vi.spyOn(console, 'warn').mockImplementation(() => {}); + + await createProgram().parseAsync(['node', 'openspec', 'schema', 'sync', '--json']); + + expect(log).toHaveBeenCalledTimes(1); + expect( + [...log.mock.calls, ...error.mock.calls, ...warn.mock.calls] + .flat() + .map(String) + .join('\n') + ).not.toContain(secret); + expect(process.exitCode).toBe(1); + }); +}); diff --git a/test/commands/store-root-selection.test.ts b/test/commands/store-root-selection.test.ts index 334c05830f..17c7769c0a 100644 --- a/test/commands/store-root-selection.test.ts +++ b/test/commands/store-root-selection.test.ts @@ -1,7 +1,9 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { execFileSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; +import { pathToFileURL } from 'node:url'; import { getGlobalDataDir, @@ -34,6 +36,22 @@ const REMOVED_ONLY_DELTA_SPEC = `## REMOVED Requirements ### Requirement: Old billing SHALL go away `; +const gitEnv = { + ...process.env, + GIT_AUTHOR_NAME: 'OpenSpec Test', + GIT_AUTHOR_EMAIL: 'openspec@example.test', + GIT_COMMITTER_NAME: 'OpenSpec Test', + GIT_COMMITTER_EMAIL: 'openspec@example.test', +}; + +function git(cwd: string, ...args: string[]): void { + execFileSync('git', args, { + cwd, + env: gitEnv, + stdio: ['ignore', 'pipe', 'pipe'], + }); +} + // MODIFIED deltas against a spec that does not exist make buildUpdatedSpec // throw during the prepare pass. const MODIFIED_ONLY_DELTA_SPEC = `## MODIFIED Requirements @@ -181,6 +199,193 @@ describe('store root selection for normal commands', () => { expect(json.root.store_id).toBe('team-context'); }); + it('keeps schema ownership in the consumer repository when planning uses a store', async () => { + const localRepo = path.join(tempDir, 'schema-consumer'); + const schemaDir = path.join( + localRepo, + 'openspec', + 'schemas', + 'consumer-flow' + ); + createOpenSpecRoot(localRepo); + fs.mkdirSync(path.join(schemaDir, 'templates'), { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: consumer-flow +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: proposal.md + requires: [] +` + ); + fs.writeFileSync( + path.join(schemaDir, 'templates', 'proposal.md'), + '# Proposal\n' + ); + const nested = path.join(localRepo, 'src', 'nested'); + fs.mkdirSync(nested, { recursive: true }); + + const result = await runCLI( + [ + 'new', + 'change', + 'consumer-schema-change', + '--schema', + 'consumer-flow', + '--store', + 'team-context', + '--json', + ], + { cwd: nested, env } + ); + + expect(result.exitCode).toBe(0); + expect(parseJson(result).change).toMatchObject({ + schema: 'consumer-flow', + path: path.join( + storeRoot, + 'openspec', + 'changes', + 'consumer-schema-change' + ), + }); + expect( + fs.existsSync( + path.join( + storeRoot, + 'openspec', + 'changes', + 'consumer-schema-change', + '.openspec.yaml' + ) + ) + ).toBe(true); + + const status = await runCLI( + [ + 'status', + '--change', + 'consumer-schema-change', + '--store', + 'team-context', + '--json', + ], + { cwd: nested, env } + ); + expect(status.exitCode).toBe(0); + expect(parseJson(status).schemaName).toBe('consumer-flow'); + + const instructions = await runCLI( + [ + 'instructions', + 'proposal', + '--change', + 'consumer-schema-change', + '--store', + 'team-context', + '--json', + ], + { cwd: nested, env } + ); + expect(instructions.exitCode).toBe(0); + expect(parseJson(instructions)).toMatchObject({ + schemaName: 'consumer-flow', + template: '# Proposal\n', + }); + }); + + it('keeps remote schema resolution in the consumer while instructions use store context', async () => { + const localRepo = path.join(tempDir, 'remote-schema-consumer'); + const remoteRepo = path.join(tempDir, 'remote-schema-source'); + const remoteSchemaDir = path.join(remoteRepo, 'schemas', 'remote-flow'); + createOpenSpecRoot(localRepo); + fs.mkdirSync(path.join(remoteSchemaDir, 'templates'), { recursive: true }); + fs.writeFileSync( + path.join(remoteSchemaDir, 'schema.yaml'), + `name: remote-flow +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Remote proposal + template: proposal.md + requires: [] +` + ); + fs.writeFileSync( + path.join(remoteSchemaDir, 'templates', 'proposal.md'), + '# Remote Proposal\n' + ); + git(remoteRepo, 'init', '-b', 'main'); + git(remoteRepo, 'add', '-A'); + git(remoteRepo, 'commit', '-m', 'remote schema'); + + fs.writeFileSync( + path.join(localRepo, 'openspec', 'config.yaml'), + `schema: remote-flow +schemaSources: + remote-flow: + git: ${pathToFileURL(remoteRepo).href} + ref: main + path: schemas/remote-flow +` + ); + fs.writeFileSync( + path.join(storeRoot, 'openspec', 'config.yaml'), + `schema: spec-driven +context: Store planning context +rules: + proposal: + - Keep the store rule +` + ); + + const sync = await runCLI(['schema', 'sync', 'remote-flow', '--json'], { + cwd: localRepo, + env, + }); + expect(sync.exitCode).toBe(0); + expect(parseJson(sync)).toMatchObject({ synced: true }); + expect( + fs.existsSync(path.join(localRepo, 'openspec', 'schemas.lock.yaml')) + ).toBe(true); + expect( + fs.existsSync(path.join(storeRoot, 'openspec', 'schemas.lock.yaml')) + ).toBe(false); + + const nested = path.join(localRepo, 'src', 'nested'); + fs.mkdirSync(nested, { recursive: true }); + const created = await runCLI( + ['new', 'change', 'remote-schema-change', '--store', 'team-context', '--json'], + { cwd: nested, env } + ); + expect(created.exitCode).toBe(0); + expect(parseJson(created).change).toMatchObject({ schema: 'remote-flow' }); + + const instructions = await runCLI( + [ + 'instructions', + 'proposal', + '--change', + 'remote-schema-change', + '--store', + 'team-context', + '--json', + ], + { cwd: nested, env } + ); + expect(instructions.exitCode).toBe(0); + expect(parseJson(instructions)).toMatchObject({ + schemaName: 'remote-flow', + template: '# Remote Proposal\n', + context: 'Store planning context', + rules: ['Keep the store rule'], + }); + }); + it('lists an empty team store before any changes exist', async () => { const blankStoreRoot = path.join(tempDir, 'stores', 'blank-context'); fs.mkdirSync(path.join(blankStoreRoot, 'openspec'), { recursive: true }); diff --git a/test/core/artifact-graph/remote-resolver.test.ts b/test/core/artifact-graph/remote-resolver.test.ts new file mode 100644 index 0000000000..e5dc466921 --- /dev/null +++ b/test/core/artifact-graph/remote-resolver.test.ts @@ -0,0 +1,168 @@ +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { + getSchemaDir, + listSchemas, + listSchemasWithInfo, +} from '../../../src/core/artifact-graph/resolver.js'; +import { computeBundleIntegrity } from '../../../src/core/remote-schema/bundle.js'; +import { installRemoteSchemaCache } from '../../../src/core/remote-schema/cache.js'; +import { writeSchemaLock } from '../../../src/core/remote-schema/lockfile.js'; + +function writeSchema(dir: string, name: string, marker: string): void { + fs.mkdirSync(path.join(dir, 'templates'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'schema.yaml'), + `name: ${name} +version: 1 +description: ${marker} +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: proposal.md + requires: [] +` + ); + fs.writeFileSync(path.join(dir, 'templates', 'proposal.md'), `# ${marker}\n`); +} + +describe('remote schema resolver', () => { + let tempDir: string; + let projectRoot: string; + let originalEnv: NodeJS.ProcessEnv; + let integrity: string; + let cacheDir: string; + + beforeEach(() => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-remote-resolver-')); + projectRoot = path.join(tempDir, 'project'); + originalEnv = { ...process.env }; + process.env.XDG_DATA_HOME = path.join(tempDir, 'data'); + fs.mkdirSync(path.join(projectRoot, 'openspec'), { recursive: true }); + fs.writeFileSync( + path.join(projectRoot, 'openspec', 'config.yaml'), + `schema: team-flow +schemaSources: + team-flow: + git: https://example.test/team.git + ref: main + path: schemas/team-flow +` + ); + const source = path.join(tempDir, 'bundle'); + writeSchema(source, 'team-flow', 'remote'); + integrity = computeBundleIntegrity(source).integrity; + cacheDir = installRemoteSchemaCache(source, integrity); + writeSchemaLock(projectRoot, { + version: 1, + schemas: { + 'team-flow': { + git: 'https://example.test/team.git', + requestedRef: 'main', + resolvedCommit: 'a'.repeat(40), + bundlePath: 'schemas/team-flow', + integrity, + }, + }, + }); + }); + + afterEach(() => { + process.env = originalEnv; + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + it('resolves declared remote, undeclared user, and package schemas', () => { + const user = path.join(process.env.XDG_DATA_HOME!, 'openspec', 'schemas', 'team-flow'); + writeSchema(user, 'team-flow', 'user'); + expect(getSchemaDir('team-flow', projectRoot)).toBe(cacheDir); + + const local = path.join(projectRoot, 'openspec', 'schemas', 'team-flow'); + writeSchema(local, 'team-flow', 'project'); + expect(() => getSchemaDir('team-flow', projectRoot)).toThrow( + /project-local schema.*conflicts with declared remote schema/i + ); + expect(getSchemaDir('team-flow')).toBe(user); + expect(getSchemaDir('spec-driven', projectRoot)).toContain( + path.join('schemas', 'spec-driven') + ); + }); + + it.each([ + ['missing lock', () => fs.rmSync(path.join(projectRoot, 'openspec', 'schemas.lock.yaml'))], + ['missing cache', () => fs.rmSync(cacheDir, { recursive: true })], + [ + 'digest mismatch', + () => fs.appendFileSync(path.join(cacheDir, 'templates', 'proposal.md'), 'tampered'), + ], + ])('fails closed on %s instead of using a same-named user schema', (_name, breakState) => { + const user = path.join(process.env.XDG_DATA_HOME!, 'openspec', 'schemas', 'team-flow'); + writeSchema(user, 'team-flow', 'user'); + breakState(); + expect(() => getSchemaDir('team-flow', projectRoot)).toThrow( + /openspec schema sync/ + ); + }); + + it('fails closed when config and lock source metadata drift', () => { + const configPath = path.join(projectRoot, 'openspec', 'config.yaml'); + fs.writeFileSync( + configPath, + fs.readFileSync(configPath, 'utf8').replace('ref: main', 'ref: next') + ); + expect(() => getSchemaDir('team-flow', projectRoot)).toThrow( + /does not match.*schema sync/ + ); + }); + + it('discovers a locked remote schema and reports its source once', () => { + expect(listSchemas(projectRoot).filter((name) => name === 'team-flow')).toEqual([ + 'team-flow', + ]); + expect(listSchemasWithInfo(projectRoot)).toContainEqual({ + name: 'team-flow', + description: 'remote', + artifacts: ['proposal'], + source: 'remote', + }); + }); + + it('reports an unsynchronized declaration as unavailable', () => { + fs.rmSync(path.join(projectRoot, 'openspec', 'schemas.lock.yaml')); + expect(listSchemasWithInfo(projectRoot)).toContainEqual({ + name: 'team-flow', + description: '', + artifacts: [], + source: 'remote', + available: false, + error: expect.stringMatching(/schema sync/), + status: [ + expect.objectContaining({ + code: 'remote_not_locked', + }), + ], + }); + }); + + it('reports a project-local bundle with a declared remote name as unavailable', () => { + const local = path.join(projectRoot, 'openspec', 'schemas', 'team-flow'); + writeSchema(local, 'team-flow', 'project'); + + expect(listSchemasWithInfo(projectRoot)).toContainEqual({ + name: 'team-flow', + description: '', + artifacts: [], + source: 'remote', + available: false, + error: expect.stringMatching(/project-local schema.*conflicts/i), + status: [ + expect.objectContaining({ + code: 'schema_name_conflict', + }), + ], + }); + }); +}); diff --git a/test/core/artifact-graph/schema-directory.test.ts b/test/core/artifact-graph/schema-directory.test.ts new file mode 100644 index 0000000000..024cd60572 --- /dev/null +++ b/test/core/artifact-graph/schema-directory.test.ts @@ -0,0 +1,190 @@ +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { + SchemaDirectoryValidationError, + validateLocalSchemaDirectory, + validateRemoteSchemaDirectory, + validateSchemaDirectory, +} from '../../../src/core/artifact-graph/schema-directory.js'; + +function schemaYaml(name: string, template = 'proposal.md'): string { + return `name: ${name} +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: ${template} + requires: [] +`; +} + +describe('validateSchemaDirectory', () => { + let schemaDir: string; + + beforeEach(() => { + schemaDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-schema-directory-')); + }); + + afterEach(() => { + fs.rmSync(schemaDir, { recursive: true, force: true }); + }); + + it('returns a parsed remote schema only when templates stay in templates/', () => { + fs.mkdirSync(path.join(schemaDir, 'templates')); + fs.writeFileSync(path.join(schemaDir, 'schema.yaml'), schemaYaml('qeda-sdd')); + fs.writeFileSync(path.join(schemaDir, 'templates', 'proposal.md'), '# Proposal\n'); + + expect( + validateSchemaDirectory(schemaDir, { + expectedName: 'qeda-sdd', + requireTemplatesDirectory: true, + }).schema.name + ).toBe('qeda-sdd'); + }); + + it('accepts nested template paths inside templates/', () => { + fs.mkdirSync(path.join(schemaDir, 'templates', 'nested'), { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + schemaYaml('qeda-sdd', 'nested/proposal.md') + ); + fs.writeFileSync( + path.join(schemaDir, 'templates', 'nested', 'proposal.md'), + '# Proposal\n' + ); + + expect( + validateSchemaDirectory(schemaDir, { + expectedName: 'qeda-sdd', + requireTemplatesDirectory: true, + }).templatePaths.proposal + ).toBe(path.join(schemaDir, 'templates', 'nested', 'proposal.md')); + }); + + it('rejects project-local template lookup outside templates/', () => { + const sharedTemplate = path.join(path.dirname(schemaDir), 'shared-proposal.md'); + fs.writeFileSync(sharedTemplate, '# Shared\n'); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + schemaYaml('local-flow', '../shared-proposal.md') + ); + + try { + expect(() => validateLocalSchemaDirectory(schemaDir)).toThrow( + /relative path inside its allowed directory/ + ); + } finally { + fs.rmSync(sharedTemplate, { force: true }); + } + }); + + it('preserves every strict remote validation issue', () => { + fs.mkdirSync(path.join(schemaDir, 'templates')); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: wrong-name +version: 1 +artifacts: + - id: escaped + generates: escaped.md + description: Escaped + template: ../escaped.md + requires: [] + - id: missing + generates: missing.md + description: Missing + template: missing.md + requires: [] +` + ); + + try { + validateRemoteSchemaDirectory(schemaDir, 'declared-name'); + throw new Error('expected remote validation to fail'); + } catch (error) { + expect(error).toBeInstanceOf(SchemaDirectoryValidationError); + const validationError = error as SchemaDirectoryValidationError; + expect(validationError.issues).toEqual([ + expect.objectContaining({ + message: expect.stringMatching(/relative path inside its allowed directory/), + }), + ]); + } + }); + + it.each([ + ['missing schema', false, true, 'proposal.md', /schema.yaml not found/], + ['missing templates directory', true, false, 'proposal.md', /templates directory not found/], + ['missing referenced template', true, true, 'missing.md', /Template file 'missing.md' not found/], + ['parent template escape', true, true, '../outside.md', /relative path inside its allowed directory/], + ['absolute template escape', true, true, '/outside.md', /relative path inside its allowed directory/], + ['backslash template path', true, true, String.raw`nested\proposal.md`, /unsafe template path/], + ])('rejects %s', (_name, withSchema, withTemplates, template, expected) => { + if (withSchema) { + fs.writeFileSync(path.join(schemaDir, 'schema.yaml'), schemaYaml('qeda-sdd', template)); + } + if (withTemplates) { + fs.mkdirSync(path.join(schemaDir, 'templates')); + } + expect(() => + validateSchemaDirectory(schemaDir, { + expectedName: 'qeda-sdd', + requireTemplatesDirectory: true, + }) + ).toThrow(expected); + }); + + it('rejects schema and template symlinks', (ctx) => { + const outsideDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-schema-outside-')); + fs.writeFileSync(path.join(outsideDir, 'schema.yaml'), schemaYaml('qeda-sdd')); + fs.writeFileSync(path.join(outsideDir, 'proposal.md'), '# Proposal\n'); + try { + fs.symlinkSync(path.join(outsideDir, 'schema.yaml'), path.join(schemaDir, 'schema.yaml')); + fs.mkdirSync(path.join(schemaDir, 'templates')); + fs.symlinkSync( + path.join(outsideDir, 'proposal.md'), + path.join(schemaDir, 'templates', 'proposal.md') + ); + } catch { + fs.rmSync(outsideDir, { recursive: true, force: true }); + ctx.skip(); + return; + } + + try { + expect(() => + validateSchemaDirectory(schemaDir, { + expectedName: 'qeda-sdd', + requireTemplatesDirectory: true, + }) + ).toThrow(/schema.yaml must be a regular file/); + + fs.rmSync(path.join(schemaDir, 'schema.yaml')); + fs.writeFileSync(path.join(schemaDir, 'schema.yaml'), schemaYaml('qeda-sdd')); + expect(() => + validateSchemaDirectory(schemaDir, { + expectedName: 'qeda-sdd', + requireTemplatesDirectory: true, + }) + ).toThrow(/Template file 'proposal.md' not found/); + } finally { + fs.rmSync(outsideDir, { recursive: true, force: true }); + } + }); + + it('rejects a bundle whose parsed schema name conflicts with the declaration', () => { + fs.mkdirSync(path.join(schemaDir, 'templates')); + fs.writeFileSync(path.join(schemaDir, 'schema.yaml'), schemaYaml('other-name')); + fs.writeFileSync(path.join(schemaDir, 'templates', 'proposal.md'), '# Proposal\n'); + + expect(() => + validateSchemaDirectory(schemaDir, { + expectedName: 'qeda-sdd', + requireTemplatesDirectory: true, + }) + ).toThrow(/declared as 'qeda-sdd'.*name is 'other-name'/); + }); +}); diff --git a/test/core/project-config.test.ts b/test/core/project-config.test.ts index 2adbdf9ad6..be910b2dcf 100644 --- a/test/core/project-config.test.ts +++ b/test/core/project-config.test.ts @@ -638,6 +638,156 @@ rules: }); }); + describe('schemaSources parsing', () => { + it('parses a valid Git-backed schema source without changing schema selection', () => { + const configDir = path.join(tempDir, 'openspec'); + fs.mkdirSync(configDir, { recursive: true }); + fs.writeFileSync( + path.join(configDir, 'config.yaml'), + `schema: qeda-sdd +schemaSources: + qeda-sdd: + git: https://github.com/example/QEDASDD.git + ref: v1.0.0 + path: schemas/qeda-sdd +` + ); + + const config = readProjectConfig(tempDir); + + expect(config).toEqual({ + schema: 'qeda-sdd', + schemaSources: { + 'qeda-sdd': { + git: 'https://github.com/example/QEDASDD.git', + ref: 'v1.0.0', + path: 'schemas/qeda-sdd', + }, + }, + }); + expect(consoleWarnSpy).not.toHaveBeenCalled(); + }); + + it('keeps valid sources and unrelated fields when other declarations are invalid', () => { + const configDir = path.join(tempDir, 'openspec'); + fs.mkdirSync(configDir, { recursive: true }); + fs.writeFileSync( + path.join(configDir, 'config.yaml'), + `schema: qeda-sdd +context: Keep this +schemaSources: + Valid_Name: + git: https://example.com/invalid-name.git + ref: main + path: schema + missing-ref: + git: https://example.com/missing-ref.git + path: schema + qeda-sdd: + git: git@github.com:example/QEDASDD.git + ref: main + path: schemas/qeda-sdd +` + ); + + expect(readProjectConfig(tempDir)).toEqual({ + schema: 'qeda-sdd', + context: 'Keep this', + schemaSources: { + 'qeda-sdd': { + git: 'git@github.com:example/QEDASDD.git', + ref: 'main', + path: 'schemas/qeda-sdd', + }, + }, + }); + expect(consoleWarnSpy).toHaveBeenCalledWith( + expect.stringContaining("Invalid schema source name 'Valid_Name'") + ); + expect(consoleWarnSpy).toHaveBeenCalledWith( + expect.stringContaining("Invalid 'ref' for schema source 'missing-ref'") + ); + }); + + it('rejects credential-bearing HTTPS URLs without repeating the secret', () => { + const configDir = path.join(tempDir, 'openspec'); + fs.mkdirSync(configDir, { recursive: true }); + fs.writeFileSync( + path.join(configDir, 'config.yaml'), + `schema: spec-driven +schemaSources: + private-flow: + git: https://oauth2:super-secret-token@example.com/team/schema.git + ref: main + path: schema +` + ); + + expect(readProjectConfig(tempDir)).toEqual({ schema: 'spec-driven' }); + const warnings = consoleWarnSpy.mock.calls.flat().join('\n'); + expect(warnings).toContain("Credentials are not allowed in Git URL for schema source 'private-flow'"); + expect(warnings).not.toContain('super-secret-token'); + }); + + it('rejects Git remote-helper transports without invoking them', () => { + const configDir = path.join(tempDir, 'openspec'); + fs.mkdirSync(configDir, { recursive: true }); + fs.writeFileSync( + path.join(configDir, 'config.yaml'), + `schema: spec-driven +schemaSources: + unsafe-flow: + git: ext::malicious-helper + ref: main + path: schema +` + ); + + expect(readProjectConfig(tempDir)).toEqual({ schema: 'spec-driven' }); + expect(consoleWarnSpy).toHaveBeenCalledWith( + expect.stringContaining("Unsupported Git source for schema source 'unsafe-flow'") + ); + }); + + it('rejects prototype keys explicitly without mutating object prototypes', () => { + const configDir = path.join(tempDir, 'openspec'); + fs.mkdirSync(configDir, { recursive: true }); + fs.writeFileSync( + path.join(configDir, 'config.yaml'), + `schema: spec-driven +schemaSources: + __proto__: + git: https://example.com/proto.git + ref: main + path: schema + constructor: + git: https://example.com/constructor.git + ref: main + path: schema + prototype: + git: https://example.com/prototype.git + ref: main + path: schema +` + ); + + expect(readProjectConfig(tempDir)).toEqual({ schema: 'spec-driven' }); + const objectPrototype = {} as Record; + expect(objectPrototype.git).toBeUndefined(); + expect(objectPrototype.ref).toBeUndefined(); + expect(objectPrototype.path).toBeUndefined(); + expect(consoleWarnSpy).toHaveBeenCalledWith( + expect.stringContaining("Invalid schema source name '__proto__'") + ); + expect(consoleWarnSpy).toHaveBeenCalledWith( + expect.stringContaining("Invalid schema source name 'constructor'") + ); + expect(consoleWarnSpy).toHaveBeenCalledWith( + expect.stringContaining("Invalid schema source name 'prototype'") + ); + }); + }); + describe('context size limit enforcement', () => { it('should accept context under 50KB limit', () => { const configDir = path.join(tempDir, 'openspec'); diff --git a/test/core/remote-schema/bundle-size-race.test.ts b/test/core/remote-schema/bundle-size-race.test.ts new file mode 100644 index 0000000000..7147436941 --- /dev/null +++ b/test/core/remote-schema/bundle-size-race.test.ts @@ -0,0 +1,43 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import * as actualFs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +const mockedFile = vi.hoisted(() => ({ path: '' })); + +vi.mock('node:fs', async (importOriginal) => { + const fs = await importOriginal(); + return { + ...fs, + statSync(filePath: fs.PathLike, options?: fs.StatOptions) { + const stat = fs.statSync(filePath, options); + if (String(filePath) === mockedFile.path) { + return { ...stat, size: 1 }; + } + return stat; + }, + }; +}); + +import { computeBundleIntegrity } from '../../../src/core/remote-schema/bundle.js'; + +describe('remote schema bundle size race', () => { + let bundleDir: string | undefined; + + afterEach(() => { + if (bundleDir) { + actualFs.rmSync(bundleDir, { recursive: true, force: true }); + } + mockedFile.path = ''; + }); + + it('rechecks bytes read when a file grows after the stat preflight', () => { + bundleDir = actualFs.mkdtempSync(path.join(os.tmpdir(), 'openspec-bundle-race-')); + mockedFile.path = path.join(bundleDir, 'growing.bin'); + actualFs.writeFileSync(mockedFile.path, '123456'); + + expect(() => + computeBundleIntegrity(bundleDir!, { maxFiles: 10, maxBytes: 5 }) + ).toThrow(/more than 5 bytes/); + }); +}); diff --git a/test/core/remote-schema/bundle.test.ts b/test/core/remote-schema/bundle.test.ts new file mode 100644 index 0000000000..0584114fc8 --- /dev/null +++ b/test/core/remote-schema/bundle.test.ts @@ -0,0 +1,153 @@ +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { + assertPortableBundleEntries, + computeBundleIntegrity, + normalizeBundlePath, +} from '../../../src/core/remote-schema/bundle.js'; + +describe('remote schema bundle boundary', () => { + describe('normalizeBundlePath', () => { + it('accepts a repository-relative Git path without platform conversion', () => { + expect(normalizeBundlePath('schemas/qeda-sdd')).toBe('schemas/qeda-sdd'); + }); + + it.each([ + ['parent traversal', '../schemas/qeda-sdd'], + ['nested parent traversal', 'schemas/../qeda-sdd'], + ['POSIX absolute', '/schemas/qeda-sdd'], + ['Windows drive path', 'C:/schemas/qeda-sdd'], + ['Windows drive with backslashes', String.raw`C:\schemas\qeda-sdd`], + ['UNC path', String.raw`\\server\share\qeda-sdd`], + ['backslash separator', String.raw`schemas\qeda-sdd`], + ['dot segment', './schemas/qeda-sdd'], + ['empty path', ''], + ['control character', 'schemas/qeda-sdd\nother'], + ['Windows wildcard', 'schemas/qeda?sdd'], + ['Windows reserved segment', 'schemas/CON/templates'], + ['trailing dot segment', 'schemas/qeda-sdd./templates'], + ])('rejects %s', (_name, value) => { + expect(() => normalizeBundlePath(value)).toThrow(/Invalid schema bundle path/); + }); + }); + + it('rejects portable path collisions independently of host filesystem case rules', () => { + expect(() => + assertPortableBundleEntries([ + 'schema.yaml', + 'templates/Proposal.md', + 'templates/proposal.md', + ]) + ).toThrow(/portable path collision/i); + }); + + it('rejects directory-prefix collisions on case-insensitive filesystems', () => { + expect(() => + assertPortableBundleEntries([ + 'Templates/proposal.md', + 'templates/design.md', + ]) + ).toThrow(/portable path collision/i); + }); +}); + +describe('remote schema bundle integrity', () => { + let bundleDir: string; + + beforeEach(() => { + bundleDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-schema-bundle-')); + fs.mkdirSync(path.join(bundleDir, 'templates'), { recursive: true }); + fs.writeFileSync(path.join(bundleDir, 'schema.yaml'), 'name: example\n'); + fs.writeFileSync(path.join(bundleDir, 'templates', 'proposal.md'), '# Proposal\n'); + }); + + afterEach(() => { + fs.rmSync(bundleDir, { recursive: true, force: true }); + }); + + it('is deterministic and changes when a file is added, removed, or modified', () => { + const first = computeBundleIntegrity(bundleDir); + const second = computeBundleIntegrity(bundleDir); + expect(second).toEqual(first); + expect(first).toMatchObject({ fileCount: 2 }); + expect(first.integrity).toMatch(/^sha256:[0-9a-f]{64}$/); + + fs.writeFileSync(path.join(bundleDir, 'templates', 'design.md'), '# Design\n'); + const added = computeBundleIntegrity(bundleDir); + expect(added.integrity).not.toBe(first.integrity); + + fs.rmSync(path.join(bundleDir, 'templates', 'design.md')); + fs.writeFileSync(path.join(bundleDir, 'templates', 'proposal.md'), '# Changed\n'); + const modified = computeBundleIntegrity(bundleDir); + expect(modified.integrity).not.toBe(first.integrity); + + fs.rmSync(path.join(bundleDir, 'templates', 'proposal.md')); + const removed = computeBundleIntegrity(bundleDir); + expect(removed.integrity).not.toBe(first.integrity); + }); + + it('rejects file-count and byte limits before accepting cache content', () => { + expect(() => computeBundleIntegrity(bundleDir, { maxFiles: 1, maxBytes: 1024 })).toThrow( + /more than 1 files/ + ); + expect(() => computeBundleIntegrity(bundleDir, { maxFiles: 10, maxBytes: 5 })).toThrow( + /more than 5 bytes/ + ); + }); + + it('rejects an oversized file before reading it into memory', (ctx) => { + if (process.platform === 'win32') { + ctx.skip(); + return; + } + const oversizedDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-oversized-')); + const oversized = path.join(oversizedDir, 'oversized.bin'); + fs.writeFileSync(oversized, '123456'); + fs.chmodSync(oversized, 0o000); + + try { + expect(() => + computeBundleIntegrity(oversizedDir, { maxFiles: 10, maxBytes: 5 }) + ).toThrow(/more than 5 bytes/); + } finally { + fs.chmodSync(oversized, 0o600); + fs.rmSync(oversizedDir, { recursive: true, force: true }); + } + }); + + it('enforces byte limits while traversing the bundle', () => { + fs.writeFileSync(path.join(bundleDir, 'a-oversized.bin'), '123456'); + const link = path.join(bundleDir, 'z-link'); + try { + fs.symlinkSync(path.join(bundleDir, 'missing-target'), link, 'file'); + } catch { + // The byte-limit assertion remains valid on hosts that cannot create symlinks. + } + + expect(() => + computeBundleIntegrity(bundleDir, { maxFiles: 10, maxBytes: 5 }) + ).toThrow(/more than 5 bytes/); + }); + + it('rejects symlinks without reading their targets', (ctx) => { + const outsideDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-schema-outside-')); + const outside = path.join(outsideDir, 'secret.txt'); + fs.writeFileSync(outside, 'do-not-read'); + const link = path.join(bundleDir, 'templates', 'linked.md'); + try { + fs.symlinkSync(outside, link, 'file'); + } catch { + fs.rmSync(outsideDir, { recursive: true, force: true }); + ctx.skip(); + return; + } + + try { + expect(() => computeBundleIntegrity(bundleDir)).toThrow(/symbolic link/); + } finally { + fs.rmSync(outsideDir, { recursive: true, force: true }); + } + }); +}); diff --git a/test/core/remote-schema/cache.test.ts b/test/core/remote-schema/cache.test.ts new file mode 100644 index 0000000000..ace5083cc7 --- /dev/null +++ b/test/core/remote-schema/cache.test.ts @@ -0,0 +1,84 @@ +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { + getRemoteSchemaCacheDir, + installRemoteSchemaCache, + verifyRemoteSchemaCache, +} from '../../../src/core/remote-schema/cache.js'; +import { computeBundleIntegrity } from '../../../src/core/remote-schema/bundle.js'; + +describe('remote schema cache', () => { + let tempDir: string; + let dataDir: string; + + beforeEach(() => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-cache-test-')); + dataDir = path.join(tempDir, 'data'); + }); + + afterEach(() => { + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + it('installs a verified bundle in a content-addressed directory', () => { + const source = path.join(tempDir, 'source'); + fs.mkdirSync(source); + fs.writeFileSync(path.join(source, 'schema.yaml'), 'name: demo\nversion: 1\n'); + const { integrity } = computeBundleIntegrity(source); + + const installed = installRemoteSchemaCache(source, integrity, dataDir); + + expect(installed).toBe(getRemoteSchemaCacheDir(integrity, dataDir)); + expect(verifyRemoteSchemaCache(integrity, dataDir)).toBe(installed); + expect(fs.readFileSync(path.join(installed, 'schema.yaml'), 'utf8')).toContain('demo'); + }); + + it('rejects cache content that no longer matches the lock digest', () => { + const source = path.join(tempDir, 'source'); + fs.mkdirSync(source); + fs.writeFileSync(path.join(source, 'schema.yaml'), 'name: demo\nversion: 1\n'); + const { integrity } = computeBundleIntegrity(source); + const installed = installRemoteSchemaCache(source, integrity, dataDir); + fs.appendFileSync(path.join(installed, 'schema.yaml'), '# changed\n'); + + expect(() => verifyRemoteSchemaCache(integrity, dataDir)).toThrow( + /does not match its locked integrity/ + ); + }); + + it('does not replace an existing valid cache entry', () => { + const source = path.join(tempDir, 'source'); + fs.mkdirSync(source); + fs.writeFileSync(path.join(source, 'schema.yaml'), 'name: demo\nversion: 1\n'); + const { integrity } = computeBundleIntegrity(source); + const installed = installRemoteSchemaCache(source, integrity, dataDir); + const preservedTime = new Date('2000-01-01T00:00:00.000Z'); + fs.utimesSync(installed, preservedTime, preservedTime); + + expect(installRemoteSchemaCache(source, integrity, dataDir)).toBe(installed); + expect(fs.statSync(installed).mtimeMs).toBe(preservedTime.getTime()); + }); + + it('atomically replaces a corrupt cache and removes the displaced directory', () => { + const source = path.join(tempDir, 'source'); + fs.mkdirSync(source); + fs.writeFileSync(path.join(source, 'schema.yaml'), 'name: demo\nversion: 1\n'); + const { integrity } = computeBundleIntegrity(source); + const installed = installRemoteSchemaCache(source, integrity, dataDir); + fs.writeFileSync(path.join(installed, 'schema.yaml'), 'corrupt\n'); + + expect(installRemoteSchemaCache(source, integrity, dataDir)).toBe(installed); + expect(verifyRemoteSchemaCache(integrity, dataDir)).toBe(installed); + expect( + fs.readdirSync(path.dirname(installed)).filter((entry) => entry.startsWith('.displaced-')) + ).toEqual([]); + }); + + it('rejects malformed integrity values before constructing a path', () => { + expect(() => getRemoteSchemaCacheDir('../escape', dataDir)).toThrow( + /valid SHA-256 integrity/ + ); + }); +}); diff --git a/test/core/remote-schema/git.test.ts b/test/core/remote-schema/git.test.ts new file mode 100644 index 0000000000..7a8907ec8c --- /dev/null +++ b/test/core/remote-schema/git.test.ts @@ -0,0 +1,327 @@ +import { execFileSync } from 'node:child_process'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { + buildNonInteractiveGitEnvironment, + fetchSchemaBundleFromGit, + verifyLockedCommitIsAncestor, +} from '../../../src/core/remote-schema/git.js'; + +const gitEnv = { + ...process.env, + GIT_AUTHOR_NAME: 'OpenSpec Test', + GIT_AUTHOR_EMAIL: 'openspec@example.test', + GIT_COMMITTER_NAME: 'OpenSpec Test', + GIT_COMMITTER_EMAIL: 'openspec@example.test', +}; + +function git(cwd: string, ...args: string[]): string { + return execFileSync('git', args, { + cwd, + env: gitEnv, + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + }).trim(); +} + +function writeSchema(repo: string, marker: string): void { + const schemaDir = path.join(repo, 'schemas', 'team-flow'); + fs.mkdirSync(path.join(schemaDir, 'templates'), { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: team-flow +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: proposal.md + requires: [] +` + ); + fs.writeFileSync(path.join(schemaDir, 'templates', 'proposal.md'), `# ${marker}\n`); +} + +describe('fetchSchemaBundleFromGit', () => { + let tempDir: string; + let remoteRepo: string; + + beforeEach(() => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-schema-git-')); + remoteRepo = path.join(tempDir, 'remote repo'); + fs.mkdirSync(remoteRepo); + git(remoteRepo, 'init', '-b', 'main'); + writeSchema(remoteRepo, 'Version One'); + git(remoteRepo, 'add', '-A'); + git(remoteRepo, 'commit', '-m', 'version one'); + git(remoteRepo, 'tag', '-a', 'v1.0.0', '-m', 'version one'); + }); + + afterEach(() => { + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + it('resolves a branch to its immutable commit and extracts only the selected tracked tree', async () => { + const destinationDir = path.join(tempDir, 'bundle'); + const expectedCommit = git(remoteRepo, 'rev-parse', 'HEAD'); + + const result = await fetchSchemaBundleFromGit({ + git: pathToFileURL(remoteRepo).href, + requestedRef: 'main', + bundlePath: 'schemas/team-flow', + destinationDir, + }); + + expect(result.resolvedCommit).toBe(expectedCommit); + expect(result.fileCount).toBe(2); + expect(result.integrity).toMatch(/^sha256:[0-9a-f]{64}$/); + expect(fs.readFileSync(path.join(destinationDir, 'templates', 'proposal.md'), 'utf8')).toBe( + '# Version One\n' + ); + expect(fs.existsSync(path.join(destinationDir, '.git'))).toBe(false); + }); + + it('resolves annotated tags and restores an exact locked commit after the branch advances', async () => { + const firstCommit = git(remoteRepo, 'rev-parse', 'HEAD'); + const tagged = await fetchSchemaBundleFromGit({ + git: pathToFileURL(remoteRepo).href, + requestedRef: 'v1.0.0', + bundlePath: 'schemas/team-flow', + destinationDir: path.join(tempDir, 'tagged'), + }); + expect(tagged.resolvedCommit).toBe(firstCommit); + + writeSchema(remoteRepo, 'Version Two'); + git(remoteRepo, 'add', '-A'); + git(remoteRepo, 'commit', '-m', 'version two'); + + const restored = await fetchSchemaBundleFromGit({ + git: pathToFileURL(remoteRepo).href, + requestedRef: 'main', + lockedCommit: firstCommit, + bundlePath: 'schemas/team-flow', + destinationDir: path.join(tempDir, 'restored'), + }); + expect(restored.resolvedCommit).toBe(firstCommit); + expect( + fs.readFileSync(path.join(tempDir, 'restored', 'templates', 'proposal.md'), 'utf8') + ).toBe('# Version One\n'); + }); + + it('resolves a lightweight tag to its commit', async () => { + const expectedCommit = git(remoteRepo, 'rev-parse', 'HEAD'); + git(remoteRepo, 'tag', 'latest-schema'); + const result = await fetchSchemaBundleFromGit({ + git: pathToFileURL(remoteRepo).href, + requestedRef: 'latest-schema', + bundlePath: 'schemas/team-flow', + destinationDir: path.join(tempDir, 'lightweight'), + }); + expect(result.resolvedCommit).toBe(expectedCommit); + }); + + it('rejects a locked commit that is not reachable from the requested ref', async () => { + const mainCommit = git(remoteRepo, 'rev-parse', 'HEAD'); + git(remoteRepo, 'checkout', '-b', 'other'); + writeSchema(remoteRepo, 'Other Branch'); + git(remoteRepo, 'add', '-A'); + git(remoteRepo, 'commit', '-m', 'other branch'); + const otherCommit = git(remoteRepo, 'rev-parse', 'HEAD'); + git(remoteRepo, 'checkout', 'main'); + expect(otherCommit).not.toBe(mainCommit); + + await expect( + fetchSchemaBundleFromGit({ + git: pathToFileURL(remoteRepo).href, + requestedRef: 'main', + lockedCommit: otherCommit, + bundlePath: 'schemas/team-flow', + destinationDir: path.join(tempDir, 'unreachable'), + }) + ).rejects.toThrow(/locked commit verification/i); + }); + + it('directly distinguishes present non-ancestors from missing locked commits', async () => { + const mainCommit = git(remoteRepo, 'rev-parse', 'main'); + git(remoteRepo, 'checkout', '-b', 'other'); + writeSchema(remoteRepo, 'Other Branch'); + git(remoteRepo, 'add', '-A'); + git(remoteRepo, 'commit', '-m', 'other branch'); + const otherCommit = git(remoteRepo, 'rev-parse', 'HEAD'); + git(remoteRepo, 'checkout', 'main'); + + const verificationRepo = path.join(tempDir, 'verification'); + fs.mkdirSync(verificationRepo); + git(verificationRepo, 'init'); + git(verificationRepo, 'remote', 'add', 'origin', pathToFileURL(remoteRepo).href); + git( + verificationRepo, + 'fetch', + 'origin', + 'main:refs/remotes/origin/main', + 'other:refs/remotes/origin/other' + ); + + await expect( + verifyLockedCommitIsAncestor( + verificationRepo, + mainCommit, + 'refs/remotes/origin/main', + 2_000 + ) + ).resolves.toBe(mainCommit); + await expect( + verifyLockedCommitIsAncestor( + verificationRepo, + otherCommit, + 'refs/remotes/origin/main', + 2_000 + ) + ).rejects.toThrow(/not reachable from the requested ref/i); + await expect( + verifyLockedCommitIsAncestor( + verificationRepo, + 'f'.repeat(40), + 'refs/remotes/origin/main', + 2_000 + ) + ).rejects.toThrow(/not present after fetching the requested ref/i); + }); + + it('rejects a tracked symbolic link without reading its target', async () => { + const secret = path.join(tempDir, 'secret.txt'); + fs.writeFileSync(secret, 'never-copy'); + const link = path.join(remoteRepo, 'schemas', 'team-flow', 'templates', 'linked.md'); + try { + fs.symlinkSync(secret, link, 'file'); + } catch { + return; + } + git(remoteRepo, 'add', '-A'); + git(remoteRepo, 'commit', '-m', 'malicious symlink'); + + await expect( + fetchSchemaBundleFromGit({ + git: pathToFileURL(remoteRepo).href, + requestedRef: 'main', + bundlePath: 'schemas/team-flow', + destinationDir: path.join(tempDir, 'unsafe'), + }) + ).rejects.toThrow(/symbolic link/); + expect(fs.existsSync(path.join(tempDir, 'unsafe', 'templates', 'linked.md'))).toBe(false); + }); + + it('rejects a Git submodule entry in the selected bundle', async () => { + const nested = path.join(tempDir, 'nested'); + fs.mkdirSync(nested); + git(nested, 'init', '-b', 'main'); + fs.writeFileSync(path.join(nested, 'README.md'), 'nested\n'); + git(nested, 'add', '-A'); + git(nested, 'commit', '-m', 'nested'); + execFileSync( + 'git', + [ + '-c', + 'protocol.file.allow=always', + 'submodule', + 'add', + pathToFileURL(nested).href, + 'schemas/team-flow/vendor', + ], + { cwd: remoteRepo, env: gitEnv, stdio: ['ignore', 'pipe', 'pipe'] } + ); + git(remoteRepo, 'commit', '-am', 'submodule'); + + await expect( + fetchSchemaBundleFromGit({ + git: pathToFileURL(remoteRepo).href, + requestedRef: 'main', + bundlePath: 'schemas/team-flow', + destinationDir: path.join(tempDir, 'submodule'), + }) + ).rejects.toThrow(/submodule/); + }); + + it('rejects a bundle larger than the byte limit', async () => { + fs.writeFileSync( + path.join(remoteRepo, 'schemas', 'team-flow', 'templates', 'oversized.bin'), + Buffer.alloc(10 * 1024 * 1024 + 1) + ); + git(remoteRepo, 'add', '-A'); + git(remoteRepo, 'commit', '-m', 'oversized'); + + await expect( + fetchSchemaBundleFromGit({ + git: pathToFileURL(remoteRepo).href, + requestedRef: 'main', + bundlePath: 'schemas/team-flow', + destinationDir: path.join(tempDir, 'oversized'), + }) + ).rejects.toThrow(/more than 10485760 bytes/); + }); + + it('does not expose a credential-bearing source or untrusted Git stderr', async () => { + const secret = 'super-secret-token'; + await expect( + fetchSchemaBundleFromGit({ + git: `https://oauth2:${secret}@127.0.0.1:1/private.git`, + requestedRef: 'main', + bundlePath: 'schemas/team-flow', + destinationDir: path.join(tempDir, 'auth-failure'), + timeoutMs: 2_000, + }) + ).rejects.not.toThrow(new RegExp(secret)); + }); + + it('rejects option-like refs before invoking Git fetch', async () => { + await expect( + fetchSchemaBundleFromGit({ + git: pathToFileURL(remoteRepo).href, + requestedRef: '--upload-pack=malicious', + bundlePath: 'schemas/team-flow', + destinationDir: path.join(tempDir, 'option-ref'), + }) + ).rejects.toThrow(/Invalid remote schema ref/); + }); +}); + +describe('buildNonInteractiveGitEnvironment', () => { + it('adds an explicit non-interactive SSH policy', () => { + expect(buildNonInteractiveGitEnvironment({ PATH: '/bin' })).toMatchObject({ + PATH: '/bin', + GIT_TERMINAL_PROMPT: '0', + GIT_SSH_COMMAND: + 'ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new', + }); + }); + + it('preserves existing SSH options and an explicit host-key policy', () => { + const result = buildNonInteractiveGitEnvironment({ + PATH: '/bin', + GIT_SSH_COMMAND: + 'ssh -i "/tmp/key file" -J bastion -o BatchMode=no -o StrictHostKeyChecking=no', + }); + + expect(result.GIT_SSH_COMMAND).toContain('-i "/tmp/key file"'); + expect(result.GIT_SSH_COMMAND).toContain('-J bastion'); + expect(result.GIT_SSH_COMMAND).not.toMatch(/BatchMode=no/i); + expect(result.GIT_SSH_COMMAND).toMatch(/StrictHostKeyChecking=no/i); + expect(result.GIT_SSH_COMMAND).toMatch(/-o BatchMode=yes/); + expect(result.GIT_SSH_COMMAND).not.toMatch(/StrictHostKeyChecking=accept-new/i); + }); + + it('preserves a quoted strict host-key policy while normalizing BatchMode', () => { + const result = buildNonInteractiveGitEnvironment({ + GIT_SSH_COMMAND: + `ssh -i key -o "BatchMode=no" -o 'StrictHostKeyChecking=yes'`, + }); + + expect(result.GIT_SSH_COMMAND).not.toMatch(/BatchMode=no/i); + expect(result.GIT_SSH_COMMAND).toMatch(/StrictHostKeyChecking=yes/i); + expect(result.GIT_SSH_COMMAND).toMatch(/-o BatchMode=yes/); + expect(result.GIT_SSH_COMMAND).not.toMatch(/StrictHostKeyChecking=accept-new/i); + }); +}); diff --git a/test/core/remote-schema/lockfile.test.ts b/test/core/remote-schema/lockfile.test.ts new file mode 100644 index 0000000000..b71d437f70 --- /dev/null +++ b/test/core/remote-schema/lockfile.test.ts @@ -0,0 +1,159 @@ +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { + getSchemaLockPath, + readSchemaLock, + writeSchemaLock, +} from '../../../src/core/remote-schema/lockfile.js'; +import type { RemoteSchemaLock } from '../../../src/core/remote-schema/types.js'; + +describe('remote schema lockfile', () => { + let projectRoot: string; + + beforeEach(() => { + projectRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-schema-lock-')); + fs.mkdirSync(path.join(projectRoot, 'openspec'), { recursive: true }); + }); + + afterEach(() => { + fs.rmSync(projectRoot, { recursive: true, force: true }); + }); + + it('writes entries in deterministic schema-name order and reads them strictly', () => { + const lock: RemoteSchemaLock = { + version: 1, + schemas: { + 'z-flow': { + git: 'git@github.com:example/z.git', + requestedRef: 'main', + resolvedCommit: 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb', + bundlePath: 'schemas/z-flow', + integrity: `sha256:${'2'.repeat(64)}`, + }, + 'a-flow': { + git: 'https://github.com/example/a.git', + requestedRef: 'v1.0.0', + resolvedCommit: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', + bundlePath: 'schemas/a-flow', + integrity: `sha256:${'1'.repeat(64)}`, + }, + }, + }; + + writeSchemaLock(projectRoot, lock); + + expect(fs.readFileSync(getSchemaLockPath(projectRoot), 'utf8')).toBe( + `version: 1 +schemas: + a-flow: + git: https://github.com/example/a.git + requestedRef: v1.0.0 + resolvedCommit: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + bundlePath: schemas/a-flow + integrity: sha256:${'1'.repeat(64)} + z-flow: + git: git@github.com:example/z.git + requestedRef: main + resolvedCommit: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb + bundlePath: schemas/z-flow + integrity: sha256:${'2'.repeat(64)} +` + ); + expect(readSchemaLock(projectRoot)).toEqual({ + version: 1, + schemas: { + 'a-flow': lock.schemas['a-flow'], + 'z-flow': lock.schemas['z-flow'], + }, + }); + }); + + it.each([ + ['unsupported version', `version: 2\nschemas: {}\n`], + [ + 'short commit', + `version: 1 +schemas: + bad: + git: https://example.com/a.git + requestedRef: main + resolvedCommit: abc + bundlePath: schema + integrity: sha256:${'1'.repeat(64)} +`, + ], + [ + 'malformed digest', + `version: 1 +schemas: + bad: + git: https://example.com/a.git + requestedRef: main + resolvedCommit: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + bundlePath: schema + integrity: sha256:not-a-digest +`, + ], + [ + 'credential-bearing source', + `version: 1 +schemas: + bad: + git: https://oauth2:secret@example.com/a.git + requestedRef: main + resolvedCommit: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + bundlePath: schema + integrity: sha256:${'1'.repeat(64)} +`, + ], + [ + 'Git remote-helper source', + `version: 1 +schemas: + bad: + git: ext::malicious + requestedRef: main + resolvedCommit: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + bundlePath: schema + integrity: sha256:${'1'.repeat(64)} +`, + ], + ])('rejects %s metadata', (_name, content) => { + fs.writeFileSync(getSchemaLockPath(projectRoot), content); + + expect(() => readSchemaLock(projectRoot)).toThrow(/Invalid remote schema lockfile/); + }); + + it('returns null when the project has no lockfile', () => { + expect(readSchemaLock(projectRoot)).toBeNull(); + }); + + it('preserves the existing lock when replacement data is invalid', () => { + const existing: RemoteSchemaLock = { + version: 1, + schemas: { + demo: { + git: 'https://example.com/demo.git', + requestedRef: 'main', + resolvedCommit: 'a'.repeat(40), + bundlePath: 'schemas/demo', + integrity: `sha256:${'1'.repeat(64)}`, + }, + }, + }; + writeSchemaLock(projectRoot, existing); + const before = fs.readFileSync(getSchemaLockPath(projectRoot)); + + expect(() => + writeSchemaLock(projectRoot, { + ...existing, + schemas: { + demo: { ...existing.schemas.demo, resolvedCommit: 'not-a-commit' }, + }, + }) + ).toThrow(/Invalid remote schema lockfile data/); + expect(fs.readFileSync(getSchemaLockPath(projectRoot))).toEqual(before); + }); +}); diff --git a/test/core/remote-schema/sync-lock.test.ts b/test/core/remote-schema/sync-lock.test.ts new file mode 100644 index 0000000000..0516de3345 --- /dev/null +++ b/test/core/remote-schema/sync-lock.test.ts @@ -0,0 +1,258 @@ +import { execFileSync, spawn } from 'node:child_process'; +import { once } from 'node:events'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { + getSchemaSyncLockPath, + withSchemaSyncLock, +} from '../../../src/core/remote-schema/sync-lock.js'; + +describe('schema synchronization lock', () => { + let projectRoot: string; + + beforeEach(() => { + projectRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-sync-lock-')); + fs.mkdirSync(path.join(projectRoot, 'openspec'), { recursive: true }); + }); + + function writeTicket(token: string, pid: number): string { + const lockPath = getSchemaSyncLockPath(projectRoot); + fs.mkdirSync(lockPath, { recursive: true }); + fs.writeFileSync( + path.join(lockPath, `claim-${token}.ticket.json`), + JSON.stringify({ + token, + pid, + hostname: os.hostname(), + startedAt: new Date().toISOString(), + number: 1, + }) + ); + return lockPath; + } + + afterEach(() => { + fs.rmSync(projectRoot, { recursive: true, force: true }); + }); + + it('serializes concurrent owners of one project lock', async () => { + const order: string[] = []; + let releaseFirst!: () => void; + const firstCanFinish = new Promise((resolve) => { + releaseFirst = resolve; + }); + + const first = withSchemaSyncLock(projectRoot, async () => { + order.push('first-enter'); + await firstCanFinish; + order.push('first-exit'); + }); + while (order.length === 0) { + await new Promise((resolve) => setTimeout(resolve, 1)); + } + const second = withSchemaSyncLock( + projectRoot, + async () => { + order.push('second-enter'); + }, + { timeoutMs: 500, retryDelayMs: 5 } + ); + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(order).toEqual(['first-enter']); + + releaseFirst(); + await Promise.all([first, second]); + expect(order).toEqual(['first-enter', 'first-exit', 'second-enter']); + }); + + it('does not steal a lock from a live owner', async () => { + const lockPath = writeTicket('live-owner', process.pid); + + await expect( + withSchemaSyncLock(projectRoot, async () => undefined, { + timeoutMs: 20, + retryDelayMs: 5, + }) + ).rejects.toThrow(/schema_sync_locked/); + expect(fs.existsSync(lockPath)).toBe(true); + }); + + it('does not steal a lock held by another process', async () => { + const child = spawn( + process.execPath, + [ + '-e', + ` + const fs = require('node:fs'); + const os = require('node:os'); + const path = require('node:path'); + const lockPath = path.join(process.argv[1], 'openspec', '.schemas.lock'); + fs.mkdirSync(lockPath); + fs.writeFileSync( + path.join(lockPath, 'claim-child-owner.ticket.json'), + JSON.stringify({ + token: 'child-owner', + pid: process.pid, + hostname: os.hostname(), + startedAt: new Date().toISOString(), + number: 1, + }) + ); + process.stdout.write('ready\\n'); + setInterval(() => {}, 1_000); + `, + projectRoot, + ], + { stdio: ['ignore', 'pipe', 'pipe'] } + ); + + try { + await Promise.race([ + once(child.stdout, 'data'), + once(child, 'exit').then(([code]) => { + throw new Error(`lock-holder child exited before ready (${code})`); + }), + ]); + + await expect( + withSchemaSyncLock(projectRoot, async () => undefined, { + timeoutMs: 50, + retryDelayMs: 5, + }) + ).rejects.toThrow(/schema_sync_locked/); + expect(fs.existsSync(getSchemaSyncLockPath(projectRoot))).toBe(true); + } finally { + child.kill(); + if (child.exitCode === null) { + await once(child, 'exit'); + } + } + }); + + it('reclaims an abandoned same-host lock', async () => { + const lockPath = writeTicket('dead-owner', 2_147_483_647); + + await expect( + withSchemaSyncLock(projectRoot, async () => 'acquired', { + timeoutMs: 100, + retryDelayMs: 5, + }) + ).resolves.toBe('acquired'); + expect(fs.readdirSync(lockPath)).toEqual(['.gitignore']); + }); + + it('recovers an aged corrupt ticket without manual deletion', async () => { + const lockPath = getSchemaSyncLockPath(projectRoot); + fs.mkdirSync(lockPath, { recursive: true }); + const corruptPath = path.join(lockPath, 'claim-corrupt.ticket.json'); + fs.writeFileSync(corruptPath, '{"token"'); + const old = new Date(Date.now() - 1_000); + fs.utimesSync(corruptPath, old, old); + + await expect( + withSchemaSyncLock(projectRoot, async () => 'acquired', { + timeoutMs: 50, + retryDelayMs: 5, + }) + ).resolves.toBe('acquired'); + expect(fs.existsSync(corruptPath)).toBe(false); + }); + + it('waits the full timeout before reclaiming a fresh unparseable participant', async () => { + const lockPath = getSchemaSyncLockPath(projectRoot); + fs.mkdirSync(lockPath, { recursive: true }); + const corruptPath = path.join(lockPath, 'claim-corrupt.choosing.json'); + fs.writeFileSync(corruptPath, '{"token"'); + const startedAt = Date.now(); + + await expect( + withSchemaSyncLock(projectRoot, async () => 'acquired', { + timeoutMs: 40, + retryDelayMs: 5, + }) + ).resolves.toBe('acquired'); + expect(Date.now() - startedAt).toBeGreaterThanOrEqual(25); + expect(fs.existsSync(corruptPath)).toBe(false); + }); + + it('reclaims an aged ticket whose bakery number is missing', async () => { + const lockPath = getSchemaSyncLockPath(projectRoot); + fs.mkdirSync(lockPath, { recursive: true }); + const invalidPath = path.join(lockPath, 'claim-numberless.ticket.json'); + fs.writeFileSync( + invalidPath, + JSON.stringify({ + token: 'numberless', + pid: process.pid, + hostname: os.hostname(), + startedAt: new Date().toISOString(), + }) + ); + const old = new Date(Date.now() - 1_000); + fs.utimesSync(invalidPath, old, old); + + await expect( + withSchemaSyncLock(projectRoot, async () => 'acquired', { + timeoutMs: 50, + retryDelayMs: 5, + }) + ).resolves.toBe('acquired'); + expect(fs.existsSync(invalidPath)).toBe(false); + }); + + it('keeps runtime coordination files out of Git status', async () => { + execFileSync('git', ['init', '-q'], { cwd: projectRoot }); + + await withSchemaSyncLock(projectRoot, async () => { + const lockPath = getSchemaSyncLockPath(projectRoot); + expect(fs.readFileSync(path.join(lockPath, '.gitignore'), 'utf8')).toBe('*\n'); + expect( + execFileSync( + 'git', + ['status', '--porcelain', '--untracked-files=all'], + { cwd: projectRoot, encoding: 'utf8' } + ) + ).toBe(''); + }); + + expect( + fs.existsSync( + path.join(getSchemaSyncLockPath(projectRoot), '.gitignore') + ) + ).toBe(true); + }); + + it('repairs an incomplete self-ignore file before publishing participants', async () => { + execFileSync('git', ['init', '-q'], { cwd: projectRoot }); + const lockPath = getSchemaSyncLockPath(projectRoot); + fs.mkdirSync(lockPath, { recursive: true }); + fs.writeFileSync(path.join(lockPath, '.gitignore'), ''); + + await withSchemaSyncLock(projectRoot, async () => { + expect(fs.readFileSync(path.join(lockPath, '.gitignore'), 'utf8')).toBe('*\n'); + expect( + execFileSync( + 'git', + ['status', '--porcelain', '--untracked-files=all'], + { cwd: projectRoot, encoding: 'utf8' } + ) + ).toBe(''); + }); + }); + + it('does not remove a successor lock when ownership changes', async () => { + const lockPath = getSchemaSyncLockPath(projectRoot); + + await withSchemaSyncLock(projectRoot, async () => { + fs.rmSync(lockPath, { recursive: true, force: true }); + writeTicket('successor', process.pid); + }); + + expect(fs.existsSync(lockPath)).toBe(true); + expect(fs.readdirSync(lockPath)).toContain( + 'claim-successor.ticket.json' + ); + }); +}); diff --git a/test/core/remote-schema/sync.test.ts b/test/core/remote-schema/sync.test.ts new file mode 100644 index 0000000000..1f260ff869 --- /dev/null +++ b/test/core/remote-schema/sync.test.ts @@ -0,0 +1,340 @@ +import { execFileSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { readSchemaLock } from '../../../src/core/remote-schema/lockfile.js'; +import { syncRemoteSchemas } from '../../../src/core/remote-schema/sync.js'; +import { getSchemaDir } from '../../../src/core/artifact-graph/resolver.js'; + +const gitEnv = { + ...process.env, + GIT_AUTHOR_NAME: 'OpenSpec Test', + GIT_AUTHOR_EMAIL: 'openspec@example.test', + GIT_COMMITTER_NAME: 'OpenSpec Test', + GIT_COMMITTER_EMAIL: 'openspec@example.test', +}; + +function git(cwd: string, ...args: string[]): string { + return execFileSync('git', args, { + cwd, + env: gitEnv, + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + }).trim(); +} + +function writeSchema(repo: string, name: string, marker: string): void { + const schemaDir = path.join(repo, 'schemas', name); + fs.mkdirSync(path.join(schemaDir, 'templates'), { recursive: true }); + fs.writeFileSync( + path.join(schemaDir, 'schema.yaml'), + `name: ${name} +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: proposal.md + requires: [] +` + ); + fs.writeFileSync(path.join(schemaDir, 'templates', 'proposal.md'), `# ${marker}\n`); +} + +describe('syncRemoteSchemas', () => { + let tempDir: string; + let projectRoot: string; + let repo: string; + let globalDataDir: string; + let originalEnv: NodeJS.ProcessEnv; + + beforeEach(() => { + originalEnv = { ...process.env }; + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'openspec-schema-sync-')); + projectRoot = path.join(tempDir, 'project'); + repo = path.join(tempDir, 'remote'); + globalDataDir = path.join(tempDir, 'data', 'openspec'); + fs.mkdirSync(path.join(projectRoot, 'openspec'), { recursive: true }); + fs.mkdirSync(repo); + git(repo, 'init', '-b', 'main'); + writeSchema(repo, 'team-flow', 'one'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'one'); + fs.writeFileSync( + path.join(projectRoot, 'openspec', 'config.yaml'), + `schema: team-flow +schemaSources: + team-flow: + git: ${pathToFileURL(repo).href} + ref: main + path: schemas/team-flow +` + ); + }); + + afterEach(() => { + process.env = originalEnv; + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + it('locks a moving ref to a commit and installs a verified cache entry', async () => { + const result = await syncRemoteSchemas(projectRoot, { globalDataDir }); + const lock = readSchemaLock(projectRoot); + + expect(result.schemas).toHaveLength(1); + expect(result.schemas[0].resolvedCommit).toBe(git(repo, 'rev-parse', 'HEAD')); + expect(lock?.schemas['team-flow']).toMatchObject({ + requestedRef: 'main', + bundlePath: 'schemas/team-flow', + }); + expect(fs.readFileSync(path.join(result.schemas[0].cachePath, 'templates', 'proposal.md'), 'utf8')) + .toBe('# one\n'); + }); + + it('rejects synchronization when a project-local schema owns the declared remote name', async () => { + const localSchema = path.join(projectRoot, 'openspec', 'schemas', 'team-flow'); + fs.cpSync(path.join(repo, 'schemas', 'team-flow'), localSchema, { + recursive: true, + }); + + await expect(syncRemoteSchemas(projectRoot, { globalDataDir })).rejects.toThrow( + /project-local schema.*conflicts with declared remote schema/i + ); + expect(readSchemaLock(projectRoot)).toBeNull(); + }); + + it('preserves both lock entries when named synchronizations run concurrently', async () => { + writeSchema(repo, 'review-flow', 'review'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'review flow'); + fs.writeFileSync( + path.join(projectRoot, 'openspec', 'config.yaml'), + `schema: team-flow +schemaSources: + team-flow: + git: ${pathToFileURL(repo).href} + ref: main + path: schemas/team-flow + review-flow: + git: ${pathToFileURL(repo).href} + ref: main + path: schemas/review-flow +` + ); + + await Promise.all([ + syncRemoteSchemas(projectRoot, { name: 'team-flow', globalDataDir }), + syncRemoteSchemas(projectRoot, { name: 'review-flow', globalDataDir }), + ]); + + expect(Object.keys(readSchemaLock(projectRoot)?.schemas ?? {}).sort()).toEqual([ + 'review-flow', + 'team-flow', + ]); + }); + + it('keeps using the old lock until an explicit update and supports locked restoration', async () => { + const first = await syncRemoteSchemas(projectRoot, { globalDataDir }); + const lockPath = path.join(projectRoot, 'openspec', 'schemas.lock.yaml'); + const firstLock = fs.readFileSync(lockPath); + fs.rmSync(first.schemas[0].cachePath, { recursive: true }); + + writeSchema(repo, 'team-flow', 'two'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'two'); + + const restored = await syncRemoteSchemas(projectRoot, { + locked: true, + globalDataDir, + }); + expect(fs.readFileSync(path.join(restored.schemas[0].cachePath, 'templates', 'proposal.md'), 'utf8')) + .toBe('# one\n'); + expect(fs.readFileSync(lockPath)).toEqual(firstLock); + + const upgraded = await syncRemoteSchemas(projectRoot, { globalDataDir }); + expect(upgraded.schemas[0].resolvedCommit).toBe(git(repo, 'rev-parse', 'HEAD')); + expect(upgraded.schemas[0].resolvedCommit).not.toBe(first.schemas[0].resolvedCommit); + }); + + it('resolves offline from the old lock after its branch advances', async () => { + const first = await syncRemoteSchemas(projectRoot, { globalDataDir }); + process.env.XDG_DATA_HOME = path.dirname(globalDataDir); + writeSchema(repo, 'team-flow', 'two'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'two'); + const originalPath = process.env.PATH; + process.env.PATH = path.join(tempDir, 'no-programs'); + try { + const resolved = getSchemaDir('team-flow', projectRoot); + expect(resolved).toBe(first.schemas[0].cachePath); + expect(fs.readFileSync(path.join(resolved!, 'templates', 'proposal.md'), 'utf8')) + .toBe('# one\n'); + } finally { + process.env.PATH = originalPath; + } + }); + + it('repairs a corrupt locked cache entry without changing the lock', async () => { + const first = await syncRemoteSchemas(projectRoot, { globalDataDir }); + const lockPath = path.join(projectRoot, 'openspec', 'schemas.lock.yaml'); + const firstLock = fs.readFileSync(lockPath); + fs.appendFileSync( + path.join(first.schemas[0].cachePath, 'templates', 'proposal.md'), + 'tampered' + ); + + const restored = await syncRemoteSchemas(projectRoot, { + locked: true, + globalDataDir, + }); + + expect(fs.readFileSync(lockPath)).toEqual(firstLock); + expect( + fs.readFileSync( + path.join(restored.schemas[0].cachePath, 'templates', 'proposal.md'), + 'utf8' + ) + ).toBe('# one\n'); + }); + + it('syncs one selected source and rejects unknown names', async () => { + writeSchema(repo, 'second-flow', 'second'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'second'); + fs.appendFileSync( + path.join(projectRoot, 'openspec', 'config.yaml'), + ` second-flow: + git: ${pathToFileURL(repo).href} + ref: main + path: schemas/second-flow +` + ); + + const result = await syncRemoteSchemas(projectRoot, { + name: 'second-flow', + globalDataDir, + }); + expect(result.schemas.map((entry) => entry.name)).toEqual(['second-flow']); + await expect( + syncRemoteSchemas(projectRoot, { name: 'missing', globalDataDir }) + ).rejects.toThrow(/not declared/); + }); + + it('preserves the old lock and cache when an upgrade is invalid', async () => { + const first = await syncRemoteSchemas(projectRoot, { globalDataDir }); + const lockPath = path.join(projectRoot, 'openspec', 'schemas.lock.yaml'); + const firstLock = fs.readFileSync(lockPath); + fs.rmSync(path.join(repo, 'schemas', 'team-flow', 'templates'), { + recursive: true, + }); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'invalid'); + + await expect(syncRemoteSchemas(projectRoot, { globalDataDir })).rejects.toThrow( + /templates directory not found/ + ); + expect(fs.readFileSync(lockPath)).toEqual(firstLock); + expect(fs.existsSync(first.schemas[0].cachePath)).toBe(true); + }); + + it('does not partially replace the lock when a later source fails', async () => { + await syncRemoteSchemas(projectRoot, { globalDataDir }); + const lockPath = path.join(projectRoot, 'openspec', 'schemas.lock.yaml'); + const firstLock = fs.readFileSync(lockPath); + writeSchema(repo, 'team-flow', 'two'); + const brokenDir = path.join(repo, 'schemas', 'z-broken'); + fs.mkdirSync(brokenDir, { recursive: true }); + fs.writeFileSync( + path.join(brokenDir, 'schema.yaml'), + `name: z-broken +version: 1 +artifacts: + - id: proposal + generates: proposal.md + description: Proposal + template: proposal.md + requires: [] +` + ); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'partial failure'); + fs.appendFileSync( + path.join(projectRoot, 'openspec', 'config.yaml'), + ` z-broken: + git: ${pathToFileURL(repo).href} + ref: main + path: schemas/z-broken +` + ); + + await expect(syncRemoteSchemas(projectRoot, { globalDataDir })).rejects.toThrow( + /templates directory not found/ + ); + expect(fs.readFileSync(lockPath)).toEqual(firstLock); + }); + + it('preserves the active lock and cache when cache installation fails', async () => { + const first = await syncRemoteSchemas(projectRoot, { globalDataDir }); + const lockPath = path.join(projectRoot, 'openspec', 'schemas.lock.yaml'); + const firstLock = fs.readFileSync(lockPath); + writeSchema(repo, 'team-flow', 'two'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-m', 'two'); + const unusableDataDir = path.join(tempDir, 'not-a-directory'); + fs.writeFileSync(unusableDataDir, 'file'); + + await expect( + syncRemoteSchemas(projectRoot, { globalDataDir: unusableDataDir }) + ).rejects.toThrow(); + expect(fs.readFileSync(lockPath)).toEqual(firstLock); + expect(fs.existsSync(first.schemas[0].cachePath)).toBe(true); + }); + + it('fails locked mode when source metadata drifts from the lock', async () => { + await syncRemoteSchemas(projectRoot, { globalDataDir }); + fs.appendFileSync( + path.join(projectRoot, 'openspec', 'config.yaml'), + '\n# intentional edit\n' + ); + const configPath = path.join(projectRoot, 'openspec', 'config.yaml'); + fs.writeFileSync( + configPath, + fs.readFileSync(configPath, 'utf8').replace('ref: main', 'ref: other') + ); + await expect( + syncRemoteSchemas(projectRoot, { locked: true, globalDataDir }) + ).rejects.toThrow(/does not match the configured source/); + }); + + it('rebuilds a malformed lockfile in all-source update mode', async () => { + const lockPath = path.join(projectRoot, 'openspec', 'schemas.lock.yaml'); + fs.writeFileSync(lockPath, 'version: 99\nschemas: {}\n'); + + const result = await syncRemoteSchemas(projectRoot, { globalDataDir }); + + expect(result.schemas.map((schema) => schema.name)).toEqual(['team-flow']); + expect(readSchemaLock(projectRoot)?.schemas['team-flow'].resolvedCommit).toMatch( + /^[0-9a-f]{40}$/ + ); + }); + + it.each(['../schemas/team-flow', '/schemas/team-flow', 'C:/schemas/team-flow'])( + 'rejects unsafe configured bundle path %s', + async (unsafePath) => { + const configPath = path.join(projectRoot, 'openspec', 'config.yaml'); + fs.writeFileSync( + configPath, + fs.readFileSync(configPath, 'utf8').replace( + 'path: schemas/team-flow', + `path: ${unsafePath}` + ) + ); + await expect(syncRemoteSchemas(projectRoot, { globalDataDir })).rejects.toThrow( + /Invalid schema bundle path/ + ); + expect(readSchemaLock(projectRoot)).toBeNull(); + } + ); +}); diff --git a/test/utils/change-utils.test.ts b/test/utils/change-utils.test.ts index 4f32914aa6..35d690f0ea 100644 --- a/test/utils/change-utils.test.ts +++ b/test/utils/change-utils.test.ts @@ -263,5 +263,31 @@ describe('createChange', () => { const stats = await fs.stat(changeDir); expect(stats.isDirectory()).toBe(true); }); + + it('keeps generated config at the schema root when planning elsewhere', async () => { + const planningRoot = path.join(testDir, 'planning-store'); + const schemaRoot = path.join(testDir, 'consumer'); + await fs.mkdir(planningRoot); + await fs.mkdir(schemaRoot); + + await createChange(planningRoot, 'store-backed-change', { schemaRoot }); + + await expect( + fs.readFile(path.join(schemaRoot, 'openspec', 'config.yaml'), 'utf8') + ).resolves.toBe('schema: spec-driven\n'); + await expect( + fs.stat( + path.join( + planningRoot, + 'openspec', + 'changes', + 'store-backed-change' + ) + ) + ).resolves.toMatchObject({}); + await expect( + fs.access(path.join(planningRoot, 'openspec', 'config.yaml')) + ).rejects.toThrow(); + }); }); });