diff --git a/.changeset/harness-skills.md b/.changeset/harness-skills.md new file mode 100644 index 0000000000..de7f2585e5 --- /dev/null +++ b/.changeset/harness-skills.md @@ -0,0 +1,5 @@ +--- +'@tanstack/ai-skills': minor +--- + +`@tanstack/ai-skills/harness`: the new `skills({ dirs })` plugin gives the harness model the skills of a list of folders, and makes each skill a `/` command that starts a turn with that skill. A skill with the name of another command is `/skill:`, and `/skills` lists them. The plugin watches the folders, so a skill that you add or remove while the session runs changes the commands at once. diff --git a/docs/config.json b/docs/config.json index 0d0de346cd..668cfc7a94 100644 --- a/docs/config.json +++ b/docs/config.json @@ -992,6 +992,11 @@ "label": "Run agents from a harness", "to": "harness/subagents", "addedAt": "2026-10-13" + }, + { + "label": "Add skills", + "to": "harness/skills", + "addedAt": "2026-10-13" } ] }, diff --git a/docs/harness/plugins.md b/docs/harness/plugins.md index 05a2143851..c71fe88184 100644 --- a/docs/harness/plugins.md +++ b/docs/harness/plugins.md @@ -131,6 +131,8 @@ export const greetings = definePlugin({ }) ``` +The [skills plugin](./skills) works this way: one command for each skill in a folder. + ## Change agents while the session runs Agents can come from data that changes too. `ctx.agents.set` and `ctx.agents.delete` work like `ctx.commands`. Here `/helper` adds an agent that the session can run at once: diff --git a/docs/harness/skills.md b/docs/harness/skills.md new file mode 100644 index 0000000000..89a399d56e --- /dev/null +++ b/docs/harness/skills.md @@ -0,0 +1,103 @@ +--- +title: Add skills to a harness +id: harness-skills +order: 10 +description: "Give the harness model the skills in a list of folders, and make each skill a slash command. A skill that you add while the session runs works at once." +keywords: + - tanstack ai + - harness + - skills + - SKILL.md + - slash commands +--- + +Your users keep skills as `SKILL.md` folders, in the repo and in their home folder, and they want to call them from your agent like any other command. The skills plugin reads those folders, gives the model the list of skills, and makes each skill a `/` command. A skill that a user adds while the session runs works at once. + +The plugin reads the folders with `node:fs`, so it runs on Node.js. It does not work on Cloudflare Workers. + +## 1. Add the plugin + + + +react: @tanstack/ai-skills +vue: @tanstack/ai-skills +solid: @tanstack/ai-skills +svelte: @tanstack/ai-skills +preact: @tanstack/ai-skills +angular: @tanstack/ai-skills +octane: @tanstack/ai-skills +vanilla: @tanstack/ai-skills + + + +```ts group=harness-skills +import { homedir } from 'node:os' +import { join } from 'node:path' +import { defineHarness } from '@tanstack/ai-harness' +import { openaiText } from '@tanstack/ai-openai' +import { skills } from '@tanstack/ai-skills/harness' + +export const assistant = defineHarness({ + name: 'acme/assistant', + adapter: openaiText('gpt-6.1-sol'), + plugins: () => [ + skills({ + dirs: ['./.agents/skills', join(homedir(), '.acme', 'skills')], + }), + ], +}) +``` + +- `dirs` lists the skill folders. Each one holds skill folders with a `SKILL.md`. +- When two folders have a skill with the same name, the first folder wins. Put the repo folder first, so a project can replace a user's skill. +- A folder that does not exist yet is fine. Its skills show once the user makes it. + +## 2. Call a skill + +`/release-notes write them for v2` starts a turn with "Use the release-notes skill: write them for v2". The model loads the skill with `load_skill`, then does the task. `/release-notes` alone asks for the skill with no task. + +The model also sees the list of skills, so it can load one on its own when a task matches. + +## Names that are taken + +A skill with the name of another command gets a prefix: a `model` skill is `/skill:model`, and `/model` stays the model picker. Pass `reserved` with the names that your UI handles itself: + +```ts group=harness-skills +export const cliSkills = skills({ + dirs: ['./.agents/skills'], + reserved: ['help', 'exit'], +}) +``` + +`/skills` lists every skill, its folder, and its command. + +## Skills that change while the session runs + +The plugin watches the folders, and it lists them again at the start of each turn: + +- A new `SKILL.md` adds its command. +- A removed skill takes its command away. +- An edited description updates the command. +- A broken `SKILL.md` is left out until it is fixed. + +Each change sends a `harness.commands.changed` event, so a UI can show the new command list at once. + +## Many skills + +The model's list has one line for each skill: the name and the first sentence of the description. When the list is longer than `maxCatalogTokens` (default 4000), the skills of the earlier folders stay, and the rest leave the model's list. A skill that the user calls with its command always stays in it. + +The other options of `withSkills` pass through, for example `maxCatalogTokens` and `requireApproval`: + +```ts group=harness-skills +export const manySkills = skills({ + dirs: ['./.agents/skills'], + maxCatalogTokens: 8000, +}) +``` + +## What you have now + +- The skills of your folders, in the model's list and as `/` commands. +- New and changed skills that work in a running session, with no restart. + +Next: [Agent Skills](../skills/agent-skills) has the `SKILL.md` format, and [Write a harness plugin](./plugins#change-commands-while-the-session-runs) shows the `ctx.commands` API that the plugin uses. diff --git a/packages/ai-skills/package.json b/packages/ai-skills/package.json index a437cd0e0c..14aeabc3b0 100644 --- a/packages/ai-skills/package.json +++ b/packages/ai-skills/package.json @@ -25,6 +25,10 @@ "types": "./dist/esm/index.d.ts", "import": "./dist/esm/index.js" }, + "./harness": { + "types": "./dist/esm/harness.d.ts", + "import": "./dist/esm/harness.js" + }, "./node": { "types": "./dist/esm/node/index.d.ts", "import": "./dist/esm/node/index.js" @@ -75,15 +79,20 @@ }, "peerDependencies": { "@tanstack/ai": "workspace:^", + "@tanstack/ai-harness": "workspace:^", "vitest": "^4.1.10", "zod": "^3.0.0 || ^4.0.0" }, "peerDependenciesMeta": { + "@tanstack/ai-harness": { + "optional": true + }, "vitest": { "optional": true } }, "devDependencies": { + "@tanstack/ai-harness": "workspace:*", "@vitest/coverage-v8": "4.1.10", "vitest": "^4.1.10", "zod": "^4.2.0" diff --git a/packages/ai-skills/src/harness.ts b/packages/ai-skills/src/harness.ts new file mode 100644 index 0000000000..98241905da --- /dev/null +++ b/packages/ai-skills/src/harness.ts @@ -0,0 +1,323 @@ +import { watch } from 'node:fs' +import { dirname } from 'node:path' +import { firstSentence } from '@tanstack/ai' +import { defineCommand, definePlugin } from '@tanstack/ai-harness' +import { renderCatalog } from './catalog' +import { estimateTokens, withSkills } from './middleware' +import { skillDirectory } from './node' +import type { FSWatcher } from 'node:fs' +import type { AnyChatMiddleware } from '@tanstack/ai' +import type { SkillsOptions } from './middleware' +import type { ModelFamily, SkillMetadata, SkillSource } from './types' + +export interface SkillsPluginOptions extends SkillsOptions { + /** + * The skill folders. Each folder holds skill folders with a `SKILL.md`. + * When two folders have a skill with the same name, the first folder wins. + * A folder that does not exist yet is used once it exists. + */ + dirs: ReadonlyArray + /** + * Names that a UI handles itself, for example its own `/help`. A skill + * with one of these names gets the command `/skill:`. + */ + reserved?: ReadonlyArray +} + +/** A skill, and the folder it comes from. */ +interface FoundSkill { + skill: SkillMetadata + dir: string +} + +/** + * The skills of `dirs`, by name, in folder order. The first folder wins a + * name. A folder that is missing or cannot be read gives no skills, and a + * broken `SKILL.md` is left out. + */ +async function scan(dirs: ReadonlyArray) { + const found = new Map() + for (const dir of dirs) { + // A folder that throws, at once or later, gives no skills. + const listed = await Promise.resolve() + .then(() => skillDirectory(dir, { strict: false }).list()) + .catch(() => []) + for (const skill of listed) { + if (!found.has(skill.name)) found.set(skill.name, { skill, dir }) + } + } + return found +} + +/** The catalog with only the first sentence of each description. */ +function shortCatalog(listed: Array, family: ModelFamily) { + return renderCatalog( + listed.map((skill) => ({ + ...skill, + description: firstSentence(skill.description), + })), + family, + ) +} + +/** The text of the turn that `/ [task]` starts. */ +function skillPrompt(name: string, input: unknown) { + const task = typeof input === 'string' ? input.trim() : '' + return task === '' + ? `Use the ${name} skill.` + : `Use the ${name} skill: ${task}` +} + +/** + * A harness plugin that gives the model the skills of `dirs`, and makes each + * skill a command: `/ [task]` starts a turn that tells the model to use + * the skill. A skill whose name another command has is `/skill:`. + * `/skills` lists every skill, its folder, and its command. + * + * The plugin watches the folders, and it lists them again at the start of + * each turn. A skill that you add, change, or remove while the session runs + * changes the commands at once, with no restart. + * + * The model's skill list has one line for each skill: its name and the first + * sentence of its description. When the list is longer than + * `maxCatalogTokens`, the skills of the earlier folders stay in it, and a + * skill that you called with its command always stays. + * + * @param options.dirs - The skill folders, the first one wins on a name. + * @param options.reserved - Command names a UI handles itself. + * + * @example + * ```ts + * import { homedir } from 'node:os' + * import { join } from 'node:path' + * import { skills } from '@tanstack/ai-skills/harness' + * + * defineHarness({ + * name: 'acme/agent', + * adapter, + * plugins: () => [ + * skills({ dirs: ['./.agents/skills', join(homedir(), '.acme', 'skills')] }), + * ], + * }) + * ``` + */ +export function skills(options: SkillsPluginOptions) { + const { dirs, reserved = [], ...skillsOptions } = options + // `skills` is this plugin's own command, so a skill named `skills` gets + // the prefix too. + const taken = new Set(['skills', ...reserved]) + + return definePlugin({ + name: 'tanstack/skills', + setup: async (ctx) => { + // The skill commands of this plugin: command name to skill name and + // the description the command has. + const mine = new Map() + // Skills called with their command stay in the model's list. + const called = new Set() + let found = new Map() + const watchers = await ctx.resources.acquire( + () => new Map(), + (open) => { + for (const watcher of open.values()) watcher.close() + open.clear() + }, + ) + const closed = () => ctx.resources.signal.aborted + + const skillCommand = (name: string, description: string) => + defineCommand({ + description: `Skill: ${firstSentence(description)}`, + input: { + type: 'string', + description: 'What the skill should do. Optional.', + }, + // The command's own session, so the turn runs as the user who + // ran the command. + run: (input: unknown, command) => { + called.add(name) + void command.session.prompt(skillPrompt(name, input)) + }, + }) + + /** Watch `path` under `key`. `false` when `path` does not exist. */ + const tryWatch = (key: string, path: string, recursive: boolean) => { + try { + const watcher = watch(path, { recursive, persistent: false }, soon) + watcher.on('error', () => { + watcher.close() + watchers.delete(key) + }) + watchers.set(key, watcher) + return true + } catch { + return false + } + } + + /** + * Watch each skill folder. While a folder does not exist, watch its + * parent folder, so the skills show as soon as you make the folder. + */ + const watchFolders = () => { + for (const dir of dirs) { + if (watchers.has(dir)) continue + const parentKey = `${dir} (parent)` + if (tryWatch(dir, dir, true)) { + watchers.get(parentKey)?.close() + watchers.delete(parentKey) + } else if (!watchers.has(parentKey)) { + tryWatch(parentKey, dirname(dir), false) + } + } + } + + /** Make the commands match the skills in the folders now. */ + const update = async () => { + if (closed()) return + found = await scan(dirs) + if (closed()) return + watchFolders() + const next = new Map() + for (const { skill } of found.values()) { + const isFree = + !taken.has(skill.name) && + (mine.has(skill.name) || !ctx.commands.has(skill.name)) + next.set(isFree ? skill.name : `skill:${skill.name}`, { + name: skill.name, + description: skill.description, + }) + } + for (const [command, entry] of mine) { + if (next.get(command)?.name === entry.name) continue + ctx.commands.delete(command) + mine.delete(command) + } + for (const [command, entry] of next) { + const isSame = mine.get(command)?.description === entry.description + if (isSame) continue + try { + ctx.commands.set( + command, + skillCommand(entry.name, entry.description), + ) + mine.set(command, entry) + } catch { + // Another plugin took `skill:` too. The skill stays in the + // model's list, with no command. + } + } + } + // Updates run one after the other. + let updating = Promise.resolve() + const sync = () => { + updating = updating.then(update).catch(() => {}) + return updating + } + + // A save often fires several watch events: wait until they stop. + let timer: ReturnType | undefined + function soon() { + clearTimeout(timer) + timer = setTimeout(() => void sync(), 200) + timer.unref?.() + } + ctx.resources.signal.addEventListener('abort', () => clearTimeout(timer)) + + // Commands of the plugins after this one exist only once all are set + // up, so the first sync waits for that. + void ctx.commands.ready.then(sync) + + const source: SkillSource = { + list: async () => + [...(await scan(dirs)).values()].map((entry) => entry.skill), + load: async (name) => (await folderOf(name)).load(name), + listResources: async (name) => + (await (await folderOf(name)).listResources?.(name)) ?? [], + listScripts: async (name) => + (await (await folderOf(name)).listScripts?.(name)) ?? [], + readResource: async (name, path) => { + const folder = await folderOf(name) + if (!folder.readResource) { + throw new Error(`Skill "${name}" has no resource files.`) + } + return folder.readResource(name, path) + }, + } + /** The source of the folder that has the skill `name` now. */ + async function folderOf(name: string) { + const entry = (await scan(dirs)).get(name) + if (!entry) + throw new Error(`No skill named "${name}" in ${dirs.join(', ')}.`) + return skillDirectory(entry.dir, { strict: false }) + } + + const hasOwnCatalog = + skillsOptions.renderCatalog !== undefined || + skillsOptions.instructionTemplate !== undefined + // The catalog that `withSkills` renders, so the count matches it. + const render = + skillsOptions.renderCatalog ?? + (hasOwnCatalog ? renderCatalog : shortCatalog) + + /** + * Over the token cap: the called skills first, then the folder order, + * until the list is full. The XML list of Anthropic models is the + * longest form, so the count uses it. + */ + const keepEarlier = (listed: Array, limit: number) => { + const ordered = [ + ...listed.filter((skill) => called.has(skill.name)), + ...listed.filter((skill) => !called.has(skill.name)), + ] + const kept: Array = [] + for (const skill of ordered) { + const tooLong = + estimateTokens(render([...kept, skill], 'anthropic')) > limit + if (tooLong) break + kept.push(skill) + } + return kept + } + + // The sync runs first, so the commands match the folders at the start + // of each turn. + const middleware: Array = [ + { name: 'tanstack/skills:sync', setup: () => sync() }, + withSkills(source, { + ...skillsOptions, + ...(hasOwnCatalog ? {} : { renderCatalog: shortCatalog }), + onLimitExceeded: skillsOptions.onLimitExceeded ?? keepEarlier, + }), + ] + + return { + middleware, + commands: { + skills: defineCommand({ + description: 'List the skills, their folders, and their commands', + run: async () => { + await sync() + if (found.size === 0) { + return `No skills yet. Add a folder with a SKILL.md to one of: ${dirs.join(', ')}` + } + const commandOf = new Map( + [...mine].map(([command, entry]) => [entry.name, command]), + ) + const lines = [...found.values()].map(({ skill, dir }) => { + const command = commandOf.get(skill.name) + const name = command ? `/${command}` : skill.name + const note = + command && command !== skill.name + ? ` (/${skill.name} is taken)` + : '' + return `${name} ${dir}${note}` + }) + return lines.join('\n') + }, + }), + }, + } + }, + }) +} diff --git a/packages/ai-skills/src/middleware.ts b/packages/ai-skills/src/middleware.ts index 7a290c51e6..67e682d1ae 100644 --- a/packages/ai-skills/src/middleware.ts +++ b/packages/ai-skills/src/middleware.ts @@ -59,7 +59,7 @@ interface SkillsRuntime { const SkillsCapability = createCapability()('skills') /** ~4 chars/token — good enough to guard a runaway catalog. */ -const estimateTokens = (s: string) => Math.ceil(s.length / 4) +export const estimateTokens = (s: string) => Math.ceil(s.length / 4) function fillTemplate(template: string, catalog: string): string { // Escape `{{`/`}}` to sentinels, substitute `{skills}`, then restore braces. diff --git a/packages/ai-skills/tests/harness.test.ts b/packages/ai-skills/tests/harness.test.ts new file mode 100644 index 0000000000..7620f1b9f6 --- /dev/null +++ b/packages/ai-skills/tests/harness.test.ts @@ -0,0 +1,436 @@ +import { watch } from 'node:fs' +import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { fakeText } from '@tanstack/ai/testing' +import { + createHarnessHost, + defineCommand, + defineHarness, + definePlugin, +} from '@tanstack/ai-harness' +import { skills } from '../src/harness' +import type { TextOptions } from '@tanstack/ai' +import type { FakeResponse } from '@tanstack/ai/testing' +import type { HarnessHost, HarnessPlugin } from '@tanstack/ai-harness' +import type { SkillsPluginOptions } from '../src/harness' +import type { SkillSource } from '../src/types' + +/** The sources that the plugin gives to `withSkills`, newest last. */ +const seen = vi.hoisted(() => ({ sources: new Array() })) +/** Folders whose source fails: `list` rejects, or `open` throws when it is made. */ +const failing = vi.hoisted(() => ({ + list: new Set(), + open: new Set(), +})) + +// Pass-through spies. The tests reach the plugin's watchers, its skill +// source, and a failing folder through them. Everything else is unchanged. +vi.mock('node:fs', async (importOriginal) => { + const original = await importOriginal() + return { ...original, watch: vi.fn(original.watch) } +}) +vi.mock('../src/middleware', async (importOriginal) => { + const original = await importOriginal() + return { + ...original, + withSkills: (...args: Parameters) => { + const [sources] = args + if (!Array.isArray(sources)) seen.sources.push(sources) + return original.withSkills(...args) + }, + } +}) +vi.mock('../src/node', async (importOriginal) => { + const original = await importOriginal() + return { + ...original, + skillDirectory: (...args: Parameters) => { + const key = String(args[0]) + if (failing.open.has(key)) throw new Error(`cannot open ${key}`) + const source = original.skillDirectory(...args) + if (!failing.list.has(key)) return source + return { + ...source, + list: () => Promise.reject(new Error(`cannot read ${key}`)), + } + }, + } +}) + +const fixtures = fileURLToPath(new URL('./fixtures/skills', import.meta.url)) + +const temps: Array = [] +const hosts: Array = [] +afterEach(async () => { + await Promise.all(hosts.splice(0).map((host) => host.close())) + await Promise.all( + temps.splice(0).map((dir) => rm(dir, { recursive: true, force: true })), + ) + seen.sources.length = 0 + failing.list.clear() + failing.open.clear() + vi.mocked(watch).mockClear() +}) + +async function tempDir() { + const dir = await mkdtemp(join(tmpdir(), 'skills-harness-')) + temps.push(dir) + return dir +} + +/** Write the skill `name` into `root`, with a two-sentence description. */ +async function addSkill(root: string, name: string, about = name) { + await mkdir(join(root, name), { recursive: true }) + await writeFile( + join(root, name, 'SKILL.md'), + `---\nname: ${name}\ndescription: Helps with ${about}. Second sentence here.\n---\n\nDo ${about}.\n`, + ) +} + +/** Write `references/guide.md` with `text` into the skill `name` of `root`. */ +async function addGuide(root: string, name: string, text: string) { + await mkdir(join(root, name, 'references'), { recursive: true }) + await writeFile(join(root, name, 'references', 'guide.md'), text) +} + +/** The watchers made for `path`, oldest first. */ +function watchersOf(path: string) { + const spy = vi.mocked(watch) + return spy.mock.results.filter( + (_, index) => spy.mock.calls[index]?.[0] === path, + ) +} + +/** An answer of the fake model, or a function that makes it. */ +type Answer = FakeResponse | (() => Promise) + +/** + * A session with the skills plugin and `others` after it. The model gives + * `answers` first and then answers "ok". It keeps each request. + */ +async function open( + options: SkillsPluginOptions, + { + others = [], + answers = [], + }: { others?: Array; answers?: Array } = {}, +) { + const requests: Array = [] + const fake = fakeText() + const ok: Answer = { text: 'ok' } + const steps = [...answers, ...Array.from({ length: 10 }, () => ok)] + fake.setResponses( + steps.map((answer) => async ({ request }: { request: TextOptions }) => { + requests.push(request) + return typeof answer === 'function' ? answer() : answer + }), + ) + const host = createHarnessHost() + hosts.push(host) + const session = await host.open( + defineHarness({ + name: 'test/skills', + adapter: fake, + plugins: () => [skills(options), ...others], + }), + { threadId: 't' }, + ) + const names = () => session.commands().map((command) => command.name) + // `/skills` syncs the commands with the folders first, so a test needs + // no watch event. + const list = async () => String(await session.command('skills')) + return { session, names, list, requests } +} + +/** The system prompt text of a model request. */ +const promptOf = (request: TextOptions | undefined) => + JSON.stringify(request?.systemPrompts ?? []) + +/** The tool result text that a model request gives back to the model. */ +const toolResultOf = (request: TextOptions | undefined) => + String(request?.messages.find((message) => message.role === 'tool')?.content) + +/** The model's call of the `load_skill` tool for `name`. */ +const loadSkill = (name: string): FakeResponse => ({ + toolCalls: [{ name: 'load_skill', input: { name } }], +}) + +describe('skills()', () => { + it('makes a command for each skill, from the first folder that has it', async () => { + const first = await tempDir() + const second = await tempDir() + await addSkill(first, 'alpha') + await addSkill(first, 'shared', 'the first folder') + await addSkill(second, 'beta') + await addSkill(second, 'shared', 'the second folder') + const { session, names, list } = await open({ + dirs: [first, join(first, 'missing'), second], + }) + + const listed = await list() + expect(names()).toEqual( + expect.arrayContaining(['skills', 'alpha', 'beta', 'shared']), + ) + expect( + session.commands().find((command) => command.name === 'shared') + ?.description, + ).toBe('Skill: Helps with the first folder.') + expect(listed).toContain(`/shared ${first}`) + expect(listed).toContain(`/beta ${second}`) + }) + + it('prefixes a skill whose name another command has', async () => { + const dir = await tempDir() + await addSkill(dir, 'model') + await addSkill(dir, 'help') + await addSkill(dir, 'alpha') + const picker = definePlugin({ + name: 'test/picker', + setup: () => ({ + commands: { + model: defineCommand({ description: 'Pick', run: () => 'picked' }), + }, + }), + }) + const { session, names, list } = await open( + { dirs: [dir], reserved: ['help'] }, + { others: [picker] }, + ) + + const listed = await list() + expect(names()).toEqual( + expect.arrayContaining(['skill:model', 'skill:help', 'alpha']), + ) + expect(names()).not.toContain('help') + expect(await session.command('model')).toBe('picked') + expect(listed).toContain(`/skill:model ${dir} (/model is taken)`) + }) + + it('starts a turn that asks for the skill, with or without a task', async () => { + const dir = await tempDir() + await addSkill(dir, 'alpha') + const { session, list, requests } = await open({ dirs: [dir] }) + await list() + + await session.command('alpha', 'make a banner') + await vi.waitFor(() => expect(requests).toHaveLength(1)) + expect(JSON.stringify(requests[0]?.messages)).toContain( + 'Use the alpha skill: make a banner', + ) + expect(promptOf(requests[0])).toContain('Helps with alpha.') + expect(promptOf(requests[0])).not.toContain('Second sentence here.') + + await session.command('alpha') + await vi.waitFor(() => expect(requests).toHaveLength(2)) + expect(JSON.stringify(requests[1]?.messages)).toContain( + 'Use the alpha skill.', + ) + }) + + it('follows skills that come and go while the session runs', async () => { + const dir = await tempDir() + await addSkill(dir, 'alpha') + const { names } = await open({ dirs: [dir] }) + await vi.waitFor(() => expect(names()).toContain('alpha')) + + await addSkill(dir, 'gamma') + await vi.waitFor(() => expect(names()).toContain('gamma'), { + timeout: 5000, + }) + await rm(join(dir, 'alpha'), { recursive: true, force: true }) + await vi.waitFor(() => expect(names()).not.toContain('alpha'), { + timeout: 5000, + }) + }) + + it('shows the skills of a folder made after the start', async () => { + const root = await tempDir() + const { names } = await open({ dirs: [join(root, 'later')] }) + + await addSkill(join(root, 'later'), 'alpha') + await vi.waitFor(() => expect(names()).toContain('alpha'), { + timeout: 5000, + }) + }) + + it('uses a folder made after the start, from the next turn', async () => { + const root = await tempDir() + const later = join(root, 'later') + const { session, names } = await open({ dirs: [later] }) + expect(names()).not.toContain('alpha') + + await addSkill(later, 'alpha') + await session.prompt('hi') + expect(names()).toContain('alpha') + }) + + it('leaves out a broken SKILL.md until it is fixed', async () => { + const dir = await tempDir() + await mkdir(join(dir, 'alpha')) + await writeFile(join(dir, 'alpha', 'SKILL.md'), '---\nname: alpha\n') + const { names, list } = await open({ dirs: [dir] }) + + expect(await list()).toContain('No skills yet.') + expect(names()).not.toContain('alpha') + await addSkill(dir, 'alpha') + await list() + expect(names()).toContain('alpha') + }) + + it('keeps the earlier folders, and the called skills, when the list is too long', async () => { + const first = await tempDir() + const second = await tempDir() + // With this description, one skill fits in 36 tokens and two do not. + const about = 'a topic with a long and wordy name for the test' + await addSkill(first, 'alpha', about) + await addSkill(second, 'zeta', about) + const { session, list, requests } = await open({ + dirs: [first, second], + maxCatalogTokens: 36, + }) + await list() + + await session.prompt('hi') + expect(promptOf(requests[0])).toContain('alpha') + expect(promptOf(requests[0])).not.toContain('zeta') + + await session.command('zeta') + await vi.waitFor(() => expect(requests).toHaveLength(2)) + expect(promptOf(requests[1])).toContain('zeta') + }) + + it('loads a skill, with its files, from the folder that has it', async () => { + const first = await tempDir() + await addSkill(first, 'beta') + const { session, requests } = await open( + { dirs: [first, fixtures] }, + { answers: [loadSkill('alpha')] }, + ) + + await session.prompt('use alpha') + expect(JSON.parse(toolResultOf(requests[1]))).toEqual({ + skill: 'alpha', + content: '# Alpha\n\nDo the alpha thing.', + resources: ['references/note.md'], + scripts: [ + { path: 'scripts/run.py', executable: false, reason: 'no-runtime' }, + ], + compatibility: 'any runtime', + }) + }) + + it('tells the model when a skill is gone before it loads', async () => { + const dir = await tempDir() + await addSkill(dir, 'alpha') + const removeThenLoad = async () => { + await rm(join(dir, 'alpha'), { recursive: true, force: true }) + return loadSkill('alpha') + } + const { session, requests } = await open( + { dirs: [dir] }, + { answers: [removeThenLoad] }, + ) + + await session.prompt('use alpha') + expect(JSON.parse(toolResultOf(requests[1]))).toEqual({ + error: `No skill named "alpha" in ${dir}.`, + }) + }) + + it('stops following the folders when the session closes', async () => { + const dir = await tempDir() + await addSkill(dir, 'alpha') + const opened = await open({ dirs: [dir] }) + await opened.list() + await Promise.all(hosts.splice(0).map((host) => host.close())) + + await addSkill(dir, 'gamma') + await new Promise((resolve) => setTimeout(resolve, 500)) + expect(opened.names()).not.toContain('gamma') + }) + + it('loads a skill from its folder when the model calls load_skill', async () => { + const dir = await tempDir() + await addSkill(dir, 'alpha') + await addGuide(dir, 'alpha', 'guide') + await mkdir(join(dir, 'alpha', 'scripts')) + await writeFile(join(dir, 'alpha', 'scripts', 'run.sh'), 'echo hi') + const { session, requests } = await open( + { dirs: [dir] }, + { + answers: [ + { toolCalls: [{ name: 'load_skill', input: { name: 'alpha' } }] }, + ], + }, + ) + + await session.prompt('use alpha') + await vi.waitFor(() => expect(requests).toHaveLength(2)) + const sent = JSON.stringify(requests[1]?.messages) + expect(sent).toContain('Do alpha.') + expect(sent).toContain('references/guide.md') + expect(sent).toContain('scripts/run.sh') + }) + + it('reads a resource from the folder that has the skill now', async () => { + const first = await tempDir() + const second = await tempDir() + await addSkill(second, 'alpha') + await addGuide(second, 'alpha', 'second guide') + await open({ dirs: [first, second] }) + const source = seen.sources.at(-1) + const read = (name: string) => + source?.readResource?.(name, 'references/guide.md') + + expect(await read('alpha')).toBe('second guide') + // The first folder wins a name, also for a skill made after the start. + await addSkill(first, 'alpha') + await addGuide(first, 'alpha', 'first guide') + expect(await read('alpha')).toBe('first guide') + await expect(read('missing')).rejects.toThrow('No skill named "missing"') + }) + + it('watches a folder again after a watch error', async () => { + const dir = await tempDir() + await addSkill(dir, 'alpha') + const { names, list } = await open({ dirs: [dir] }) + await list() + expect(watchersOf(dir)).toHaveLength(1) + + watchersOf(dir)[0]?.value.emit('error', new Error('watch failed')) + await list() + expect(watchersOf(dir)).toHaveLength(2) + await addSkill(dir, 'gamma') + await vi.waitFor(() => expect(names()).toContain('gamma'), { + timeout: 5000, + }) + }) + + it('skips a folder whose skills cannot be read', async () => { + const broken = await tempDir() + const dir = await tempDir() + await addSkill(broken, 'beta') + await addSkill(dir, 'alpha') + failing.list.add(broken) + const { names, list } = await open({ dirs: [broken, dir] }) + + expect(await list()).toContain(`/alpha ${dir}`) + expect(names()).toContain('alpha') + expect(names()).not.toContain('beta') + }) + + it('syncs again after a sync that failed', async () => { + const dir = await tempDir() + await addSkill(dir, 'alpha') + failing.open.add(dir) + const { names, list } = await open({ dirs: [dir] }) + + expect(await list()).toContain('No skills yet.') + failing.open.delete(dir) + await list() + expect(names()).toContain('alpha') + }) +}) diff --git a/packages/ai-skills/vite.config.ts b/packages/ai-skills/vite.config.ts index c02dcd7697..836a48d7b4 100644 --- a/packages/ai-skills/vite.config.ts +++ b/packages/ai-skills/vite.config.ts @@ -31,6 +31,7 @@ export default mergeConfig( tanstackViteConfig({ entry: [ './src/index.ts', + './src/harness.ts', './src/node/index.ts', './src/static/index.ts', './src/testing/index.ts', diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 7df08d78ac..ee9e2e10be 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -3298,6 +3298,9 @@ importers: specifier: workspace:^ version: link:../ai devDependencies: + '@tanstack/ai-harness': + specifier: workspace:* + version: link:../ai-harness '@vitest/coverage-v8': specifier: 4.1.10 version: 4.1.10(vitest@4.1.10)