From 66def51a9206b23e3e4f6833b931b1e2419d06b6 Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Thu, 22 Oct 2026 09:23:28 -0400 Subject: [PATCH] feat(cli): run session component tests through REPL.Test --- architecture.md | 28 + packages/cli/src/cli.ts | 30 + .../cli/src/documents/sidekick/entries.md | 100 ++- .../cli/src/documents/sidekick/environment.md | 6 + packages/cli/src/repl-profile.ts | 11 +- packages/cli/src/repl/program.ts | 8 + packages/cli/src/repl/sidekick/admission.ts | 40 + packages/cli/src/repl/sidekick/controls.ts | 28 + packages/cli/src/repl/sidekick/owner.ts | 19 +- packages/cli/src/repl/sidekick/profile.ts | 2 +- packages/cli/src/repl/sidekick/requests.ts | 8 +- packages/cli/src/repl/sidekick/testing.ts | 259 ++++++ .../repl/issue-883/sidekick-harness.ts | 19 +- .../cli/tests/repl-sidekick-controls.test.ts | 4 +- .../cli/tests/repl-sidekick-testing.test.ts | 794 ++++++++++++++++++ specs/repl-spec.md | 84 ++ specs/testing-spec.md | 8 + 17 files changed, 1438 insertions(+), 10 deletions(-) create mode 100644 packages/cli/src/repl/sidekick/testing.ts create mode 100644 packages/cli/tests/repl-sidekick-testing.test.ts diff --git a/architecture.md b/architecture.md index 0ac5b497b..bbc6e49b3 100644 --- a/architecture.md +++ b/architecture.md @@ -6428,6 +6428,34 @@ no project fallback. The Sidekick's native provider keeps the captured project `agentCwd`, or the caller's explicit dependency, and its existing identity and configuration. +### Response-owned component testing + +`repl/sidekick/testing.ts` executes REPL.Test documents in fresh children of the +command host captured during profile assembly. Each child installs one real +useTesting session and executes the finalized REPL profile, including immutable +host declarations and the selected TypeScript loader. The CLI supplies the +existing nested testing-host installation factory with its captured host +settings. Response and entry bindings or live registrations never enter those +roots. Child outcomes travel back as Results so a clean host child cannot +reassign response failure ownership to its parent. The response owns halting +and joining every child; document cleanup finishes before the next starts. + +The root's existing admission state holds a process-local test slot beside its +entry reservation. Both submitters consult it; testing checks that reservation +and session readiness, then takes the slot synchronously before discovery. The +slot remains held through joined cleanup. Stop cancels response-owned tests and +leaves admitted ordinary entries with their session owner; Exit joins both +owners before returning. + +Actual Testing results, execution completion and drained output are collected +separately. Ordered completion callbacks distinguish a valid zero-test policy +rejection from earlier document failure without inspecting error names. Fatal +persistence, Files, disclosure and owning-cleanup failures retain their existing +authority. Completed detached reports use the existing designated-control +observation records; unmatched intents remain unknown and inspection does no +work. The exact report and status fold live in the +[REPL testing contract](specs/repl-spec.md#testing-session-components). + ### Source ownership in retained entries A source position identifies authored bytes, not which invocation used them. diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index a8140b8d9..66aa0fa29 100755 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -2984,7 +2984,37 @@ function* dispatch( // line carried and nothing more: the profile settles the fallback, and // whether anybody answers to that name is the first explicit message's // question rather than this command's. + const moduleLoader = yield* ComponentModuleLoader.get(); + const testAgentWorker = yield* readWorkerCommand(); + const bundledPlugin = yield* BundledRunPlugin.expect(); const profile = yield* assembleReplProfile(replStack, plugins, { + *testingHarness(includes) { + return testHarnessInstallation( + testingExecutionHost({ + componentModuleLoader: moduleLoader, + includes: [...includes], + plugins: plugins.plugins, + pluginArgs: plugins.args, + secretDetection: true, + installService, + bundledPlugin, + testAgentWorker, + planDeclaration: (request) => + planComponentDeclaration({ + surface: "component", + includes, + plugins, + context: request.context, + ...(request.planWriterRoot === undefined + ? {} + : { planWriterRoot: request.planWriterRoot }), + host: request.host, + installElicitation: request.installElicitation, + }), + }), + [testAgentChildDeclaration()], + ); + }, includes: command.config.include, ...(command.config.sidekickAgent === undefined ? {} diff --git a/packages/cli/src/documents/sidekick/entries.md b/packages/cli/src/documents/sidekick/entries.md index a56e3b47f..a8d59d40b 100644 --- a/packages/cli/src/documents/sidekick/entries.md +++ b/packages/cli/src/documents/sidekick/entries.md @@ -1,7 +1,7 @@ # Entries Read this before preparing an entry for the first time, or whenever you need the -authoring pattern for putting source into the shared draft. +authoring pattern for preparing, testing and reusing session components. ## Applies to @@ -140,6 +140,104 @@ arguments for continuation, or `null` when there is no more. If a later read's cursor refuses because the answer changed, restart the read without the cursor. Nothing these reads return authorizes a repository change. +### Test, fix and reuse a session component + +When asked to create and test a component, write its definition and a companion +Markdown test through an ordinary entry. This Greeting writes `greeting.txt` +when invoked and returns its greeting. Authoring its file performs neither: + +````xmd + +```xmd +Hello from this session.Hello from this session."} /> +{definition} +"} /> +{test} +``` + +{code} + +```` + +Follow the receipt's read instruction and check the actual File outcomes. A +completed entry alone does not prove either write succeeded. `greeting.txt` +does not exist yet, and authoring the test has run no assertion. + +After successful writes, request the document explicitly: + +````xmd + +```` + +Feedback contains a passed document and its named assertion. Invocation created +`greeting.txt`. Testing executes authored effects under the command's ordinary +permissions; it does not append an entry or consume the draft. + +To see a real failure, replace the expectation through another ordinary entry, +then request the directory suite: + +````xmd + +```xmd +"} /> +{test} +``` + +{code} + + +```` + +The completed report is `failed`. Read the named assertion's full diagnostic, +actual and expected values before changing the file. Fix that expectation and +request a fresh suite: + +````xmd + +```xmd +"} /> +{test} +``` + +{code} + + +```` + +A passed report belongs to this request. Earlier failed reports stay as facts. +You can now invoke the component in a later ordinary entry without includes, +pasted definitions or a restart: + +````xmd + +```xmd + +``` + +{code} + +```` + +The entry returns `Hello from this session.`. A person can also enter +`` manually. Both use the selected component; configured directories +still precede session components during a test, including a directory suite. + +Use `as` when this response needs the structured report. Capturing and leaving +it uncaptured both deliver the same feedback and publish no chat. To show it to +the person explicitly, write: + +```xmd + + +``` + +`no-tests` means no document matched, or otherwise valid documents reached no +Test. It is not a pass. A parse or runtime error before the first Test is a +failed document; inspect its `error` and any partial `report`. An admission or +discovery refusal produces a diagnostic instead of a completed report. Wait for +entry or test cleanup to finish before requesting another run. Stop cancels and +joins response-owned tests; it leaves an already admitted ordinary entry alone. + ## Common mistake Writing the entry source directly into your response, or inside a fence marked diff --git a/packages/cli/src/documents/sidekick/environment.md b/packages/cli/src/documents/sidekick/environment.md index 244263687..0738b5509 100644 --- a/packages/cli/src/documents/sidekick/environment.md +++ b/packages/cli/src/documents/sidekick/environment.md @@ -23,6 +23,12 @@ answer. `` before invoking a new component: a configured definition can win. A selected malformed definition fails; fix it rather than assuming another candidate runs. +- Request real tests with `` + or ``. Targets are relative to the session + workspace even inside a response Dir; absolute and escaping paths refuse. + A directory selects `**/*.test.md` recursively and retains configured-first + component lookup. Tests have effects and use the command's ordinary permissions. + Read `` for the create, test, fix and reuse sequence. - A nested `` selects another known, authorized directory for its content; later siblings return to the workspace. To copy project input, enter its explicitly known project path, capture the read there, then write the copy diff --git a/packages/cli/src/repl-profile.ts b/packages/cli/src/repl-profile.ts index fb21a7f45..d551a7f44 100644 --- a/packages/cli/src/repl-profile.ts +++ b/packages/cli/src/repl-profile.ts @@ -5,6 +5,9 @@ * id before the includes-dependent Plan declaration and entry vocabulary exist. */ +import { replTesting } from "./repl/sidekick/testing.ts"; +import type { ReplTesting } from "./repl/sidekick/testing.ts"; + import type { AcpxProviderDependencies } from "@executablemd/acp"; import { agentIdentityComponents } from "@executablemd/core"; import type { PermissionMode } from "@executablemd/core"; @@ -52,6 +55,7 @@ export interface ReplProfileSeams { * parameter: a caller states what it knows and leaves out the rest. */ export interface ReplProfileFacts extends ReplProfileSeams { + readonly testingHarness?: (includes: readonly string[]) => Operation; readonly includes?: readonly string[]; /** * The name `--sidekick-agent` carried, or none when the line named none. @@ -93,6 +97,7 @@ export interface ReplProfileAssembly { /** One finalized execution profile, shared by validation, descriptions and entries. */ export interface ReplExecutionProfile { + readonly testing?: ReplTesting; readonly projectCwd?: string; /** Where a declared name is looked for, outermost first. */ readonly includes: readonly string[]; @@ -252,7 +257,7 @@ export function* assembleReplProfile( installElicitation: noElicitationOfItsOwn, }); - return Object.freeze({ + const profile = Object.freeze({ includes, projectCwd, vocabulary: entryVocabulary(includes, plugins), @@ -282,6 +287,10 @@ export function* assembleReplProfile( // assembly rather than a second one standing in for it. sidekickAssembly: sidekick, }); + return Object.freeze({ + ...profile, + testing: replTesting(host, profile, workspace, facts.testingHarness), + }); }, }); } diff --git a/packages/cli/src/repl/program.ts b/packages/cli/src/repl/program.ts index 9f759c4f9..5d4c45521 100644 --- a/packages/cli/src/repl/program.ts +++ b/packages/cli/src/repl/program.ts @@ -1,3 +1,4 @@ +import { testForSidekick } from "./sidekick/admission.ts"; /** * The command, as one scope. * @@ -1068,6 +1069,13 @@ function* drive( return Ok(established); }, root: sidekickRoot, + test: (path) => + testForSidekick(path, { + held: admitting, + session: current, + testing: options.profile.testing, + ...(reopening.blocked === undefined ? {} : { blocked: reopening.blocked }), + }), submit(at, journal): Operation> { return submitForSidekick(at, { session: { diff --git a/packages/cli/src/repl/sidekick/admission.ts b/packages/cli/src/repl/sidekick/admission.ts index bbb2f322a..a09c21878 100644 --- a/packages/cli/src/repl/sidekick/admission.ts +++ b/packages/cli/src/repl/sidekick/admission.ts @@ -49,6 +49,8 @@ * guessed at here, and why reopening resolves before it admits anything else. */ +import type { ReplTesting, ReplTestReport } from "./testing.ts"; + import { ensure, Err, Ok } from "effection"; import type { Operation, Result } from "effection"; @@ -76,6 +78,7 @@ import type { ReplSubmission, StartingCheck } from "../session.ts"; * slow append and a slow entry never hold the keyboard. */ export interface Admitting { + testing?: boolean; /** The reservation a submission in flight holds, or none. */ held: Reservation | undefined; /** What that work reported, for the loop to apply on its next pass. */ @@ -114,6 +117,9 @@ export function take( slot: { readonly entry: string; readonly ordinal: number }, held: Admitting, ): Result { + if (held.testing === true) { + return Err(new Error("session tests are running or cleaning up")); + } const taken = admissible(by, facts, held.held); if (!taken.ok) { return taken; @@ -226,6 +232,9 @@ export function* submitForSidekick( // record claiming an attempt on terms this could have asked about. It is also // what keeps a busy request a refusal rather than work that queues behind the // entry in flight: nothing below waits for that entry to finish. + if (held.testing === true) { + return Err(new Error("session tests are running or cleaning up")); + } const admits = session.admits(); if (!admits.ok) { return admits; @@ -542,3 +551,34 @@ export function* retainResolution(writer: SidekickWriter, held: ColdReservation) export function blocking(error: Error): string { return `${error.message} Nothing can be admitted into this execution until that is resolved.`; } + +export function* testForSidekick( + path: string, + options: { + held: Admitting; + session: Pick; + testing: ReplTesting | undefined; + blocked?: string; + }, +): Operation> { + const { held, session, testing } = options; + if (held.testing === true || held.held !== undefined) { + return Err(new Error("an entry or test is running, preparing or cleaning up")); + } + const ready = session.admits(); + if (!ready.ok) { + return ready; + } + if (options.blocked !== undefined) { + return Err(new Error(options.blocked)); + } + if (testing === undefined) { + return Err(new Error("this REPL host has no test runner")); + } + held.testing = true; + try { + return yield* testing(path); + } finally { + held.testing = false; + } +} diff --git a/packages/cli/src/repl/sidekick/controls.ts b/packages/cli/src/repl/sidekick/controls.ts index 3b1b90cd1..29aa618a9 100644 --- a/packages/cli/src/repl/sidekick/controls.ts +++ b/packages/cli/src/repl/sidekick/controls.ts @@ -1,3 +1,4 @@ +import type { ReplTesting, ReplTestReport } from "./testing.ts"; /** * The words a reply may use, and what each of them asks the root to do. * @@ -123,6 +124,7 @@ export type GuideChapter = (typeof GUIDE_CHAPTERS)[number]; * unavailable may well ask for something else next. */ export interface SidekickRequests { + readonly test?: ReplTesting; /** The acknowledged published root values, detached. */ bindings(): Operation>; /** The active entry vocabulary, or the lookup of particular names. */ @@ -271,6 +273,7 @@ export const DESIGNATED: ReadonlySet = new Set([ "REPL.Environment", "REPL.EntryInput", "REPL.Submit", + "REPL.Test", "REPL.EntryOutput", ...ENTRY_INSPECTIONS, ELICIT_CONTROL, @@ -706,6 +709,27 @@ export function sidekickMutations( ); }, }), + writing("REPL.Test", "Test", "self-closing", { + kind: "function", + name: "REPL.Test", + props: { + type: "object", + properties: { path: { type: "string" } }, + required: ["path"], + additionalProperties: false, + }, + description: + 'Run a session test document or directory suite with ``. Read its completed results as feedback; add Output when the person needs to see them.', + *fn(props: Record): Operation { + const path = props.path; + if (typeof path !== "string") { + throw new Error("REPL.Test requires path"); + } + return yield* inspected(journal, { component: "REPL.Test", arguments: { path } }, () => + requests.test === undefined ? refusedTesting() : requests.test(path), + ); + }, + }), writing("REPL.Submit", "Submit", "self-closing", { kind: "function", name: "REPL.Submit", @@ -826,3 +850,7 @@ function longestRun(text: string, character: string): number { } return longest; } + +function* refusedTesting(): Operation> { + return Err(new Error("this REPL host has no test runner")); +} diff --git a/packages/cli/src/repl/sidekick/owner.ts b/packages/cli/src/repl/sidekick/owner.ts index 466ebf610..55aba1c01 100644 --- a/packages/cli/src/repl/sidekick/owner.ts +++ b/packages/cli/src/repl/sidekick/owner.ts @@ -1,3 +1,4 @@ +import type { ReplTesting } from "./testing.ts"; /** * The conversation this execution owns, and the one cycle it runs at a time. * @@ -341,6 +342,7 @@ export interface SidekickLiveTurn { /** What the owner needs, and nothing it could reach the rest of the root with. */ export interface SidekickOwnerInput { + readonly test?: ReplTesting; readonly workspace?: string; readonly assembly: SidekickAssembly; /** This execution's chat storage as it stands, which may hold nothing yet. */ @@ -534,6 +536,7 @@ export interface SidekickOwner { /** Which response this conversation's captured controls are answering for. */ interface ServingResponse { + readonly test?: ReplTesting; readonly journal: ControlJournal; /** Admit the draft for this response, recording a refusal that wrote nothing. */ submit(): Operation>; @@ -592,9 +595,18 @@ export function useSidekickOwner(input: SidekickOwnerInput): Operation> { - return yield* servingNow().submit(); - }), + sidekickRequests( + root, + function* (): Operation> { + return yield* servingNow().submit(); + }, + function* (path) { + const test = servingNow().test; + return test === undefined + ? Err(new Error("this REPL host has no test runner")) + : yield* test(path); + }, + ), { *began(occurrence, reserved): Operation { return yield* servingNow().journal.began(occurrence, reserved); @@ -1518,6 +1530,7 @@ export function useSidekickOwner(input: SidekickOwnerInput): Operation> { const taken = pairs; const answer = yield* input.submit(at, journal); diff --git a/packages/cli/src/repl/sidekick/profile.ts b/packages/cli/src/repl/sidekick/profile.ts index 1ebd71a9d..141e988d5 100644 --- a/packages/cli/src/repl/sidekick/profile.ts +++ b/packages/cli/src/repl/sidekick/profile.ts @@ -22,7 +22,7 @@ * Two classes. `read` observes and returns: the acknowledged bindings, the * entry vocabulary, the six guide chapters, the environment, and an entry's * output. `write` is what changes what a person sees or asks them something: - * filling the shared draft, submitting it, and canonical ``. Keeping + * filling the shared draft, submitting it, executing tests, and canonical ``. Keeping * them separate is what lets a captured ceiling state that a reply which only * read could not have written and interrupted nobody — something nobody can say * about a single undifferentiated list. diff --git a/packages/cli/src/repl/sidekick/requests.ts b/packages/cli/src/repl/sidekick/requests.ts index c2f920181..aec3442db 100644 --- a/packages/cli/src/repl/sidekick/requests.ts +++ b/packages/cli/src/repl/sidekick/requests.ts @@ -1,3 +1,4 @@ +import type { ReplTesting } from "./testing.ts"; /** * What the root actually does when a reply asks, and where each answer comes from. * @@ -205,8 +206,13 @@ export type SubmitForReply = () => Operation>; * The reads are stateless over the root, so they cost a response that does not * use them nothing. */ -export function sidekickRequests(root: RootReading, submit: SubmitForReply): SidekickRequests { +export function sidekickRequests( + root: RootReading, + submit: SubmitForReply, + test?: ReplTesting, +): SidekickRequests { return { + ...(test === undefined ? {} : { test }), *bindings(): Operation> { // The root data as it stands: the retained head for every entry that has // settled, and the current root of the entry running now on top of it. diff --git a/packages/cli/src/repl/sidekick/testing.ts b/packages/cli/src/repl/sidekick/testing.ts new file mode 100644 index 000000000..58c8ddfe3 --- /dev/null +++ b/packages/cli/src/repl/sidekick/testing.ts @@ -0,0 +1,259 @@ +import { allSettled, ensure, Err, Ok, scoped, spawn } from "effection"; +import type { Operation, Result, Scope, Task } from "effection"; +import { Stdio } from "@effectionx/process"; +import { forEach } from "@effectionx/stream-helpers"; +import { isAbsolute, relative, resolve, sep } from "node:path"; +import { + Component, + Execution, + SecretDetectedError, + SecretScannerError, + useNormalizedOutput, +} from "@executablemd/core"; +import { executeInstalled } from "@executablemd/core/host"; +import type { ExecutionInstallation } from "@executablemd/core/host"; +import { API, asFilesFatal } from "@executablemd/runtime"; +import { + InMemoryStream, + DurablePersistenceError, + DivergenceError, + StaleInputError, + TerminalDivergenceError, + ContinuePastCloseDivergenceError, +} from "@executablemd/durable-streams"; +import { useTesting, testHarnessInstallation } from "@executablemd/testing"; +import type { TestResult } from "@executablemd/testing"; +import { installTestAgentComponents } from "@executablemd/test-agent"; +import { installAgentComponents } from "@executablemd/core"; +import { resolveTestTarget } from "../../test-target.ts"; +import type { ReplExecutionProfile } from "../../repl-profile.ts"; + +type TestStatus = "passed" | "failed" | "no-tests"; +type CompletedTest = { + status: "pass" | "fail"; + location: string; + name?: string; + error?: { + kind: "assertion" | "timeout" | "teardown" | "error"; + message: string; + actual?: string; + expected?: string; + }; +}; +type TestDocumentReport = { + path: string; + status: TestStatus; + tests: CompletedTest[]; + error: { name: string; message: string } | null; + report: string; +}; +export type ReplTestReport = { path: string; status: TestStatus; documents: TestDocumentReport[] }; +export type ReplTesting = (path: string) => Operation>; + +export function replTesting( + host: Scope, + profile: ReplExecutionProfile, + workspace: string | undefined, + harness?: (includes: readonly string[]) => Operation, +): ReplTesting { + return (path) => + owned(host, function* () { + if (workspace === undefined) { + return Err(new Error("this execution has no session workspace for testing")); + } + const selected = resolve(workspace, path); + const local = relative(workspace, selected); + if ( + path.trim() === "" || + isAbsolute(path) || + local === ".." || + local.startsWith(`..${sep}`) + ) { + return Err( + new Error( + "REPL.Test path must be a non-empty session-relative path that stays in the session workspace", + ), + ); + } + let documents: readonly string[]; + try { + const target = yield* resolveTestTarget(selected, ["**/*.test.md"]); + documents = + target.kind === "file" + ? [target.path] + : target.documents.map((document) => document.path); + } catch (error) { + const fatal = fatalFailure(error); + if (fatal !== undefined) { + throw fatal; + } + return Err(error); + } + const reports: TestDocumentReport[] = []; + for (const document of documents) { + reports.push( + yield* owned(host, function* () { + yield* API.Env.around({ + *cwd() { + return workspace; + }, + }); + yield* useNormalizedOutput(); + let nativeOutput = ""; + const stdout = new TextDecoder(); + const stderr = new TextDecoder(); + yield* Stdio.around( + { + *stdout([bytes]) { + nativeOutput += stdout.decode(bytes, { stream: true }); + }, + *stderr([bytes]) { + nativeOutput += stderr.decode(bytes, { stream: true }); + }, + }, + { at: "min" }, + ); + let reachedPolicy = false; + let acceptedPolicy = false; + yield* Execution.around({ + *execute([request], next) { + request.addCompletionFailure(() => { + reachedPolicy = true; + return undefined; + }); + yield* next(request); + }, + }); + const testing = yield* useTesting(); + yield* Execution.around({ + *execute([request], next) { + yield* next(request); + request.addCompletionFailure(() => { + acceptedPolicy = true; + return undefined; + }); + }, + }); + yield* installTestAgentComponents(); + yield* installAgentComponents(); + const installations = [ + ...profile.installations, + harness === undefined ? testHarnessInstallation() : yield* harness(profile.includes), + ]; + yield* Component.around({ generated: (_args, next) => next() ?? profile.generated }); + const execution = yield* executeInstalled( + { path: document, includes: [...profile.includes], stream: new InMemoryStream() }, + installations, + ); + const draining = yield* spawn(() => + forEach(function* (_chunk: string) {}, execution.output), + ); + const outcome = yield* execution; + const rendered = yield* draining; + const report = nativeOutput + stdout.decode() + stderr.decode() + rendered; + const tests = (yield* testing.results).map(detachTest); + if (!outcome.ok) { + const fatal = fatalFailure(outcome.error); + if (fatal !== undefined) { + throw fatal; + } + if (acceptedPolicy) { + throw outcome.error; + } + } + return { + path: relative(workspace, document).split(sep).join("/"), + status: + outcome.ok && tests.length > 0 + ? "passed" + : reachedPolicy && tests.length === 0 + ? "no-tests" + : "failed", + tests, + error: outcome.ok + ? null + : { name: outcome.error.name, message: outcome.error.message }, + report, + }; + }), + ); + } + const status = + reports.length === 0 || reports.every((document) => document.status === "no-tests") + ? "no-tests" + : reports.every((document) => document.status === "passed") + ? "passed" + : "failed"; + return Ok({ path, status, documents: reports }); + }); +} + +function owned(host: Scope, work: () => Operation): Operation { + return scoped(function* () { + let child: Task> | undefined; + yield* ensure(function* () { + if (child !== undefined) { + yield* child.halt(); + } + }); + child = host.run(function* () { + const [result] = yield* allSettled([work()]); + if (result === undefined) { + throw new Error("test task produced no result"); + } + return result; + }); + const result = yield* child; + if (!result.ok) { + throw result.error; + } + return result.value; + }); +} + +function detachTest(result: TestResult): CompletedTest { + return { + status: result.status, + location: result.location, + ...(result.name === undefined ? {} : { name: result.name }), + ...(result.error === undefined + ? {} + : { + error: { + kind: result.error.kind, + message: result.error.message, + ...(result.error.actual === undefined ? {} : { actual: result.error.actual }), + ...(result.error.expected === undefined ? {} : { expected: result.error.expected }), + }, + }), + }; +} + +function fatalFailure(error: unknown): Error | undefined { + if ( + error instanceof DurablePersistenceError || + error instanceof DivergenceError || + error instanceof StaleInputError || + error instanceof TerminalDivergenceError || + error instanceof ContinuePastCloseDivergenceError || + error instanceof SecretDetectedError || + error instanceof SecretScannerError + ) { + return error; + } + const files = asFilesFatal(error); + if (files !== undefined) { + return files; + } + if (error instanceof AggregateError) { + for (const cause of error.errors) { + const fatal = fatalFailure(cause); + if (fatal !== undefined) { + return fatal; + } + } + } + return error instanceof Error && error.cause !== undefined + ? fatalFailure(error.cause) + : undefined; +} diff --git a/packages/cli/tests/fixtures/repl/issue-883/sidekick-harness.ts b/packages/cli/tests/fixtures/repl/issue-883/sidekick-harness.ts index d7f6b14d0..dbf9312fc 100644 --- a/packages/cli/tests/fixtures/repl/issue-883/sidekick-harness.ts +++ b/packages/cli/tests/fixtures/repl/issue-883/sidekick-harness.ts @@ -1,3 +1,5 @@ +import { testForSidekick } from "../../../../src/repl/sidekick/admission.ts"; +import { pairedLocation } from "../../../../src/repl/sidekick/history.ts"; /** * One Sidekick, assembled the way the command assembles it (#883). * @@ -63,6 +65,7 @@ export const AGENT = "scripted-sidekick"; /** One case's world: the real pieces, and the one scripted provider. */ export interface Harness { + readonly workspace: string; readonly fake: FakeAcp; /** The repository, so a case can reopen this execution as a cold one would. */ readonly repository: PairedReplRepository; @@ -154,6 +157,7 @@ export function* useHarness( if (!stack.ok) { throw stack.error; } + const workspace = pairedLocation(join(dataRoot, "repl"), execution.id).directory; const profile = yield* (yield* assembleReplProfile(stack.value, NO_PLUGINS, { acp: { createRuntime: fake.create, @@ -166,7 +170,7 @@ export function* useHarness( // availability refusal rather than reaching the fake. sidekickAgent: AGENT, planWriterRoot: yield* useTemporary("xmd-sidekick-plan-"), - })).finalize(); + })).finalize(workspace); const opened = yield* openReplSession({ execution, @@ -238,6 +242,7 @@ export function* useHarness( return Ok(execution.chat as PairedChatStorage); }, root, + test: (path) => testForSidekick(path, { held: admitting, session, testing: profile.testing }), submit(at, journal: ControlJournal): Operation> { return submitForSidekick(at, { session: { @@ -276,7 +281,17 @@ export function* useHarness( }, }); - return { fake, repository, owner, session, execution, profile, admitting, state: () => state }; + return { + workspace, + fake, + repository, + owner, + session, + execution, + profile, + admitting, + state: () => state, + }; } /** Every record this execution's chat history holds. */ diff --git a/packages/cli/tests/repl-sidekick-controls.test.ts b/packages/cli/tests/repl-sidekick-controls.test.ts index dea7d1d93..dd12617e4 100644 --- a/packages/cli/tests/repl-sidekick-controls.test.ts +++ b/packages/cli/tests/repl-sidekick-controls.test.ts @@ -637,6 +637,7 @@ describe("Tier R3 — what the ceiling admits, and what it does not", () => { "REPL.EntryInput", "REPL.Environment", "REPL.Submit", + "REPL.Test", "REPL.Syntax", // The four bounded reads of one entry's saved evidence, which are what // a response reads an entry with now. @@ -661,7 +662,7 @@ describe("Tier R3 — what the ceiling admits, and what it does not", () => { .filter((entry) => entry.effect === "write") .map((entry) => entry.name) .sort(), - ).toEqual(["Elicit", "REPL.EntryInput", "REPL.Submit"]); + ).toEqual(["Elicit", "REPL.EntryInput", "REPL.Submit", "REPL.Test"]); expect(surface.find((entry) => entry.name === "REPL.EntryInput")?.form).toBe("paired"); expect(surface.find((entry) => entry.name === "REPL.Submit")?.form).toBe("self-closing"); // Paired, because the question is the content. @@ -728,6 +729,7 @@ describe("Tier R3 — what the ceiling admits, and what it does not", () => { "REPL.EntryInput", "REPL.Environment", "REPL.Submit", + "REPL.Test", "REPL.Syntax", "REPL.Entry.Source", "REPL.Entry.Prompt", diff --git a/packages/cli/tests/repl-sidekick-testing.test.ts b/packages/cli/tests/repl-sidekick-testing.test.ts new file mode 100644 index 000000000..a6e11d4af --- /dev/null +++ b/packages/cli/tests/repl-sidekick-testing.test.ts @@ -0,0 +1,794 @@ +import { DurablePersistenceError } from "@executablemd/durable-streams"; +import { submitForSidekick } from "../src/repl/sidekick/admission.ts"; +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { useComponentModuleLoader } from "@executablemd/test-support/component-modules"; +import { ensureDir, exists, readTextFile, rm, writeTextFile } from "@effectionx/fs"; +import { allSettled, ensure, Ok, scoped, spawn, suspend, until, withResolvers } from "effection"; +import type { Operation } from "effection"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { symlink } from "node:fs/promises"; +import { z } from "zod"; +import { SecretScannerError, registerComponents } from "@executablemd/core"; +import { Execution } from "@executablemd/core/api"; +import { + Permissions, + API, + useHostFiles, + FilesProviderUnavailableError, +} from "@executablemd/runtime"; +import { readGuideChapter } from "../src/repl/sidekick/guide.ts"; +import { pairedLocation } from "../src/repl/sidekick/history.ts"; +import { testForSidekick, take } from "../src/repl/sidekick/admission.ts"; +import { encodeLocation } from "../src/repl/route.ts"; +import { initialState } from "../src/repl/application.ts"; +import { useHarness, useTemporary } from "./fixtures/repl/issue-883/sidekick-harness.ts"; +import { + useMountedRepl, + useMountedHost, + mountedExecution, + mountedReading, + sendMessage, + awaiting, + shows, + focusDraft, + typing, + click, + screenOf, +} from "./fixtures/repl/issue-883/mounted-program.ts"; +import type { MountedRepl } from "./fixtures/repl/issue-883/mounted-program.ts"; + +const reportSchema = z + .object({ + path: z.string(), + status: z.enum(["passed", "failed", "no-tests"]), + documents: z.array( + z + .object({ + path: z.string(), + status: z.enum(["passed", "failed", "no-tests"]), + tests: z.array( + z + .object({ + status: z.enum(["pass", "fail"]), + name: z.string().optional(), + location: z.string(), + error: z + .object({ + kind: z.enum(["assertion", "timeout", "teardown", "error"]), + message: z.string(), + actual: z.string().optional(), + expected: z.string().optional(), + }) + .strict() + .optional(), + }) + .strict(), + ), + error: z.object({ name: z.string(), message: z.string() }).strict().nullable(), + report: z.string(), + }) + .strict(), + ), + }) + .strict(); +const ROOT = fileURLToPath(new URL("../../../", import.meta.url)); + +function submit(source: string): string { + return `\n{code}\n`; +} +function write(path: string, text: string): string { + return `\n{file}`; +} +function* mounted( + includes?: readonly string[], +): Operation<{ repl: MountedRepl; workspace: string }> { + yield* useComponentModuleLoader(); + const project = yield* useTemporary("xmd-testing-project-"); + const repl = yield* useMountedRepl({ + projectCwd: project, + hostFiles: true, + ...(includes === undefined ? {} : { includes }), + }); + const workspace = pairedLocation(repl.root, yield* mountedExecution(repl)).directory; + yield* writeTextFile(join(workspace, "package.json"), '{"type":"module"}'); + yield* until(symlink(join(ROOT, "node_modules"), join(workspace, "node_modules"), "dir")); + return { repl, workspace }; +} +let serial = 0; +function* exchange(repl: MountedRepl, replies: readonly string[]): Operation { + const done = `testing-finished-${serial++}`; + for (const reply of replies) { + repl.fake.script({ reply }); + } + repl.fake.script({ reply: `${done}` }); + yield* sendMessage(repl, `run batch ${serial}`); + yield* awaiting(repl.terminal, done, (terminal) => shows(terminal, done)); +} +function* reports(repl: MountedRepl) { + return (yield* mountedReading(repl)).chat.flatMap((record) => + record.kind === "control-outcome" && + record.observed !== undefined && + reportSchema.safeParse(record.observed).success + ? [reportSchema.parse(record.observed)] + : [], + ); +} +function passing(name = "real assertion", expected = "yes"): string { + return ``; +} + +describe("REPL.Test accepted journeys", () => { + it("R1: authors, fails, fixes and reuses ordinary Markdown without executing its creation", function* () { + const { repl, workspace } = yield* mounted(); + const chapter = yield* readGuideChapter("Entries"); + if (!chapter.ok) { + throw chapter.error; + } + const journey = + chapter.value + .split("### Test, fix and reuse a session component")[1] + ?.split("## Common mistake")[0] ?? ""; + const examples = [...journey.matchAll(/````xmd\n([\s\S]*?)\n````/g)].map((match) => match[1]); + expect(examples).toHaveLength(5); + const [create, single, fail, fix, reuse] = examples; + if ( + create === undefined || + single === undefined || + fail === undefined || + fix === undefined || + reuse === undefined + ) { + throw new Error("incomplete shipped testing examples"); + } + const greeting = + 'Hello from this session.Hello from this session.'; + yield* exchange(repl, [create, ""]); + expect(yield* exists(join(workspace, "greeting.txt"))).toBe(false); + expect(yield* reports(repl)).toEqual([]); + expect((yield* mountedReading(repl)).model.entries).toHaveLength(1); + expect(yield* readTextFile(join(workspace, "components/Greeting.md"))).toBe(greeting); + yield* exchange(repl, [single]); + expect((yield* reports(repl)).at(-1)?.status).toBe("passed"); + expect(yield* readTextFile(join(workspace, "greeting.txt"))).toBe("Hello from this session."); + yield* exchange(repl, [fail]); + const failed = (yield* reports(repl)).at(-1); + expect(failed?.status).toBe("failed"); + expect(failed?.documents[0]?.tests[0]).toMatchObject({ + name: "greeting", + status: "fail", + error: { kind: "assertion", actual: '"Hello from this session."', expected: '"wrong"' }, + }); + expect(repl.fake.prompts.join("\n")).toContain("wrong"); + yield* exchange(repl, [fix, reuse]); + expect((yield* reports(repl)).at(-1)?.status).toBe("passed"); + yield* focusDraft(repl.terminal); + yield* typing(repl.terminal, ""); + repl.terminal.feed("\r"); + yield* awaiting( + repl.terminal, + "manual reuse", + () => + screenOf(repl.terminal).join("\n").includes("entry-5") && + screenOf(repl.terminal).join("\n").includes("[ok]"), + ); + const read = yield* mountedReading(repl); + expect(read.model.entries.slice(-2).map((entry) => entry.source)).toEqual([ + "", + "", + ]); + expect(read.model.entries.slice(-2).map((entry) => entry.terminal?.output)).toEqual([ + "Hello from this session.", + "Hello from this session.", + ]); + }); + + it("R2: tests and entries use a same-path TypeScript edit while old source and output stay retained", function* () { + const { repl, workspace } = yield* mounted(); + const test = (value: string) => + ``; + for (const version of ["first", "second"]) { + yield* exchange(repl, [ + submit( + write( + "components/Direct.ts", + `export default function*() { const value: string = ${JSON.stringify(version)}; return value; }`, + ) + + "\n" + + write("components/Direct.test.md", test(version)), + ), + '', + submit(""), + ]); + } + expect((yield* reports(repl)).map((report) => report.status)).toEqual(["passed", "passed"]); + const read = yield* mountedReading(repl); + expect( + read.model.entries + .filter((entry) => entry.source === "") + .map((entry) => entry.terminal?.output), + ).toEqual(["first", "second"]); + const journal = yield* readTextFile(pairedLocation(repl.root, read.execution).entries); + expect(journal).toContain('const value: string = \\"first\\"'); + expect(journal).toContain('const value: string = \\"second\\"'); + expect(yield* readTextFile(join(workspace, "components/Direct.ts"))).toContain('"second"'); + }); + + it("R3: reports real ordered results, empty selections, no-tests, parse/runtime failures and partial output", function* () { + const { repl, workspace } = yield* mounted(); + yield* ensureDir(join(workspace, "empty")); + yield* writeTextFile(join(workspace, "components/a.test.md"), passing("first fails", "wrong")); + yield* writeTextFile( + join(workspace, "components/b.test.md"), + "", + ); + yield* exchange(repl, ['']); + let suite = (yield* reports(repl)).at(-1); + expect(suite?.status).toBe("failed"); + expect(suite?.documents.map((document) => document.path)).toEqual([ + "components/a.test.md", + "components/b.test.md", + ]); + expect(suite?.documents.map((document) => document.tests.length)).toEqual([1, 1]); + expect(suite?.documents[1]?.tests[0]?.name).toBeUndefined(); + expect(suite?.documents[1]?.tests[0]?.location).toContain("b.test.md"); + yield* writeTextFile(join(workspace, "components/a.test.md"), passing()); + yield* exchange(repl, ['']); + expect((yield* reports(repl)).at(-1)?.status).toBe("passed"); + const scenarios: ReadonlyArray = [ + ["empty", undefined, "no-tests"], + ["components/zero.md", "no tests here", "no-tests"], + ["components/boundary.md", "", "no-tests"], + ["components/parse.md", "---\nprops: [unclosed\n---\n", "failed"], + [ + "components/runtime.md", + 'earlier rendered text\n\n', + "failed", + ], + [ + "components/unexpected.md", + '', + "failed", + ], + ]; + for (const [path, text, expected] of scenarios) { + if (text !== undefined) { + yield* writeTextFile(join(workspace, path), text); + } + yield* exchange(repl, [``]); + suite = (yield* reports(repl)).at(-1); + expect({ path, status: suite?.status }).toEqual({ path, status: expected }); + expect(suite?.path).toBe(path); + if (text === undefined) { + expect(suite?.documents).toEqual([]); + continue; + } + expect(suite?.documents[0]?.error).not.toBeNull(); + if (path.includes("zero") || path.includes("boundary")) { + expect(suite?.documents[0]?.tests).toEqual([]); + expect(suite?.documents[0]?.error?.message).toContain("no tests"); + } + if (path.includes("runtime")) { + expect(suite?.documents[0]?.report).toContain("earlier rendered text"); + } + if (path.includes("unexpected")) { + expect(suite?.documents[0]?.tests[0]?.error?.kind).toBe("error"); + } + } + yield* writeTextFile(join(workspace, "components/c.test.md"), "valid without Test"); + yield* exchange(repl, ['']); + expect((yield* reports(repl)).at(-1)?.status).toBe("failed"); + expect((yield* reports(repl)).at(-1)?.documents.map((document) => document.status)).toEqual([ + "passed", + "passed", + "no-tests", + ]); + for (const path of ["", "/tmp/absolute", "../escape"]) { + const before = (yield* reports(repl)).length; + yield* exchange(repl, [``]); + expect(yield* reports(repl)).toHaveLength(before); + } + yield* writeTextFile(join(workspace, "literal#%.md"), passing()); + yield* exchange(repl, ['']); + expect((yield* reports(repl)).at(-1)?.documents[0]?.path).toBe("literal#%.md"); + }); +}); + +describe("REPL.Test selection, display and isolation", () => { + it("R4: preserves configured selection, immutable declarations, document isolation and explicit display", function* () { + const configured = yield* useTemporary("xmd-testing-includes-"); + yield* writeTextFile(join(configured, "Greeting.md"), "configured greeting"); + yield* registerComponents([ + { + name: "HostValue", + props: { type: "object" }, + origin: "test://host-value", + fn: function* () { + return "host declaration"; + }, + }, + ]); + const { repl, workspace } = yield* mounted([configured]); + yield* writeTextFile(join(workspace, "components/Greeting.md"), "session greeting"); + yield* writeTextFile( + join(workspace, "components/selection.test.md"), + '', + ); + yield* exchange(repl, [ + '', + submit(''), + '', + '', + ]); + expect(repl.fake.prompts.join("\n")).toContain(join(configured, "Greeting.md")); + expect((yield* reports(repl)).map((report) => report.status)).toEqual(["passed", "passed"]); + expect((yield* mountedReading(repl)).model.entries[0]?.terminal?.output).toBe( + "configured greeting", + ); + const chat = (yield* mountedReading(repl)).chat.filter((record) => record.kind === "response"); + expect(chat.slice(-3, -1).every((record) => !record.hasOutput && record.output === "")).toBe( + true, + ); + yield* exchange(repl, [ + '', + ]); + const withOutput = (yield* mountedReading(repl)).chat + .filter((record) => record.kind === "response") + .at(-2); + expect(withOutput?.hasOutput).toBe(true); + expect(reportSchema.parse(JSON.parse(withOutput?.output ?? "null"))).toEqual( + (yield* reports(repl)).at(-1), + ); + yield* writeTextFile( + join(workspace, "components/Registrar.ts"), + [ + 'import { registerComponents } from "@executablemd/core";', + 'export default function*() { yield* registerComponents([{name:"DocumentPrivate", origin:"test://document",props:{type:"object"},fn:function*(){return "leaked";}}]); return ""; }', + ].join("\n"), + ); + yield* ensureDir(join(workspace, "isolated")); + yield* writeTextFile( + join(workspace, "isolated/a.test.md"), + '', + ); + yield* writeTextFile( + join(workspace, "isolated/b.test.md"), + '', + ); + yield* exchange(repl, ['']); + const isolated = (yield* reports(repl)).at(-1); + expect(isolated?.documents.map((document) => document.status)).toEqual(["passed", "failed"]); + expect(isolated?.documents[1]?.tests.map((test) => test.status)).toEqual(["pass", "fail"]); + expect(isolated?.documents[1]?.tests[1]?.error?.message).toContain( + "Cannot resolve component: DocumentPrivate", + ); + yield* exchange(repl, [ + submit(""), + submit(""), + ]); + const entries = (yield* mountedReading(repl)).model.entries; + expect(entries.map((entry) => ({ source: entry.source, terminal: entry.terminal }))).toEqual([ + expect.objectContaining({ source: '' }), + expect.objectContaining({ source: "" }), + ]); + expect(entries.at(-1)?.terminal?.output).toContain("undefined"); + expect(entries).toHaveLength(2); + expect(repl.fake.prompts.join("\n")).toContain("Cannot resolve component: DocumentPrivate"); + yield* writeTextFile(join(configured, "Broken.md"), "---\nprops: [unclosed\n---\n"); + yield* writeTextFile(join(workspace, "components/Broken.md"), "fallback must not run"); + yield* writeTextFile( + join(workspace, "components/broken.test.md"), + '', + ); + yield* exchange(repl, ['']); + expect((yield* reports(repl)).at(-1)?.status).toBe("failed"); + expect((yield* reports(repl)).at(-1)?.documents[0]?.report).not.toContain( + "fallback must not run", + ); + expect( + (yield* mountedReading(repl)).chat.filter((record) => record.kind === "response").at(-2) + ?.hasOutput, + ).toBe(false); + yield* writeTextFile(join(workspace, "protected.txt"), "canonical file"); + yield* writeTextFile(join(workspace, "components/File.md"), "unprotected spoof"); + yield* writeTextFile( + join(workspace, "components/protected.test.md"), + '', + ); + yield* exchange(repl, ['']); + expect((yield* reports(repl)).at(-1)?.status).toBe("passed"); + }); + + it("R1 refusal: a completed entry with a denied File never reports successful component availability", function* () { + yield* Permissions.around({ files: ([operation]) => operation !== "write" }); + const { repl, workspace } = yield* mounted(); + yield* exchange(repl, [ + submit(write("components/Never.md", 'ranunavailable')), + "", + ]); + expect(yield* exists(join(workspace, "components/Never.md"))).toBe(false); + expect(yield* exists(join(workspace, "never-ran.txt"))).toBe(false); + expect(yield* reports(repl)).toEqual([]); + expect(repl.fake.prompts.join("\n")).toContain("permission"); + }); +}); + +function* holdModule( + workspace: string, + key: string, + body: () => void, + cleanup: () => Operation, +): Operation { + Reflect.set(globalThis, key, { body, cleanup }); + yield* ensure(function* () { + Reflect.deleteProperty(globalThis, key); + }); + yield* writeTextFile( + join(workspace, "components/Hold.ts"), + [ + 'import { ensure, suspend } from "effection";', + `const state = Reflect.get(globalThis, ${JSON.stringify(key)});`, + 'export default function*() { yield* ensure(() => state.cleanup()); state.body(); yield* suspend(); return "unreachable"; }', + ].join("\n"), + ); + yield* writeTextFile( + join(workspace, "components/hold.test.md"), + '', + ); +} + +describe("REPL.Test admission and joined lifetime", () => { + it("R5: refuses both submitters and a second test throughout response cancellation and gated cleanup", function* () { + yield* useHostFiles(); + yield* useComponentModuleLoader(); + const held = yield* useHarness(); + yield* writeTextFile(join(held.workspace, "package.json"), '{"type":"module"}'); + yield* until(symlink(join(ROOT, "node_modules"), join(held.workspace, "node_modules"), "dir")); + const entered = withResolvers(); + const cleaning = withResolvers(); + const release = withResolvers(); + yield* holdModule(held.workspace, `held-${held.workspace}`, entered.resolve, function* () { + cleaning.resolve(); + yield* release.operation; + }); + held.fake.script({ + reply: + 'preserved draft', + }); + const responding = yield* spawn(() => held.owner.message("hold this test")); + yield* entered.operation; + const before = held.state(); + function facts() { + return { + live: true, + revisions: held.state().revisions, + takenOver: false, + draft: held.state().draft, + }; + } + function* refusal(): Operation { + expect(take("person", facts(), { entry: "entry-1", ordinal: 0 }, held.admitting).ok).toBe( + false, + ); + const second = yield* testForSidekick("components", { + held: held.admitting, + session: held.session, + testing: held.profile.testing, + }); + expect(second.ok).toBe(false); + const submitted = yield* submitForSidekick( + { conversation: "conversation:1", response: "response:1" }, + { + session: { + admits: () => held.session.admits(), + entries: () => Ok([]), + submit: (source, starting) => held.session.submit(source, undefined, starting), + }, + facts, + held: held.admitting, + writer: held.execution.stream, + woke() {}, + journal: { + *began() { + throw new Error("refused submission wrote an intent"); + }, + *settled() { + throw new Error("refused submission wrote an outcome"); + }, + }, + }, + ); + expect(submitted.ok).toBe(false); + expect(held.state()).toBe(before); + expect(held.session.model.entries).toHaveLength(0); + } + yield* refusal(); + let stopped = false; + const stopping = yield* spawn(function* () { + yield* held.owner.stop(); + stopped = true; + }); + yield* cleaning.operation; + expect(stopped).toBe(false); + expect(held.admitting.testing).toBe(true); + yield* refusal(); + release.resolve(); + yield* stopping; + yield* allSettled([responding]); + expect(held.admitting.testing).toBe(false); + expect(held.state().draft).toBe("preserved draft"); + const read = yield* held.repository.open(held.execution.id); + if (read.chat.form !== "paired") { + throw new Error("missing paired chat"); + } + expect( + read.chat.records.filter( + (record) => + record.kind === "control-outcome" && reportSchema.safeParse(record.observed).success, + ), + ).toEqual([]); + yield* writeTextFile(join(held.workspace, "components/hold.test.md"), passing()); + held.fake.script({ reply: '' }); + held.fake.script({ reply: "after cleanup" }); + const next = yield* held.owner.message("now run it"); + expect(next.ok).toBe(true); + }); + + it("R5: test admission refuses a real entry in preflight, executing, paused and tearing down; Stop preserves its owner", function* () { + yield* useHostFiles(); + const reading = withResolvers(); + const allowRead = withResolvers(); + let gated = false; + yield* API.Fs.around({ + *readTextFile([path], next) { + if (String(path).endsWith("EntryGate.md") && !gated) { + gated = true; + reading.resolve(); + yield* allowRead.operation; + } + return yield* next(path); + }, + }); + const entered = withResolvers(); + const bodyRelease = withResolvers(); + const cleaning = withResolvers(); + const cleanupRelease = withResolvers(); + yield* registerComponents([ + { + name: "EntryHold", + origin: "test://entry-hold", + props: { type: "object" }, + *fn() { + yield* ensure(function* () { + cleaning.resolve(); + yield* cleanupRelease.operation; + }); + entered.resolve(); + yield* bodyRelease.operation; + return "entry survived Stop"; + }, + }, + ]); + const held = yield* useHarness(); + yield* writeTextFile(join(held.workspace, "components/EntryGate.md"), ""); + yield* writeTextFile(join(held.workspace, "components/pass.test.md"), passing()); + held.fake.script({ reply: submit("") }); + const response = yield* spawn(() => held.owner.message("run the ordinary entry")); + yield* reading.operation; + expect(held.admitting.held).toBeDefined(); + const ask = () => + testForSidekick("components/pass.test.md", { + held: held.admitting, + session: held.session, + testing: held.profile.testing, + }); + expect((yield* ask()).ok).toBe(false); + allowRead.resolve(); + yield* entered.operation; + expect((yield* ask()).ok).toBe(false); + const controller = held.session.controller; + if (controller === undefined) { + throw new Error("entry lost its controller"); + } + controller.pause(); + expect((yield* ask()).ok).toBe(false); + controller.resume(); + yield* held.owner.stop(); + expect(held.session.admits().ok).toBe(false); + bodyRelease.resolve(); + yield* cleaning.operation; + expect((yield* ask()).ok).toBe(false); + cleanupRelease.resolve(); + yield* held.session.join(); + yield* allSettled([response]); + expect(held.session.model.entries[0]?.terminal?.output).toBe("entry survived Stop"); + expect((yield* ask()).ok).toBe(true); + }); + + it("R5: Exit joins response cleanup and normal test timeout/teardown keep their real result kinds", function* () { + const { repl, workspace } = yield* mounted(); + yield* writeTextFile( + join(workspace, "components/Timeout.ts"), + 'import { suspend } from "effection"; export default function*() { yield* suspend(); }', + ); + yield* writeTextFile( + join(workspace, "components/kinds.test.md"), + '\n\n```js persist eval\nyield* spawn(function* () { try { yield* suspend(); } finally { throw new Error("per-test teardown"); } });\n```\n\n', + ); + yield* exchange(repl, ['']); + const tests = (yield* reports(repl)).at(-1)?.documents[0]?.tests; + expect(tests?.map((test) => [test.status, test.error?.kind])).toEqual([ + ["fail", "timeout"], + ["fail", "teardown"], + ["pass", undefined], + ]); + const entered = withResolvers(); + const cleaning = withResolvers(); + const release = withResolvers(); + yield* holdModule(workspace, `exit-${workspace}`, entered.resolve, function* () { + cleaning.resolve(); + yield* release.operation; + }); + repl.fake.script({ reply: '' }); + yield* sendMessage(repl, "cancel on exit"); + yield* entered.operation; + repl.terminal.end(); + yield* cleaning.operation; + expect(repl.outcome()).toBeUndefined(); + release.resolve(); + yield* awaiting(repl.terminal, "exit joining test cleanup", () => repl.outcome() !== undefined); + expect(yield* reports(repl)).toHaveLength(1); + }); +}); + +describe("REPL.Test retained and fatal outcomes", () => { + it("R6: reopen inspection keeps completed reports inert and unmatched work unknown until a new explicit request", function* () { + yield* useComponentModuleLoader(); + const host = yield* useMountedHost({ + projectCwd: yield* useTemporary("repl-testing-history-"), + hostFiles: true, + }); + let id = ""; + let workspace = ""; + let entriesBefore = ""; + let chatBefore = ""; + let bodies = 0; + yield* scoped(function* () { + const repl = yield* useMountedRepl({ host }); + id = yield* mountedExecution(repl); + const location = pairedLocation(repl.root, id); + workspace = location.directory; + yield* writeTextFile(join(workspace, "package.json"), '{"type":"module"}'); + yield* until(symlink(join(ROOT, "node_modules"), join(workspace, "node_modules"), "dir")); + yield* exchange(repl, [ + submit( + write("components/Greeting.md", "historical greeting") + + "\n" + + write("components/pass.test.md", passing()), + ), + '', + submit(""), + ]); + const entered = withResolvers(); + yield* holdModule( + workspace, + `history-${workspace}`, + () => { + bodies++; + entered.resolve(); + }, + function* () {}, + ); + repl.fake.script({ reply: '' }); + yield* sendMessage(repl, "start interrupted tests"); + yield* entered.operation; + yield* click(repl.terminal, "[stop]"); + yield* awaiting(repl.terminal, "Stop joins", (terminal) => shows(terminal, "[send]")); + expect(yield* reports(repl)).toHaveLength(1); + repl.terminal.end(); + yield* awaiting(repl.terminal, "first close", () => repl.outcome() !== undefined); + entriesBefore = yield* readTextFile(location.entries); + chatBefore = yield* readTextFile(location.sidekick); + }); + const prompts = host.fake.prompts.length; + yield* rm(join(workspace, "components/Greeting.md")); + yield* rm(join(workspace, "components/pass.test.md")); + const location = pairedLocation(host.root, id); + const reopened = yield* useMountedRepl({ + host, + location: encodeLocation(initialState(id).route), + }); + const reading = yield* mountedReading(reopened); + expect(reading.model.entries.at(-1)?.terminal?.output).toBe("historical greeting"); + expect((yield* reports(reopened))[0]?.status).toBe("passed"); + expect(host.fake.prompts).toHaveLength(prompts); + expect(bodies).toBe(1); + expect(yield* readTextFile(location.entries)).toBe(entriesBefore); + expect(yield* readTextFile(location.sidekick)).toBe(chatBefore); + reopened.fake.script({ reply: "inspection only" }); + yield* sendMessage(reopened, "continue explicitly after interruption"); + yield* awaiting(reopened.terminal, "explicit recovery response", (terminal) => + shows(terminal, "inspection only"), + ); + expect(host.fake.prompts[prompts]).toContain("what it did is not known"); + expect(bodies).toBe(1); + yield* writeTextFile(join(workspace, "components/pass.test.md"), passing()); + yield* exchange(reopened, ['']); + expect(yield* reports(reopened)).toHaveLength(2); + }); + + for (const kind of ["persistence", "files", "disclosure", "cleanup"]) { + it(`R6: ${kind} authority stops the suite without a completed report`, function* () { + const failure = + kind === "persistence" + ? new DurablePersistenceError("yield", new Error("controlled backing failure")) + : kind === "files" + ? new FilesProviderUnavailableError() + : kind === "disclosure" + ? new SecretScannerError("report disclosure") + : new Error("owning runner cleanup failed"); + let later = 0; + yield* registerComponents([ + { + name: "Later", + props: { type: "object" }, + origin: "test://later", + fn: function* () { + later++; + return ""; + }, + }, + ]); + yield* Execution.around({ + *execute([request], next) { + if (request.options.path.endsWith("a.test.md")) { + if (kind === "cleanup") { + yield* ensure(function* () { + throw failure; + }); + } else { + request.addCompletionFailure(() => failure); + } + } + yield* next(request); + }, + }); + let checked = false; + const [result] = yield* allSettled([ + scoped(function* () { + const { repl, workspace } = yield* mounted(); + yield* ensure(function* () { + expect(later).toBe(0); + expect(yield* reports(repl)).toEqual([]); + const read = yield* mountedReading(repl); + const failures = read.chat.filter( + (record) => record.kind === "control-outcome" && record.outcome.status === "err", + ); + expect(failures.length).toBeGreaterThan(0); + expect(JSON.stringify(failures)).toContain(failure.name); + checked = true; + }); + yield* writeTextFile(join(workspace, "components/a.test.md"), passing()); + yield* writeTextFile(join(workspace, "components/b.test.md"), "" + passing()); + repl.fake.script({ reply: '' }); + yield* sendMessage(repl, `fail ${kind}`); + if (kind === "cleanup") { + yield* awaiting(repl.terminal, "cleanup failure", (terminal) => + shows(terminal, "[send]"), + ); + } else { + yield* suspend(); + } + }), + ]); + expect(checked).toBe(true); + if (kind !== "cleanup") { + expect(result?.ok).toBe(false); + if (result !== undefined && !result.ok) { + expect(result.error).toBe(failure); + } + } + }); + } +}); diff --git a/specs/repl-spec.md b/specs/repl-spec.md index ea595fbd2..699214157 100644 --- a/specs/repl-spec.md +++ b/specs/repl-spec.md @@ -108,6 +108,90 @@ Read `` and `` for the component-writing, discovery, later submission and output-reading journey. Current `` reports the response's actual cwd. +## Testing session components + +After ordinary File entries create a component and its companion test, a +Sidekick response requests real tests explicitly: + +```xmd + + +``` + +`REPL.Test` is a self-closing write control. Its only required prop is `path`; +ordinary `as` capture is optional. A directory target such as `components` +discovers `**/*.test.md` recursively, deduplicates documents and executes them +in normalized relative-path code-point order. Both target forms preserve the +finalized REPL component selection: configured directories, session components, +then registered defaults. They do not promote the document or target directory +as the CLI directory test command does. + +Targets resolve against the paired session workspace, including inside a +response Dir. Empty, absolute and lexically escaping paths refuse before +execution. Literal `#` and `%` belong to the filename. A missing selected file +is an execution failure. Discovery failure before any document starts gives +control-refusal feedback with its diagnostic and no completed report. + +A completed request returns this detached JSON-compatible value, and designated +feedback retains the same value whether or not it was captured: + +```ts +{ + path: string, + status: "passed" | "failed" | "no-tests", + documents: [{ + path: string, + status: "passed" | "failed" | "no-tests", + tests: TestResult[], + error: null | { name: string, message: string }, + report: string + }] +} +``` + +The top-level path preserves the requested spelling; document paths are +normalized session-relative paths, including `components/`. Tests are actual +Testing results: status `pass` or `fail`, source location, optional name and +optional structured error with kind, full message and optional actual/expected +values. Output is drained independently and partial rendering survives document +failure. A successful execution has error null; otherwise the actual failure's +name and full message remain. No outcome is inferred from report text. + +A document passes only after successful execution with reached tests. An +otherwise valid zero-test execution is `no-tests` with its genuine testing-policy +diagnostic. Parse or runtime failure before the first Test is `failed`, including +when there are no results. An empty explicit Testing boundary is still +`no-tests`. A suite passes only when every document passes; an empty selection +or all no-tests documents is `no-tests`; every other combination is `failed`. +Ordinary document failure allows later documents to run. Cancellation, refusal, +persistence, Files infrastructure, disclosure and owning-cleanup failures +produce no completed report and retain their authoritative failure paths. + +Each document has a fresh execution scope and real useTesting session under +the clean command host, with its providers, immutable declarations and selected +TypeScript loader. Entry and response bindings or registrations are not copied. +Document cleanup joins before the next document starts. Nested Execution uses +the existing testing host. Testing grants no permission and adds no terminal +writer, pane or automatic chat summary. Only explicit Output publishes chat. + +The root admits testing only when no entry reservation or active test exists +and the session is ready. Its test slot is acquired without suspension before +discovery and held through suite cleanup. Person Run, Sidekick Submit and a +second test refuse during this interval, including cancellation cleanup. A test +also refuses while an entry is preparing, executing, paused or tearing down. +Refusal queues nothing, preserves the draft and appends no entry. + +The response owns testing. Stop cancels and joins its test work before releasing +the slot; Exit likewise joins it before shutdown completes. An already admitted +ordinary entry remains session-owned and survives Sidekick Stop. Completed +reports remain retained facts after source changes or removal. Reopening and +inspection execute no test, provider or component. An unmatched test intent is +unknown work; only a new explicit REPL.Test starts another run. + +Read Guide.Entries for the executable create, test, fail, fix, retest and reuse +sequence. File availability follows actual File outcomes, not entry completion +alone. + ## One live journey Run `xmd repl`. The screen shows an empty draft, a Sessions list that says it is diff --git a/specs/testing-spec.md b/specs/testing-spec.md index 23570bf4b..cb9f53b68 100644 --- a/specs/testing-spec.md +++ b/specs/testing-spec.md @@ -202,6 +202,14 @@ begins with `-`. Options after `--` are not inspected. `--journal` writes one trace for one document, so a directory target rejects it before any document runs. +### Tests requested from the REPL + +`REPL.Test` uses the same directory discovery convention and real per-document +useTesting sessions. Both REPL target forms retain the finalized configured-first +REPL profile; CLI directory promotion above remains specific to `xmd test`. +The [REPL testing contract](repl-spec.md#testing-session-components) owns its +structured report, admission, feedback, cancellation and retained-history rules. + ## Atomic Tests Atomic tests use ``: