Inspect benchmark workspaces incrementally and search generated specs

Author: icecrasher321

apps/sim/app/o/[organizationId]/benchmark/components/benchmark-detail.tsx

@@ -143,7 +143,7 @@ function BenchmarkEditor({
       <BenchmarkStep
         number={3}
         title='Fill in the blanks'
-        description='A fresh reader receives only the generated plan and redacted reference, with no tools or memory.'
+        description='A fresh reader fills the blanks by reading or searching the generated plan. It has no enterprise access or prior memory.'
         pending={stage === 'reconstruct'}
         action={
           <Chip

apps/sim/app/o/[organizationId]/benchmark/components/benchmark-reference.tsx

@@ -41,11 +41,11 @@ export function BenchmarkReference({
     if (!file) return
     try {
       if (file.size > BENCHMARK_SPEC_MAX_LENGTH * 4) {
-        throw new Error('The reference is too large. Use a text file under 400 KB.')
+        throw new Error('The reference is too large. Use a text file under 4 MB.')
       }
       const text = await file.text()
       if (text.length > BENCHMARK_SPEC_MAX_LENGTH) {
-        throw new Error('The reference must be at most 100,000 characters.')
+        throw new Error('The reference must be at most 1,000,000 characters.')
       }
       onChange({ referenceSpec: text })
     } catch (error) {

apps/sim/lib/benchmarks/README.md

@@ -10,10 +10,10 @@ Open **Settings → Admin → Open benchmarks** (or `/benchmark`). Select an org
 
 Each case fixes the execution user when it is created. Changing the selector shows that user's saved cases; it never changes an existing case's identity. Legacy cases retain their original owner's execution identity. **New benchmark** always opens a creation form, and saved cases appear in a list with an **Open** action.
 
-1. Select a workspace. Generate the reference from its active workflow exports, or paste/import a reference produced by another client. Review the original task brief and reference.
+1. Select a workspace. Generate the reference by letting Mothership inspect its workflows and supporting resources, or paste/import a reference produced by another client. Review the original task brief and reference.
 2. Generate the blanks and review their expected answers. Only select facts discoverable from the permitted enterprise sources. Replacing `[[BLANK:id]]` with each exact answer must restore the reference byte for byte. Remove answer hints left elsewhere in the reference.
 3. Run the planner. It receives the brief in a fresh organization Plan conversation, with enterprise search/document reads and isolated memory. It receives neither the completed workspace inventory nor the reference, masks, or expected answers. Its final response is the generated spec.
-4. Reconstruct and grade separately. Reconstruction is a fresh tool-free execution with only the generated spec and redacted reference. Grading compares the recovered answers to the hidden targets and checks support in the generated spec. An exact supporting passage is required for credit. The score is correct details divided by the fixed number of blanks.
+4. Reconstruct and grade separately. Reconstruction is a fresh execution with the redacted questions in context and one immutable document, `generated-spec.md`, available through a bounded read/search tool. That document contains only the planner’s generated spec; enterprise tools, other documents and history are unavailable. Grading compares the recovered answers to the hidden targets and checks support in the generated spec. An exact supporting passage is required for credit. The score is correct details divided by the fixed number of blanks.
 
 The UI groups these actions into Reference, Plan, Reconstruction, and Grade. Each output is saved independently. Editing upstream inputs clears dependent current outputs. Every successful Grade automatically saves an immutable result in **Run history**, including the exact inputs, generated spec, answers, and assessments. Add an optional run label before grading to identify agent versions or experiments. History is retained when you edit the case or rerun a stage; deleting the benchmark deletes its history.
 
@@ -25,8 +25,8 @@ Reference specs can be prose or JSON text; **Import spec** accepts Markdown, tex
 
 Every stage is limited to ten minutes, with a twelve-minute persistence lease. Concurrent starts or stale completions cannot overwrite newer results. Interrupted steps are retryable; a hard server failure becomes retryable when its lease expires. This first version runs a stage within its HTTP request, so the deployment's request timeout must accommodate the run. Page reload may interrupt a pending request; saved completed stages remain available.
 
-Automatic distillation supports at most 50 workflows and 500 KB of sanitized exports. Credentials/passwords are removed by the canonical workflow exporter. Larger projects can use an imported reference. Stateless generation has a 16,384-token output cap. The planner uses the dev worker's model configuration; keep worker/model configuration constant when comparing agent changes.
+Reference generation uses the existing Mothership agent loop and workspace CLI to inspect active workflows and supporting resources incrementally. There is no benchmark-wide workflow-count or export-size gate. Reads remain authorized as the selected user; the benchmark transport refuses source mutations, workflow execution, services and scratch writes. Existing CLI pagination and searchable stored outputs keep individual model inputs bounded. Specs can contain up to 1,000,000 characters; structured output uses the worker’s 32,768-token per-response limit. The planner uses the dev worker's model configuration; keep worker/model configuration constant when comparing agent changes.
 
 This version uses **live enterprise context**, not a frozen historical snapshot. The completed Sim workspace is excluded from planning, but historical solution documents in connected enterprise sources can still reveal answers. Treat these cases as retrospective evaluations and review source availability. The score measures recovery of the selected requirements, not execution correctness or every claim in the plan.
 
-Validation includes real PostgreSQL ownership/organization/access-revocation checks and concurrent stage/lease fencing in `repository.integration.ts`; redaction integrity and dependency invalidation in `artifacts.test.ts`; reconstruction evidence and denominator checks in `evaluation.test.ts`; and executable tool isolation in the Mothership tool-executor tests. The worker tests cover fresh conversations, memory isolation, restricted tools, restart continuation, and tool-free execution.
+Validation includes real PostgreSQL ownership/organization/access-revocation checks and concurrent stage/lease fencing in `repository.integration.ts`; redaction integrity and dependency invalidation in `artifacts.test.ts`; reconstruction evidence and denominator checks in `evaluation.test.ts`; and executable tool isolation in the Mothership tool-executor tests. The worker tests cover fresh conversations, memory isolation, restricted tools, restart continuation, and isolated spec retrieval and tool-free grading.

apps/sim/lib/benchmarks/application/operations.ts

@@ -2,6 +2,12 @@ import { defineOperation } from '@/lib/core/application/operation'
 import { defineWorkspaceOperation } from '@/lib/core/application/workspace-operation'
 
 export const benchmarkOperations = {
+  /** permission-group-exempt: platform superusers administer benchmarks; source access is checked for the selected user. */
+  prepareReference: defineOperation({
+    id: 'benchmarks.reference.prepare',
+    principalKinds: ['session'],
+    capability: 'none',
+  }),
   /** permission-group-exempt: platform superusers administer benchmarks; selected-user capabilities are checked separately. */
   preparePlan: defineOperation({
     id: 'benchmarks.plan.prepare',

apps/sim/lib/benchmarks/application/prepare-reference.ts

@@ -0,0 +1,38 @@
+import type { Principal } from '@sim/auth/principal'
+import { db } from '@sim/db'
+import { copilotChats } from '@sim/db/schema'
+import { defineAuthorizedBenchmarkUseCase } from '@/lib/benchmarks/application/access'
+import { requireBenchmarkCaseAccess } from '@/lib/benchmarks/application/cases'
+import { benchmarkOperations } from '@/lib/benchmarks/application/operations'
+import { MOTHERSHIP_CHAT_DEFAULT_MODEL } from '@/lib/mothership/constants'
+
+/** Source reads use an owned workspace chat, separate from the planner's enterprise discovery context. */
+export const prepareBenchmarkReference = defineAuthorizedBenchmarkUseCase({
+  operation: benchmarkOperations.prepareReference,
+  async execute({
+    principal,
+    input,
+  }: {
+    principal: Principal
+    input: { organizationId: string; benchmarkId: string }
+  }) {
+    const benchmark = await requireBenchmarkCaseAccess(principal, input)
+    const userId = benchmark.runAsUserId ?? benchmark.userId
+    const [chat] = await db
+      .insert(copilotChats)
+      .values({
+        userId,
+        workspaceId: benchmark.sourceWorkspaceId,
+        type: 'mothership',
+        model: MOTHERSHIP_CHAT_DEFAULT_MODEL,
+        config: {
+          conversationMode: 'agent',
+          benchmark: { id: benchmark.id, operatorUserId: benchmark.userId },
+        },
+        lastSeenAt: new Date(),
+      })
+      .returning({ id: copilotChats.id })
+    if (!chat) throw new Error('Failed to create benchmark reference conversation')
+    return { chatId: chat.id, userId }
+  },
+})

apps/sim/lib/benchmarks/application/run-stage.ts

@@ -2,13 +2,10 @@ import type { Principal } from '@sim/auth/principal'
 import { createLogger } from '@sim/logger'
 import { generateId } from '@sim/utils/id'
 import { z } from 'zod'
-import {
-  benchmarkSourcePrincipal,
-  defineAuthorizedBenchmarkUseCase,
-} from '@/lib/benchmarks/application/access'
+import { defineAuthorizedBenchmarkUseCase } from '@/lib/benchmarks/application/access'
 import { requireBenchmarkCaseAccess } from '@/lib/benchmarks/application/cases'
 import { benchmarkOperations } from '@/lib/benchmarks/application/operations'
-import { readBenchmarkWorkspace } from '@/lib/benchmarks/application/source'
+import { prepareBenchmarkReference } from '@/lib/benchmarks/application/prepare-reference'
 import { applyBenchmarkPatch, validateBenchmarkRedaction } from '@/lib/benchmarks/artifacts'
 import { getBenchmarkMothershipUrl } from '@/lib/benchmarks/config'
 import { gradeReconstruction, validateReconstruction } from '@/lib/benchmarks/evaluation'
@@ -92,18 +89,18 @@ async function performStage(
   const artifacts = benchmark.artifacts
   switch (stage) {
     case 'distill': {
-      const sourcePrincipal = await benchmarkSourcePrincipal(principal, {
-        organizationId: benchmark.organizationId,
-        runAsUserId: benchmark.runAsUserId ?? benchmark.userId,
-        sourceWorkspaceId: benchmark.sourceWorkspaceId,
+      const target = await prepareBenchmarkReference.execute({
+        principal,
+        input: { organizationId: benchmark.organizationId, benchmarkId: benchmark.id },
       })
-      const source = await readBenchmarkWorkspace(sourcePrincipal, benchmark.sourceWorkspaceId)
       signal.throwIfAborted()
       const result = await executeBenchmarkJson({
         benchmark,
         signal,
         schema: distillationSchema,
-        messages: distillationMessages(source, artifacts.taskBrief),
+        messages: distillationMessages(artifacts.taskBrief),
+        profile: { stage: 'distill' },
+        chatId: target.chatId,
       })
       return { artifacts: applyBenchmarkPatch(artifacts, result), plannerChatId: null }
     }
@@ -133,7 +130,8 @@ async function performStage(
         benchmark,
         signal,
         schema: reconstructionSchema,
-        messages: reconstructionMessages(artifacts.generatedSpec!, artifacts.redactedSpec),
+        messages: reconstructionMessages(artifacts.redactedSpec),
+        profile: { stage: 'reconstruct', spec: artifacts.generatedSpec! },
       })
       validateReconstruction(artifacts.blanks, result.answers)
       return { artifacts: { ...artifacts, reconstruction: result.answers, grade: null } }

apps/sim/lib/benchmarks/application/source.ts

@@ -11,10 +11,10 @@ function messages(instruction: string, data: unknown): ExecuteMessage[] {
   ]
 }
 
-export function distillationMessages(workflows: string, taskBrief: string): ExecuteMessage[] {
+export function distillationMessages(taskBrief: string): ExecuteMessage[] {
   return messages(
-    'Describe the implemented business behavior of these workflow exports as a self-contained reference specification. Cover triggers, conditions, ownership, mappings, actions, destinations and failure behavior. Do not invent intent, requirements or missing values. Omit layout, internal block IDs, credential IDs and implementation trivia. Also draft a short taskBrief expressing the business goal a user would originally request, without disclosing the enterprise-specific answers. If a taskBrief was supplied, preserve it. The reference is for human review before evaluation.',
-    { workflows, taskBrief }
+    'Explore the selected workspace using the read-only workspace CLI and describe its implemented behavior as a detailed, self-contained reference specification. Begin by listing all active workflows, follow list pagination, inspect each workflow and its referenced workspace resources, and follow large-output continuations. Cover exact triggers and input shapes, conditions, ownership, field mappings, prompts and code behavior, actions, destinations, cross-workflow relationships, outputs and failure/recovery behavior. Preserve concrete names, values and business rules; do not compress them into a high-level overview. Distinguish implemented behavior from unresolved configuration or inferred intent. Do not invent missing values. Omit editor layout and credential values. Also draft a short taskBrief expressing the business goal a user would originally request, without disclosing the enterprise-specific answers. If a taskBrief was supplied, preserve it. The reference is for human review before evaluation. Use tools to inspect; return the final JSON only when inspection is complete.',
+    { taskBrief }
   )
 }
 
@@ -26,13 +26,10 @@ export function redactionMessages(referenceSpec: string): ExecuteMessage[] {
 }
 
 /** This projection is the reader's complete input; reference answers and enterprise history have no path into it. */
-export function reconstructionMessages(
-  generatedSpec: string,
-  redactedSpec: string
-): ExecuteMessage[] {
+export function reconstructionMessages(redactedSpec: string): ExecuteMessage[] {
   return messages(
-    'Fill every [[BLANK:id]] in redactedSpec using generatedSpec as your only evidence. For each distinct id return answer and support, where support is an exact contiguous quote from generatedSpec that establishes the answer. Use an empty answer and empty support when the generated spec does not establish it or contradicts itself. Do not use background knowledge, surviving reference text, or guesses to supply missing facts.',
-    { generatedSpec, redactedSpec }
+    'Fill every [[BLANK:id]] in redactedSpec using generated-spec.md as your only evidence. Read or search that file with read_spec. For each distinct id return answer and support, where support is an exact contiguous quote from the file that establishes the answer. Use an empty answer and empty support when the generated spec does not establish it or contradicts itself. Do not use background knowledge, surviving reference text, or guesses to supply missing facts.',
+    { redactedSpec }
   )
 }
 

apps/sim/lib/benchmarks/prompts.ts

@@ -22,6 +22,7 @@ vi.mock('@sim/db', () => ({
 
 import { createBenchmark, getBenchmark, listBenchmarks } from '@/lib/benchmarks/application/cases'
 import { prepareBenchmarkPlan } from '@/lib/benchmarks/application/prepare-plan'
+import { prepareBenchmarkReference } from '@/lib/benchmarks/application/prepare-reference'
 import {
   getBenchmarkRun,
   listBenchmarkRuns,
@@ -231,6 +232,21 @@ describe('private benchmark persistence and attempt fencing', () => {
       },
     })
     expect(benchmark).toMatchObject({ userId: 'owner', runAsUserId: 'target' })
+    const reference = await prepareBenchmarkReference.execute({
+      principal,
+      input: { organizationId: 'org', benchmarkId: benchmark.id },
+    })
+    const [referenceChat] =
+      await connection`SELECT user_id, workspace_id, organization_id, config FROM copilot_chats WHERE id = ${reference.chatId}`
+    expect(referenceChat).toMatchObject({
+      user_id: 'target',
+      workspace_id: 'workspace',
+      organization_id: null,
+      config: {
+        conversationMode: 'agent',
+        benchmark: { id: benchmark.id, operatorUserId: 'owner' },
+      },
+    })
     const target = await prepareBenchmarkPlan.execute({
       principal,
       input: { organizationId: 'org', benchmarkId: benchmark.id },
@@ -269,7 +285,13 @@ describe('private benchmark persistence and attempt fencing', () => {
         input: { organizationId: 'org', benchmarkId: benchmark.id },
       })
     ).rejects.toMatchObject({ code: 'forbidden' })
-    expect(await connection`SELECT id FROM copilot_chats`).toHaveLength(2)
+    await expect(
+      prepareBenchmarkReference.execute({
+        principal,
+        input: { organizationId: 'org', benchmarkId: benchmark.id },
+      })
+    ).rejects.toMatchObject({ code: 'forbidden' })
+    expect(await connection`SELECT id FROM copilot_chats`).toHaveLength(3)
   })
 
   it('refuses a target outside the organization or with revoked membership, disabled account, or workspace access', async () => {

apps/sim/lib/benchmarks/repository.integration.ts

@@ -1,6 +1,6 @@
 import { z } from 'zod'
 
-export const BENCHMARK_SPEC_MAX_LENGTH = 100_000
+export const BENCHMARK_SPEC_MAX_LENGTH = 1_000_000
 export const BENCHMARK_MAX_BLANKS = 50
 export const benchmarkStageSchema = z.enum(['distill', 'redact', 'plan', 'reconstruct', 'grade'])
 export const benchmarkNameSchema = z.string().trim().min(1, 'A benchmark name is required').max(200)