From 737e071784c3e4eb26a9c6d5abf8e13aaf73d13b Mon Sep 17 00:00:00 2001 From: "Jose I. Paris" Date: Wed, 21 Oct 2026 13:18:15 +0200 Subject: [PATCH 1/3] feat(trace): add the Cursor harness to the end-to-end suite Run the Cursor CLI (cursor-agent) headless in the Trace end-to-end suite, with HOME next to the workspace so the configuration and hooks of the user do not load. Pin the CLI version in the suite config. Refs #3635 Assisted-by: Claude Code Signed-off-by: Jose I. Paris Chainloop-Trace-Sessions: 0f698192-ddc8-487e-b214-474edc859d4e --- app/cli/internal/trace/PROVIDERS.md | 4 +- app/cli/internal/trace/README.md | 2 +- app/cli/internal/trace/e2e/claude.go | 4 +- app/cli/internal/trace/e2e/config.yaml | 7 + app/cli/internal/trace/e2e/cursor.go | 225 ++++++++++++++++++++++ app/cli/internal/trace/e2e/cursor_test.go | 128 ++++++++++++ app/cli/internal/trace/e2e/e2e_test.go | 1 + app/cli/internal/trace/e2e/harness.go | 3 + 8 files changed, 369 insertions(+), 5 deletions(-) create mode 100644 app/cli/internal/trace/e2e/cursor.go create mode 100644 app/cli/internal/trace/e2e/cursor_test.go diff --git a/app/cli/internal/trace/PROVIDERS.md b/app/cli/internal/trace/PROVIDERS.md index 6d00a4e3c..34a5f4425 100644 --- a/app/cli/internal/trace/PROVIDERS.md +++ b/app/cli/internal/trace/PROVIDERS.md @@ -220,7 +220,7 @@ Each provider must have an end-to-end harness. The suite in `e2e/` runs the agen A cell that the suite did not check stays at **Unknown**, or you add a check for it. -1. Add `e2e/myagent.go` with a type that implements `e2e.Harness` (`e2e/harness.go`): `Column`, `TraceFlag`, `Version`, `Run`, `HookConfig`, `DisableSessionStart`, and `SkillPath`. Use `e2e/claude.go`, `e2e/opencode.go`, and `e2e/pi.go` as examples. Run the agent with its configuration and session folders next to the workspace, so the run does not touch the configuration of the user. +1. Add `e2e/myagent.go` with a type that implements `e2e.Harness` (`e2e/harness.go`): `Column`, `TraceFlag`, `Version`, `Run`, `HookConfig`, `DisableSessionStart`, and `SkillPath`. Use `e2e/claude.go`, `e2e/cursor.go`, `e2e/opencode.go`, and `e2e/pi.go` as examples. Run the agent with its configuration and session folders next to the workspace, so the run does not touch the configuration of the user. 2. Add the harness to the `harnesses` map in `e2e/e2e_test.go`. The key is the column name in the README. 3. Add the harness to `e2e/config.yaml`, with the pinned version and the models. Write the API key that the models need in a comment. 4. If the agent shows hook messages in its headless output, set `RunOutput.Hooks` and `HooksReported`. Otherwise the hook message checks are "not covered". @@ -230,7 +230,7 @@ A cell that the suite did not check stays at **Unknown**, or you add a check for CHAINLOOP_E2E_HARNESS="My Agent" make -C app/cli test-trace-e2e ``` -Cursor has no end-to-end harness yet, so its **Unknown** cells are not verified. Do not add a provider with this gap. +Do not add a provider with no end-to-end harness: its **Unknown** cells stay not verified. ### Compatibility matrix (required) diff --git a/app/cli/internal/trace/README.md b/app/cli/internal/trace/README.md index d75cca488..91da1f70c 100644 --- a/app/cli/internal/trace/README.md +++ b/app/cli/internal/trace/README.md @@ -164,7 +164,7 @@ All agents share the git hooks `commit-msg`, `post-commit` and `pre-push`. ## End-to-end tests -The `e2e/` package checks the tables of this page against real agents (spec issue-3635). It runs each agent in headless mode with real models, in throwaway repositories with Trace in local storage mode. For now, it runs the Claude Code, OpenCode 2, Pi, and Codex columns. The Pi and Codex cells are **No**, so the suite skips them until Pi and Codex support is added. OpenCode runs on a private server (`opencode run --standalone`) with `PWD` set to the workspace, because OpenCode takes its project folder from `PWD`. Pi runs with `pi -p --mode json --approve`, with its configuration and session folders next to the workspace. Codex runs with `codex exec --json`, with its `CODEX_HOME` folder next to the workspace, no sandbox, and no hook trust prompt. The OpenCode 2, Pi, and Codex models in `e2e/config.yaml` are OpenRouter models, so the run needs `OPENROUTER_API_KEY`. A Codex model with no `openrouter/` prefix runs on the default Codex provider, which needs `CODEX_API_KEY`. +The `e2e/` package checks the tables of this page against real agents (spec issue-3635). It runs each agent in headless mode with real models, in throwaway repositories with Trace in local storage mode. For now, it runs the Claude Code, Cursor, OpenCode 2, Pi, and Codex columns. The Pi and Codex cells are **No**, so the suite skips them until Pi and Codex support is added. OpenCode runs on a private server (`opencode run --standalone`) with `PWD` set to the workspace, because OpenCode takes its project folder from `PWD`. Pi runs with `pi -p --mode json --approve`, with its configuration and session folders next to the workspace. Codex runs with `codex exec --json`, with its `CODEX_HOME` folder next to the workspace, no sandbox, and no hook trust prompt. Cursor runs with `cursor-agent -p --output-format stream-json --force --trust`, with its `HOME` folder next to the workspace, so the Cursor configuration and hooks of the user do not load. The Cursor CLI then has no stored login, so the run needs `CURSOR_API_KEY`. The headless Cursor CLI cannot attach a pasted image, and its output does not show the hook messages. The OpenCode 2, Pi, and Codex models in `e2e/config.yaml` are OpenRouter models, so the run needs `OPENROUTER_API_KEY`. A Codex model with no `openrouter/` prefix runs on the default Codex provider, which needs `CODEX_API_KEY`. A feature row maps to its checks by the feature name. When you rename a feature in a table, rename it in `e2e/checks.go` too. The suite tests a cell marked **Yes**, **Partial**, or **Unknown** only when the row has a registered check. It reports the other cells as follows: diff --git a/app/cli/internal/trace/e2e/claude.go b/app/cli/internal/trace/e2e/claude.go index 1b3463518..8cb01b0d9 100644 --- a/app/cli/internal/trace/e2e/claude.go +++ b/app/cli/internal/trace/e2e/claude.go @@ -75,7 +75,7 @@ func (c *Claude) Run(ctx context.Context, req RunRequest) (*RunOutput, error) { args := []string{ "-p", req.Prompt, flagModel, req.Model, - "--output-format", "stream-json", + "--output-format", outputStreamJSON, "--verbose", "--include-hook-events", "--setting-sources", "project,local", @@ -92,7 +92,7 @@ func (c *Claude) Run(ctx context.Context, req RunRequest) (*RunOutput, error) { } stdin = msg // The prompt is in the message, so -p takes no argument. - args = append([]string{"-p", "--input-format", "stream-json"}, args[2:]...) + args = append([]string{"-p", "--input-format", outputStreamJSON}, args[2:]...) } name := c.bin() diff --git a/app/cli/internal/trace/e2e/config.yaml b/app/cli/internal/trace/e2e/config.yaml index 86171d5ca..2340df58f 100644 --- a/app/cli/internal/trace/e2e/config.yaml +++ b/app/cli/internal/trace/e2e/config.yaml @@ -9,6 +9,13 @@ harnesses: version: "2.1.301" models: - haiku + # The Cursor CLI (cursor-agent) runs with a HOME folder next to the + # workspace, so it has no stored login: it needs CURSOR_API_KEY. The version + # is a build tag, not a semantic version. + Cursor: + version: "2026.10.19-6bbf2c4" + models: + - auto # The OpenRouter models need OPENROUTER_API_KEY in the environment. OpenCode 2: version: "2.0.22" diff --git a/app/cli/internal/trace/e2e/cursor.go b/app/cli/internal/trace/e2e/cursor.go new file mode 100644 index 000000000..54a5454aa --- /dev/null +++ b/app/cli/internal/trace/e2e/cursor.go @@ -0,0 +1,225 @@ +// +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "slices" + "strings" +) + +// cursorHooksPath is the project file where `trace init` installs the Cursor +// hooks. +const cursorHooksPath = ".cursor/hooks.json" + +// cursorSessionStartEvent is the hook event of the session start in +// cursorHooksPath. +const cursorSessionStartEvent = "sessionStart" + +// errCursorImages is the error of a run with pasted images. The headless +// Cursor CLI has no input for an image. +var errCursorImages = errors.New("cursor-agent: the headless CLI cannot attach a pasted image") + +// Cursor is the harness adapter of the Cursor CLI (cursor-agent). +type Cursor struct { + // Bin is the cursor-agent executable. Empty means "cursor-agent" from + // PATH. + Bin string +} + +func (c *Cursor) bin() string { + if c.Bin != "" { + return c.Bin + } + return "cursor-agent" +} + +// Column implements Harness. +func (c *Cursor) Column() string { return "Cursor" } + +// TraceFlag implements Harness. +func (c *Cursor) TraceFlag() string { return "--cursor" } + +// Version implements Harness. +func (c *Cursor) Version(ctx context.Context) (string, error) { + out, err := exec.CommandContext(ctx, c.bin(), "--version").Output() //nolint:gosec // the harness binary under test // nosemgrep + if err != nil { + return "", fmt.Errorf("cursor-agent --version: %w", err) + } + return parseCursorVersion(out) +} + +// parseCursorVersion reads the output of `cursor-agent --version`, for example +// "2026.01.09-6d1b8f1". The version is a build tag, not a semantic version. +func parseCursorVersion(out []byte) (string, error) { + fields := strings.Fields(string(out)) + if len(fields) == 0 { + return "", errors.New("cursor-agent --version: empty output") + } + return fields[len(fields)-1], nil +} + +// Run implements Harness. It runs `cursor-agent -p` with the stream-json +// output. +// +// HOME is a folder next to the workspace, so the configuration, the user +// hooks (~/.cursor/hooks.json) and the transcripts of the user who runs the +// suite do not load or mix with the sessions of the suite. The Trace hooks +// run in the same environment, so the provider reads the transcripts from +// that folder too. The CLI then has no stored login: it reads its key from +// CURSOR_API_KEY. --force approves the tool calls and --trust trusts the +// workspace, which a headless run cannot answer. +// +// Cursor gives the session-start context to the model only, and the +// stream-json output does not show it, so the hook messages are not reported. +func (c *Cursor) Run(ctx context.Context, req RunRequest) (*RunOutput, error) { + if len(req.Images) > 0 { + return nil, errCursorImages + } + + home := filepath.Join(filepath.Dir(req.Dir), "cursor-home") + if err := os.MkdirAll(home, 0o700); err != nil { + return nil, err + } + + args := []string{ + "-p", "--output-format", outputStreamJSON, + "--force", "--trust", + "--workspace", req.Dir, + flagModel, req.Model, + req.Prompt, + } + + name := c.bin() + if len(req.Wrapper) > 0 { + name = req.Wrapper[0] + args = append(append(append([]string{}, req.Wrapper[1:]...), c.bin()), args...) + } + + cmd := exec.CommandContext(ctx, name, args...) //nolint:gosec // the harness binary under test, with the prompt of a scenario // nosemgrep + cmd.Dir = req.Dir + cmd.Env = append(slices.Clone(req.Env), "HOME="+home) + var stderr bytes.Buffer + cmd.Stderr = &stderr + + raw, err := cmd.Output() + if err != nil { + return &RunOutput{Raw: raw}, fmt.Errorf("cursor-agent -p: %w: %s", err, strings.TrimSpace(stderr.String())) + } + if err := cursorError(raw); err != nil { + return &RunOutput{Raw: raw}, err + } + + return &RunOutput{Raw: raw}, nil +} + +// cursorEvent is the part of a stream-json line of `cursor-agent -p` that the +// suite reads. +type cursorEvent struct { + Type string `json:"type"` + Subtype string `json:"subtype"` + IsError bool `json:"is_error"` + Result string `json:"result"` +} + +// cursorError returns the error of the result event of a run, which can come +// with exit status 0. +func cursorError(raw []byte) error { + scanner := bufio.NewScanner(bytes.NewReader(raw)) + scanner.Buffer(make([]byte, 0, 1024*1024), 64*1024*1024) + for scanner.Scan() { + line := bytes.TrimSpace(scanner.Bytes()) + if len(line) == 0 { + continue + } + var ev cursorEvent + if err := json.Unmarshal(line, &ev); err != nil { + return fmt.Errorf("decode cursor-agent event: %w", err) + } + if ev.Type == "result" && ev.IsError { + return fmt.Errorf("cursor-agent -p: %s: %s", ev.Subtype, ev.Result) + } + } + if err := scanner.Err(); err != nil { + return fmt.Errorf("read cursor-agent output: %w", err) + } + return nil +} + +// HookConfig implements Harness. The hooks of the user and the Chainloop +// hooks share one file. +func (c *Cursor) HookConfig() HookConfig { + return HookConfig{ + Path: cursorHooksPath, + ChainloopMarker: "chainloop trace hook cursor", + UserConfig: `{ + "version": 1, + "hooks": { + "stop": [ + {"command": "echo e2e-user-hook"} + ] + } +} +`, + UserMarker: userHookMarker, + } +} + +// DisableSessionStart implements Harness. It deletes the sessionStart entry +// from the hooks of the project file and keeps the other entries as they +// are. +func (c *Cursor) DisableSessionStart(repo string) error { + path := filepath.Join(repo, cursorHooksPath) + data, err := os.ReadFile(path) + if err != nil { + return err + } + + var settings map[string]json.RawMessage + if err := json.Unmarshal(data, &settings); err != nil { + return fmt.Errorf("decode %s: %w", cursorHooksPath, err) + } + var hooks map[string]json.RawMessage + if err := json.Unmarshal(settings["hooks"], &hooks); err != nil { + return fmt.Errorf("decode the hooks of %s: %w", cursorHooksPath, err) + } + if _, ok := hooks[cursorSessionStartEvent]; !ok { + return fmt.Errorf("%s has no %s hook", cursorHooksPath, cursorSessionStartEvent) + } + delete(hooks, cursorSessionStartEvent) + + if settings["hooks"], err = json.Marshal(hooks); err != nil { + return err + } + if data, err = json.MarshalIndent(settings, "", " "); err != nil { + return err + } + return os.WriteFile(path, data, 0o600) +} + +// SkillPath implements Harness. Cursor finds project skills in +// .cursor/skills. +func (c *Cursor) SkillPath(name string) string { + return ".cursor/skills/" + name + "/SKILL.md" +} diff --git a/app/cli/internal/trace/e2e/cursor_test.go b/app/cli/internal/trace/e2e/cursor_test.go new file mode 100644 index 000000000..e27308910 --- /dev/null +++ b/app/cli/internal/trace/e2e/cursor_test.go @@ -0,0 +1,128 @@ +// +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/cursor" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestParseCursorVersion(t *testing.T) { + cases := []struct { + name string + out string + want string + wantErr bool + }{ + {name: "build tag", out: "2026.01.09-6d1b8f1\n", want: "2026.01.09-6d1b8f1"}, + {name: "empty", out: "\n", wantErr: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, err := parseCursorVersion([]byte(tc.out)) + if tc.wantErr { + require.Error(t, err) + return + } + require.NoError(t, err) + assert.Equal(t, tc.want, got) + }) + } +} + +func TestCursorError(t *testing.T) { + cases := []struct { + name string + raw string + wantErr string + }{ + { + name: "success", + raw: `{"type":"system","subtype":"init","session_id":"s1"} +{"type":"result","subtype":"success","is_error":false,"result":"done"} +`, + }, + { + name: "error result", + raw: `{"type":"result","subtype":"error","is_error":true,"result":"invalid API key"}` + "\n", + wantErr: "invalid API key", + }, + { + name: "line that is not JSON", + raw: "not json\n", + wantErr: "decode cursor-agent event", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + err := cursorError([]byte(tc.raw)) + if tc.wantErr == "" { + require.NoError(t, err) + return + } + assert.ErrorContains(t, err, tc.wantErr) + }) + } +} + +func TestCursorRunWithImages(t *testing.T) { + c := &Cursor{} + _, err := c.Run(t.Context(), RunRequest{Dir: t.TempDir(), Images: [][]byte{{1}}}) + assert.ErrorIs(t, err, errCursorImages) +} + +func TestCursorDisableSessionStart(t *testing.T) { + c := &Cursor{} + + t.Run("removes only the session start hook", func(t *testing.T) { + repo := t.TempDir() + hc := c.HookConfig() + path := filepath.Join(repo, hc.Path) + require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750)) + require.NoError(t, os.WriteFile(path, []byte(hc.UserConfig), 0o600)) + require.NoError(t, cursor.New().InstallHooks(repo)) + + require.NoError(t, c.DisableSessionStart(repo)) + + data, err := os.ReadFile(path) + require.NoError(t, err) + var settings struct { + Version int `json:"version"` + Hooks map[string]json.RawMessage `json:"hooks"` + } + require.NoError(t, json.Unmarshal(data, &settings)) + assert.Equal(t, 1, settings.Version) + assert.NotContains(t, settings.Hooks, cursorSessionStartEvent) + assert.Contains(t, settings.Hooks, "sessionEnd") + assert.Contains(t, settings.Hooks, "stop") + assert.Contains(t, string(data), hc.UserMarker) + }) + + t.Run("no session start hook", func(t *testing.T) { + repo := t.TempDir() + path := filepath.Join(repo, cursorHooksPath) + require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750)) + require.NoError(t, os.WriteFile(path, []byte(c.HookConfig().UserConfig), 0o600)) + + assert.Error(t, c.DisableSessionStart(repo)) + }) +} diff --git a/app/cli/internal/trace/e2e/e2e_test.go b/app/cli/internal/trace/e2e/e2e_test.go index 7d57caa48..8cd975473 100644 --- a/app/cli/internal/trace/e2e/e2e_test.go +++ b/app/cli/internal/trace/e2e/e2e_test.go @@ -129,6 +129,7 @@ func TestTraceMatrix(t *testing.T) { // ones that config.yaml lists. var harnesses = map[string]Harness{ HarnessClaudeCode: &Claude{}, + "Cursor": &Cursor{}, "OpenCode 2": &OpenCode{Major: "2"}, "Pi": &Pi{}, "Codex": &Codex{}, diff --git a/app/cli/internal/trace/e2e/harness.go b/app/cli/internal/trace/e2e/harness.go index aa33a676c..c65b9d6da 100644 --- a/app/cli/internal/trace/e2e/harness.go +++ b/app/cli/internal/trace/e2e/harness.go @@ -23,6 +23,9 @@ const HarnessClaudeCode = "Claude Code" const ( // flagModel is the model flag that all the harnesses share. flagModel = "--model" + // outputStreamJSON is the line-per-event output format of Claude Code and + // the Cursor CLI. + outputStreamJSON = "stream-json" // userHookMarker is in the hook of the user that a scenario seeds. userHookMarker = "e2e-user-hook" ) From e3403f59f5982eab6b49c65832102acc71bc2250 Mon Sep 17 00:00:00 2001 From: "Jose I. Paris" Date: Wed, 21 Oct 2026 13:39:12 +0200 Subject: [PATCH 2/3] fix(trace): run the CLI under test in the Cursor e2e shell Cursor runs its shell tool in a login shell, where the macOS path_helper puts /usr/local/bin first in PATH. The git hooks of a session then ran an installed chainloop, not the CLI under test. The harness now writes shell startup files in its HOME that set PATH back. In auto mode no output names the model, so the model detection check is not covered for an auto session. Set the Cursor cells of the feature tables from the suite results. Refs #3635 Assisted-by: Claude Code Signed-off-by: Jose I. Paris Chainloop-Trace-Sessions: 0f698192-ddc8-487e-b214-474edc859d4e --- app/cli/internal/trace/README.md | 21 ++++----- app/cli/internal/trace/e2e/checks.go | 9 ++++ app/cli/internal/trace/e2e/checks_test.go | 57 +++++++++++++++++++++++ app/cli/internal/trace/e2e/config.yaml | 2 + app/cli/internal/trace/e2e/cursor.go | 39 +++++++++++++++- app/cli/internal/trace/e2e/cursor_test.go | 32 +++++++++++++ app/cli/internal/trace/e2e/result_test.go | 2 +- 7 files changed, 149 insertions(+), 13 deletions(-) create mode 100644 app/cli/internal/trace/e2e/checks_test.go diff --git a/app/cli/internal/trace/README.md b/app/cli/internal/trace/README.md index 91da1f70c..299a4b9c8 100644 --- a/app/cli/internal/trace/README.md +++ b/app/cli/internal/trace/README.md @@ -18,13 +18,13 @@ How Trace is added to a repository and to an agent. | Feature | What it means | Claude Code | Cursor | OpenCode 1.x | OpenCode 2 | Pi | Codex | |---------|---------------|-------------|--------|--------------|------------|---|---| -| Hook installation and deletion | `chainloop trace init` installs the hooks of the selected agent. A second init changes nothing. `chainloop trace uninstall` deletes only the Chainloop hooks and keeps the other hooks of the agent. | Yes | Unknown | Unknown | Yes | No | No | -| Trace run wrapper | `chainloop trace run -- ` gives the agent temporary hooks for one session. When the command exits, the configuration of the agent is the same as before. | Yes | Unknown | Unknown | Yes | No | No | +| Hook installation and deletion | `chainloop trace init` installs the hooks of the selected agent. A second init changes nothing. `chainloop trace uninstall` deletes only the Chainloop hooks and keeps the other hooks of the agent. | Yes | Yes | Unknown | Yes | No | No | +| Trace run wrapper | `chainloop trace run -- ` gives the agent temporary hooks for one session. When the command exits, the configuration of the agent is the same as before. | Yes | Yes | Unknown | Yes | No | No | | More than one agent in one repository | Two agents initialized in the same repository record their sessions separately, and each commit names the sessions of the agent that made it. | Unknown | Unknown | Unknown | Unknown | No | No | Why not Yes: -- **Hook installation and deletion, and Trace run wrapper, Cursor and OpenCode:** not verified yet. The end-to-end suite finds the values (spec issue-3635). +- **Hook installation and deletion, and Trace run wrapper, OpenCode 1.x:** not verified yet. The end-to-end suite finds the values (spec issue-3635). - **More than one agent in one repository, all agents:** not verified yet. The end-to-end suite needs a second agent to test it (spec issue-3635). ## Session collection @@ -43,7 +43,7 @@ Why not Yes: - **Transcript, OpenCode:** OpenCode has no transcript file. The provider runs `opencode export` (1.x) or `opencode session export` (2) and rebuilds the transcript from the export. It keeps the text and the inputs of the tool calls. It drops the tool outputs, the reasoning and the file parts. OpenCode 2 exports also have no agent version and no session slug. - **Token usage and cost, Cursor:** the Cursor transcript does not hold token usage. Usage and cost are zero, and the evidence has a warning that tells so. (For OpenCode, the cost is the value that OpenCode reports. For Claude Code, Chainloop computes it from its price table.) -- **Model detection, Cursor:** the primary model comes from the session-start hook. The transcript does not tell the models, so the evidence does not list the other models. +- **Model detection, Cursor:** the primary model comes from the session-start hook. The transcript does not tell the models, so the evidence does not list the other models. In auto mode, the hook names the model `default`, not the model that ran. - **Subagents, Cursor:** the provider does not read subagent conversations. - **Subagents, OpenCode:** a subagent runs in a child session with its own export. It is not part of the evidence of its parent session. - **Missed session start, Cursor and OpenCode 1.x:** not verified yet. The end-to-end suite finds the values (spec issue-3635). @@ -70,7 +70,6 @@ Why not Yes: - **Session updates, all agents:** the update works only in the clone where the work happened, while the CLI keeps the trace state of the session (30 days), and only when the commits of the session are on the branch that the push sends. - **Renames and rewrites, all agents:** this works only in the clone where the work happened. Git finds a rename only when the file keeps at least half of its content. Two pushed commits can rename a file one after the other. A later change to the file under its middle name does not send the session again. For Cursor, the limits of line attribution apply too. - **Shell changes attribution, all agents:** the hook before a shell command records only a hash of each file, not its content. The file at the `HEAD` commit from before the command stands in for the content before the command. Uncommitted lines that an earlier agent edit recorded are not recorded again, but uncommitted human lines of a file that the command changes count as AI. -- **Shell changes attribution, Cursor:** the provider installs the `preToolUse` and `postToolUse` hooks for the `Shell` tool. These hooks are verified for the Cursor IDE only. It is not verified that the Cursor CLI sends them. ## User experience @@ -98,7 +97,7 @@ What the session was asked to build, and which skills it used. | Spec capture instruction | At session start, the agent is told to capture the specs of the session (tickets, documents, plans, images) in the spec folder. A plan is captured only when it comes from an external source or from the plan mode of the agent: an agreement in the chat is not a plan. A pasted image is not captured, also when the agent has a file path for it. The agent tells the user in one line when it captures a file. | Yes | Yes | Yes | Yes | No | No | | Spec capture reminder | At each user prompt, the agent is reminded to capture new or changed specs. | Yes | No | Yes | Yes | No | No | | Local spec sources read at push | A capture whose source is a local file holds only a header and a placeholder. Each push reads the file and records its current content, so the evidence follows the edits of the file. Only a regular text file of up to 1 MiB is read. Otherwise the push keeps the body that the agent wrote, or drops the capture with a warning when there is no body (spec issue-3561). | Yes | Yes | Yes | Yes | No | No | -| Spec source role, title, and description | Each captured spec source holds the role, the title, and the description that the agent wrote in its header (spec issue-3531). | Yes | Unknown | Unknown | Yes | No | No | +| Spec source role, title, and description | Each captured spec source holds the role, the title, and the description that the agent wrote in its header (spec issue-3531). | Yes | Yes | Unknown | Yes | No | No | | Skills tracking | The evidence lists the skills that the session used, with a copy of each skill as it ran. | Yes | No | Partial | Partial | No | No | | Spec source pointers | Exact copies of spec sources in the transcript (full file reads and file writes) are replaced with a pointer to the spec material, so the evidence does not hold the same content two or three times (spec issue-3556). This includes the copies of pasted images and spec sources in the `api-request-blob` records of a Claude Code transcript (issue 3654). | Yes | No | No | No | No | No | | Pasted image pointers | Each distinct image that the user pasted into the session, including images in prompts sent while the agent was busy, is stored once as an image spec entry, from the data in the transcript, with the title "Pasted image N" and no role. A repeated paste of the same image adds no entry. The inline image in the transcript is replaced with a pointer to that material. An image whose data cannot be decoded, or that the push could not store, stays inline, and the push continues. When the agent captured the same image, its entry is kept and no second entry is added (spec issue-3569). | Yes | No | No | No | No | No | @@ -106,7 +105,7 @@ What the session was asked to build, and which skills it used. Why not Yes: - **Spec capture reminder, Cursor:** Cursor has no hook at each user prompt. The agent gets the session-start instruction only. -- **Spec source role, title, and description, Cursor and OpenCode 1.x:** not verified yet. The end-to-end suite finds the values (spec issue-3635). +- **Spec source role, title, and description, OpenCode 1.x:** not verified yet. The end-to-end suite finds the values (spec issue-3635). - **Skills tracking, Cursor:** the provider does not track skills. The evidence has no skill entries, which means "not recorded", not "no skill used". - **Skills tracking, OpenCode:** only the skills that the model starts with the `skill` tool are counted. Skills that the user starts, and skills used in subagents, are not. A plugin installed by an earlier version does not send the skill folder for OpenCode 2: install the hooks again to update it. - **Spec source pointers, Cursor and OpenCode:** the provider has no finder for the copies in its transcript yet. For OpenCode, the rebuilt transcript holds no tool outputs, so only the write copies would apply. @@ -119,12 +118,12 @@ How the evidence is protected before it leaves the machine. | Feature | What it means | Claude Code | Cursor | OpenCode 1.x | OpenCode 2 | Pi | Codex | |---------|---------------|-------------|--------|--------------|------------|---|---| | Secret redaction | Secrets in the session evidence and in the spec materials are replaced with a placeholder before upload. | Yes | Yes | Yes | Yes | No | No | -| Media skipped by redaction | The secret redaction does not scan base64 media (images, PDFs) in the transcript, so it is faster and does not break the media. | Yes | Unknown | Not applicable | Not applicable | No | No | +| Media skipped by redaction | The secret redaction does not scan base64 media (images, PDFs) in the transcript, so it is faster and does not break the media. | Yes | Not applicable | Not applicable | Not applicable | No | No | | Allow markers in redaction | A secret on a line that carries an inline `gitleaks:allow` or `betterleaks:allow` marker, such as a test fixture the agent read, is not redacted. The marker covers only its own line. | Yes | Yes | Yes | Yes | No | No | Why not Yes: -- **Media skipped by redaction, Cursor:** the redaction skips base64 media blocks of the shape `{"type":"base64","data":...}`. The Cursor transcript is stored as it is, and it is not verified whether it holds media in that shape. +- **Media skipped by redaction, Cursor:** the Cursor transcript holds no tool outputs, so it holds no media that the agent read. - **Media skipped by redaction, OpenCode:** the rebuilt transcript holds no media. ## Git integration @@ -151,7 +150,7 @@ Run `chainloop trace init` one time for each repository. Git does not clone the Why not Yes: -- **Shell commands in another checkout, Cursor:** the shell hooks are verified for the Cursor IDE only. A relative path in the command resolves against the first workspace root, not against the folder where the shell runs. +- **Shell commands in another checkout, Cursor:** a relative path in the command resolves against the first workspace root, not against the folder where the shell runs. ## How each agent is hooked in | Agent | Hook installation | Hooks | @@ -164,7 +163,7 @@ All agents share the git hooks `commit-msg`, `post-commit` and `pre-push`. ## End-to-end tests -The `e2e/` package checks the tables of this page against real agents (spec issue-3635). It runs each agent in headless mode with real models, in throwaway repositories with Trace in local storage mode. For now, it runs the Claude Code, Cursor, OpenCode 2, Pi, and Codex columns. The Pi and Codex cells are **No**, so the suite skips them until Pi and Codex support is added. OpenCode runs on a private server (`opencode run --standalone`) with `PWD` set to the workspace, because OpenCode takes its project folder from `PWD`. Pi runs with `pi -p --mode json --approve`, with its configuration and session folders next to the workspace. Codex runs with `codex exec --json`, with its `CODEX_HOME` folder next to the workspace, no sandbox, and no hook trust prompt. Cursor runs with `cursor-agent -p --output-format stream-json --force --trust`, with its `HOME` folder next to the workspace, so the Cursor configuration and hooks of the user do not load. The Cursor CLI then has no stored login, so the run needs `CURSOR_API_KEY`. The headless Cursor CLI cannot attach a pasted image, and its output does not show the hook messages. The OpenCode 2, Pi, and Codex models in `e2e/config.yaml` are OpenRouter models, so the run needs `OPENROUTER_API_KEY`. A Codex model with no `openrouter/` prefix runs on the default Codex provider, which needs `CODEX_API_KEY`. +The `e2e/` package checks the tables of this page against real agents (spec issue-3635). It runs each agent in headless mode with real models, in throwaway repositories with Trace in local storage mode. For now, it runs the Claude Code, Cursor, OpenCode 2, Pi, and Codex columns. The Pi and Codex cells are **No**, so the suite skips them until Pi and Codex support is added. OpenCode runs on a private server (`opencode run --standalone`) with `PWD` set to the workspace, because OpenCode takes its project folder from `PWD`. Pi runs with `pi -p --mode json --approve`, with its configuration and session folders next to the workspace. Codex runs with `codex exec --json`, with its `CODEX_HOME` folder next to the workspace, no sandbox, and no hook trust prompt. Cursor runs with `cursor-agent -p --output-format stream-json --force --trust`, with its `HOME` folder next to the workspace, so the Cursor configuration and hooks of the user do not load. Cursor runs its shell tool in a login shell, and on macOS `path_helper` puts the system folders first in `PATH`. So the harness writes shell startup files in that `HOME` that set `PATH` back. Otherwise a `chainloop` in `/usr/local/bin` runs the git hooks instead of the CLI under test. The Cursor CLI then has no stored login, so the run needs `CURSOR_API_KEY`. The headless Cursor CLI cannot attach a pasted image, and its output does not show the hook messages. The OpenCode 2, Pi, and Codex models in `e2e/config.yaml` are OpenRouter models, so the run needs `OPENROUTER_API_KEY`. A Codex model with no `openrouter/` prefix runs on the default Codex provider, which needs `CODEX_API_KEY`. A feature row maps to its checks by the feature name. When you rename a feature in a table, rename it in `e2e/checks.go` too. The suite tests a cell marked **Yes**, **Partial**, or **Unknown** only when the row has a registered check. It reports the other cells as follows: diff --git a/app/cli/internal/trace/e2e/checks.go b/app/cli/internal/trace/e2e/checks.go index e6c5470c5..7a58bab50 100644 --- a/app/cli/internal/trace/e2e/checks.go +++ b/app/cli/internal/trace/e2e/checks.go @@ -160,11 +160,20 @@ func checkTokenUsage(s *Session) error { return nil } +// modelAuto is the model of a session where the harness picks the model, for +// example the Auto mode of Cursor. +const modelAuto = "auto" + func checkModel(s *Session) error { m := evidence(s).Model if m == nil || m.Primary == "" { return errors.New("the evidence has no primary model") } + // In auto mode the harness picks the model, and no output names it, so + // there is no model name to compare. + if strings.EqualFold(s.Model, modelAuto) { + return fmt.Errorf("%w: the session runs in auto mode, so its model has no name to compare with %q", errNotCovered, m.Primary) + } // A model can have a provider prefix, for example // amazon-bedrock/eu.anthropic.claude-haiku-5-5, that the evidence leaves // out, and a short name, for example haiku, that the evidence spells out. diff --git a/app/cli/internal/trace/e2e/checks_test.go b/app/cli/internal/trace/e2e/checks_test.go new file mode 100644 index 000000000..088ceddb8 --- /dev/null +++ b/app/cli/internal/trace/e2e/checks_test.go @@ -0,0 +1,57 @@ +// +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "errors" + "testing" + + "github.com/chainloop-dev/chainloop/pkg/attestation/crafter/materials/aicodingsession" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestCheckModel(t *testing.T) { + cases := []struct { + name string + model string + primary string + wantErr bool + wantNotCovered bool + }{ + {name: "short name", model: "haiku", primary: "claude-haiku-5-5"}, + {name: "provider prefix", model: "openrouter/anthropic/claude-haiku-5.5", primary: "claude-haiku-5.5"}, + {name: "other model", model: "haiku", primary: "gpt-5", wantErr: true}, + {name: "auto mode", model: modelAuto, primary: "default", wantErr: true, wantNotCovered: true}, + {name: "auto mode with no primary model", model: modelAuto, wantErr: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + s := &Session{Model: tc.model, Stored: &StoredSession{}} + if tc.primary != "" { + s.Stored.Evidence.Data.Model = &aicodingsession.Model{Primary: tc.primary} + } + + err := checkModel(s) + if !tc.wantErr { + require.NoError(t, err) + return + } + require.Error(t, err) + assert.Equal(t, tc.wantNotCovered, errors.Is(err, errNotCovered)) + }) + } +} diff --git a/app/cli/internal/trace/e2e/config.yaml b/app/cli/internal/trace/e2e/config.yaml index 2340df58f..829cb7154 100644 --- a/app/cli/internal/trace/e2e/config.yaml +++ b/app/cli/internal/trace/e2e/config.yaml @@ -12,6 +12,8 @@ harnesses: # The Cursor CLI (cursor-agent) runs with a HOME folder next to the # workspace, so it has no stored login: it needs CURSOR_API_KEY. The version # is a build tag, not a semantic version. + # A free Cursor plan runs only auto. In auto mode no output names the + # model, so the model detection check is not covered. Cursor: version: "2026.10.19-6bbf2c4" models: diff --git a/app/cli/internal/trace/e2e/cursor.go b/app/cli/internal/trace/e2e/cursor.go index 54a5454aa..b39c26590 100644 --- a/app/cli/internal/trace/e2e/cursor.go +++ b/app/cli/internal/trace/e2e/cursor.go @@ -99,7 +99,7 @@ func (c *Cursor) Run(ctx context.Context, req RunRequest) (*RunOutput, error) { } home := filepath.Join(filepath.Dir(req.Dir), "cursor-home") - if err := os.MkdirAll(home, 0o700); err != nil { + if err := writeCursorShellProfiles(home, req.Env); err != nil { return nil, err } @@ -134,6 +134,43 @@ func (c *Cursor) Run(ctx context.Context, req RunRequest) (*RunOutput, error) { return &RunOutput{Raw: raw}, nil } +// cursorShellProfiles are the startup files of the login shells that run the +// shell tool of Cursor, relative to HOME. +var cursorShellProfiles = []string{".zprofile", ".zshrc", ".bash_profile", ".bashrc", ".profile"} + +// writeCursorShellProfiles creates HOME and writes a startup file for each +// shell that sets PATH back to the PATH of env. +// +// Cursor runs its shell tool in a login shell. On macOS, /etc/zprofile runs +// path_helper, which puts the system folders first in PATH. A chainloop CLI +// in /usr/local/bin then hides the CLI under test, and the git hooks of the +// session run the wrong CLI. The startup files of HOME run after +// /etc/zprofile, so they set PATH back. +func writeCursorShellProfiles(home string, env []string) error { + if err := os.MkdirAll(home, 0o700); err != nil { + return err + } + + var path string + for _, kv := range env { + if v, ok := strings.CutPrefix(kv, "PATH="); ok { + path = v + } + } + if path == "" { + return nil + } + + // A single-quoted shell word, with each quote in the value escaped. + profile := "export PATH='" + strings.ReplaceAll(path, "'", `'\''`) + "'\n" + for _, name := range cursorShellProfiles { + if err := os.WriteFile(filepath.Join(home, name), []byte(profile), 0o600); err != nil { + return err + } + } + return nil +} + // cursorEvent is the part of a stream-json line of `cursor-agent -p` that the // suite reads. type cursorEvent struct { diff --git a/app/cli/internal/trace/e2e/cursor_test.go b/app/cli/internal/trace/e2e/cursor_test.go index e27308910..4c6b69e11 100644 --- a/app/cli/internal/trace/e2e/cursor_test.go +++ b/app/cli/internal/trace/e2e/cursor_test.go @@ -18,6 +18,7 @@ package e2e import ( "encoding/json" "os" + "os/exec" "path/filepath" "testing" @@ -84,6 +85,37 @@ func TestCursorError(t *testing.T) { } } +func TestWriteCursorShellProfiles(t *testing.T) { + cases := []struct { + name string + env []string + want string + }{ + {name: "plain path", env: []string{"PATH=/e2e/bin:/usr/bin"}, want: "/e2e/bin:/usr/bin"}, + {name: "last PATH wins", env: []string{"PATH=/old", "PATH=/e2e/bin"}, want: "/e2e/bin"}, + {name: "quote in path", env: []string{"PATH=/it's/bin:/usr/bin"}, want: "/it's/bin:/usr/bin"}, + {name: "no PATH"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + home := filepath.Join(t.TempDir(), "cursor-home") + require.NoError(t, writeCursorShellProfiles(home, tc.env)) + require.DirExists(t, home) + + for _, name := range cursorShellProfiles { + path := filepath.Join(home, name) + if tc.want == "" { + assert.NoFileExists(t, path) + continue + } + out, err := exec.Command("/bin/sh", "-c", `. "$1"; printf %s "$PATH"`, "sh", path).Output() + require.NoError(t, err) + assert.Equal(t, tc.want, string(out), name) + } + }) + } +} + func TestCursorRunWithImages(t *testing.T) { c := &Cursor{} _, err := c.Run(t.Context(), RunRequest{Dir: t.TempDir(), Images: [][]byte{{1}}}) diff --git a/app/cli/internal/trace/e2e/result_test.go b/app/cli/internal/trace/e2e/result_test.go index fc85fc9ce..b7ae2d757 100644 --- a/app/cli/internal/trace/e2e/result_test.go +++ b/app/cli/internal/trace/e2e/result_test.go @@ -155,7 +155,7 @@ func TestMarkdown(t *testing.T) { {Feature: featTranscript, Harness: HarnessClaudeCode, HarnessVersion: testHarnessVersion, Model: testModel, Status: StatusPass, Passes: 1, Runs: 1}, {Feature: "Spec capture instruction", Harness: HarnessClaudeCode, HarnessVersion: testHarnessVersion, Model: testModel, Status: StatusFail, Passes: 1, Runs: 3}, {Feature: "Session link after push", Harness: HarnessClaudeCode, HarnessVersion: testHarnessVersion, Model: testModel, Status: StatusNotCovered}, - {Feature: featTranscript, Harness: colCursor, Model: "auto", Status: StatusSkipped}, + {Feature: featTranscript, Harness: colCursor, Model: modelAuto, Status: StatusSkipped}, }, } From 0c1bc96c42e642aa432f7adbb776006a3a171e7a Mon Sep 17 00:00:00 2001 From: "Jose I. Paris" Date: Wed, 21 Oct 2026 14:05:32 +0200 Subject: [PATCH 3/3] docs(trace): keep media skipped by redaction unknown for Cursor A pasted image stays inline in the Cursor transcript, so the redaction can see media there. The headless CLI cannot attach an image, so the suite cannot check it. Assisted-by: Claude Code Signed-off-by: Jose I. Paris Chainloop-Trace-Sessions: 0f698192-ddc8-487e-b214-474edc859d4e --- app/cli/internal/trace/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/app/cli/internal/trace/README.md b/app/cli/internal/trace/README.md index 299a4b9c8..ab5539ba2 100644 --- a/app/cli/internal/trace/README.md +++ b/app/cli/internal/trace/README.md @@ -118,12 +118,12 @@ How the evidence is protected before it leaves the machine. | Feature | What it means | Claude Code | Cursor | OpenCode 1.x | OpenCode 2 | Pi | Codex | |---------|---------------|-------------|--------|--------------|------------|---|---| | Secret redaction | Secrets in the session evidence and in the spec materials are replaced with a placeholder before upload. | Yes | Yes | Yes | Yes | No | No | -| Media skipped by redaction | The secret redaction does not scan base64 media (images, PDFs) in the transcript, so it is faster and does not break the media. | Yes | Not applicable | Not applicable | Not applicable | No | No | +| Media skipped by redaction | The secret redaction does not scan base64 media (images, PDFs) in the transcript, so it is faster and does not break the media. | Yes | Unknown | Not applicable | Not applicable | No | No | | Allow markers in redaction | A secret on a line that carries an inline `gitleaks:allow` or `betterleaks:allow` marker, such as a test fixture the agent read, is not redacted. The marker covers only its own line. | Yes | Yes | Yes | Yes | No | No | Why not Yes: -- **Media skipped by redaction, Cursor:** the Cursor transcript holds no tool outputs, so it holds no media that the agent read. +- **Media skipped by redaction, Cursor:** the Cursor transcript holds no tool outputs, so a file that the agent reads leaves no media in it. A pasted image stays inline in the transcript, but it is not verified that it is in the base64 media shape that the redaction skips. The headless Cursor CLI cannot attach an image, so the end-to-end suite cannot check it. - **Media skipped by redaction, OpenCode:** the rebuilt transcript holds no media. ## Git integration