Skip to content

feat(trace): check net contribution, session updates and rewrites end to end - #3685

Merged
jiparis merged 3 commits into
chainloop-dev:mainfrom
jiparis:feat/trace-e2e-git-checks
Oct 21, 2026
Merged

jiparis merged 3 commits into
chainloop-dev:mainfrom
jiparis:feat/trace-e2e-git-checks

Conversation

@jiparis

@jiparis jiparis commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Closes three suite gaps of the Chainloop Trace end-to-end suite (spec issue-3635, R-002, R-005, R-011). The Net contribution, Session updates, and Renames and rewrites rows now have registered checks. It refs #3635.

  • A scenario can run the git commands of a human after the agent session (FollowUp), then push again. The runner reads the session that the last push stored.
  • A scenario can work on a branch (Branch), which the workspace creates from main and pushes before the session. The push reads only the commits that are not on the default branch of the remote, so a second push finds the session commits only on another branch.
  • human-update: a human commit replaces one of the three AI lines. Net contribution checks 2 AI lines, 1 human line, and the file still created. Session updates checks that the human commit is in the commit list of the session and that the replacing line is human.
  • amend: a human adds a line to the pushed session commit and force-pushes. The session follows the amended commit, with 3 AI lines and 1 human line.
  • rename: the agent commits with no push, and a human renames the file in a second commit, then pushes both. The 3 lines stay AI lines.
  • The README documents two limits that the suite found:
    • A session commit that is already on the default branch of the remote is not read again, so a later commit does not update the session.
    • A rename inside an amend or a squash of the session commit loses the AI lines. This goes against spec issue-3639 R-007, which says that attribution must survive a rename and an amend. This PR documents the gap and does not change Trace.

No cell value changes.

This PR was written with the help of Claude Code.

View guided diff

… to end

Add scenarios that run the git commands of a human after the agent
session and push again: a commit that replaces an AI line, an amend
of the session commit, and a rename in a later commit. They work on a
feature branch, because the push reads only the commits that are not
on the default branch of the remote. Register the checks of the Net
contribution, Session updates and Renames and rewrites rows.

Refs chainloop-dev#3635

Assisted-by: Claude Code
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>

Chainloop-Trace-Sessions: 0f698192-ddc8-487e-b214-474edc859d4e
@chainloop-platform

chainloop-platform Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

PR validation — ✅ 3 passing

Status Policy Material Messages
✅ Passed pr-min-approvals pr-info -
✅ Passed pr-description-required pr-info -
✅ Passed pr-user-story-linked pr-info -

View attestation ↗

AI Session Checks — 🟢 88% · ✅ 0 failing

Avg score Sessions Failing policies Attribution Files Lines Total Duration
🟢 88% 1 ✅ 0 100% AI / 0% Human 11 +516 / -16 1h25m1s

🟢 88% — 100% AI — ✅ All policies passing

Oct 21, 2026 10:40 UTC · 1h25m1s · $21.04 · 592 in / 219.1k out · claude-code 2.1.306 (claude-opus-5-5)

View session details ↗

Change Summary

  • Adds shared end-to-end scenarios and checks for Net contribution, Session updates, and Renames and rewrites.
  • Extends the runner and workspace to support post-session human git follow-ups on a feature branch.
  • Adds focused unit tests for the new check helpers and documents the discovered session-update and rename-in-amend limits in the Trace README.

AI Session Overall Score

🟢 88% — Strong implementation and validation, with one spec-named acceptance check left unshown.

AI Session Analysis Breakdown

🟢 95% · alignment

No notes.

🟢 94% · user-trust-signal

No notes.

🟢 93% · scope-discipline

🟢 It left the fourth gap and the rename-in-amend fix to separate follow-up work. · High Impact

🟢 89% · solution-quality

No notes.

🟢 88% · context-and-planning

🟢 The session had a real ticket and full spec before the bulk of coding began. · High Impact

🟡 78% · verification

🟢 The AI reran the named three-feature end-to-end matrix until all targeted rows passed. · High Impact

🟠 The spec-named check that a README cell alone changes test selection was not exercised in-session. · Medium Severity

💡 Add one focused run that flips a README cell and shows the selected checks change without other edits.


File Attribution

████████████████████ 100% AI / 0% Human

Status Attribution File Lines
created ai app/cli/internal/trace/e2e/cursor.go +262 / -0
created ai app/cli/internal/trace/e2e/cursor_test.go +160 / -0
created ai app/cli/internal/trace/e2e/checks_test.go +57 / -0
modified ai app/cli/internal/trace/README.md +10 / -11
modified ai app/cli/internal/trace/e2e/checks.go +9 / -0
modified ai app/cli/internal/trace/e2e/config.yaml +9 / -0
modified ai app/cli/internal/trace/PROVIDERS.md +2 / -2
modified ai app/cli/internal/trace/e2e/claude.go +2 / -2
modified ai app/cli/internal/trace/e2e/harness.go +3 / -0
modified ai app/cli/internal/trace/e2e/result_test.go +1 / -1
modified ai app/cli/internal/trace/e2e/e2e_test.go +1 / -0

Policies (4)

Status Policy Material Messages
✅ Passed ai-config-ai-agents-allowed ai-coding-session-0f6981 -
✅ Passed ai-config-no-dangerous-commands ai-coding-session-0f6981 -
✅ Passed ai-config-no-secrets ai-coding-session-0f6981 -
✅ Passed ai-config-mcp-servers-allowed ai-coding-session-0f6981 -

Security Checks — ✅ 5 passing

✅ secret-scan

Status Policy Messages
✅ Passed secrets-detection -

✅ sast-scan

Status Policy Messages
✅ Passed owasp-top10-2025 -
✅ Passed sast -
✅ Passed cwe-top25 -
✅ Passed cwe-top26-40-cusp -
Scans not applied (3)
Scan Reason
vulnerability-scan no manifest/lockfile changed
github-actions-scan no workflow files changed
iac-scan no IaC files changed

View attestation ↗

Security context

✅ Nothing this change touches has a recorded security-fix history.

View in Chainloop ↗ · How this works ↗


Powered by Chainloop and Chainloop Trace

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 7 files

View guided diff | Re-trigger cubic

Assisted-by: Claude Code
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>

Chainloop-Trace-Sessions: 0f698192-ddc8-487e-b214-474edc859d4e
@jiparis
jiparis requested a review from a team October 21, 2026 12:06
javirln
javirln previously approved these changes Oct 21, 2026
…checks

Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>

# Conflicts:
#	app/cli/internal/trace/e2e/checks_test.go
@jiparis
jiparis merged commit b5e2fe1 into chainloop-dev:main Oct 21, 2026
20 of 21 checks passed
@jiparis
jiparis deleted the feat/trace-e2e-git-checks branch October 21, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants