Skip to content

feat: add 5 new agentic benchmark scenarios - #1

Closed
jannemakela wants to merge 1 commit into
mainfrom
feat/additional-agentic-scenarios
Closed

feat: add 5 new agentic benchmark scenarios#1
jannemakela wants to merge 1 commit into
mainfrom
feat/additional-agentic-scenarios

Conversation

@jannemakela

Copy link
Copy Markdown

Summary

Adds 5 new agentic test scenarios that cover real-world in-house legal workflows typically handled by corporate legal teams. All scenarios are multi-turn and require read-reason-edit patterns.

New Scenarios

ID Name What it tests
comment-driven-edit Comment-Driven Edit Read margin comment, verify change, reply to comment
multi-location-update Multi-Location Update Rebrand company name across body, headers, footers, signature block
defined-term-insertion Defined Term Insertion Add defined term to definitions + propagate to another clause
liability-cap-rewrite Liability Cap Rewrite Read numeric value, reason about format, update word + numeral
clause-deletion-and-renumber Clause Deletion & Renumber Delete section entirely + renumber subsequent sections

Why These Scenarios

These represent the most common agentic editing patterns in contract review workflows:

  • Comment interaction (comment-driven-edit) — the core review loop
  • Global consistency (multi-location-update) — name/term changes must be exhaustive
  • Multi-step reasoning (defined-term-insertion) — add + propagate, requires understanding document structure
  • Read-compute-write (liability-cap-rewrite) — must parse both textual and numeric representations
  • Structural modification (clause-deletion-and-renumber) — must understand section numbering

Changes

  • src/scenarios.ts — 5 new scenario definitions (all isAgentic: true)
  • src/success.ts — success check functions for each new scenario
  • src/scenarios.test.ts — updated to expect 15 scenarios with assertions
  • README.md — updated scenario list

Tests

All 11 tests passing ✅ (lint + format + vitest)


Contributed under AGPL-3.0-only per repository license.

New scenarios covering real-world in-house legal workflows:

- comment-driven-edit: Read a margin comment, verify the described
  change, and reply to the comment. Tests comment interaction.
- multi-location-update: Rebrand a company name across body text,
  headers, footers, and signature blocks. Tests exhaustive search.
- defined-term-insertion: Add a new defined term to definitions,
  then propagate its usage into another clause. Tests multi-step
  reasoning and consistency.
- liability-cap-rewrite: Read a liability cap duration, update both
  the written word and parenthetical numeral. Tests read-reason-edit.
- clause-deletion-and-renumber: Delete an entire section and renumber
  the subsequent section. Tests structural document modification.

All scenarios are marked isAgentic=true and include success checks.
Tests updated (15 total scenarios, all passing).

Licensed under AGPL-3.0-only per repository license.
@jannemakela

Copy link
Copy Markdown
Author

Closing — scenarios need to be rebuilt with real open-source legal documents (Common Paper, NVCA, etc.) and realistic end-to-end workflows (form fill, contract clone, policy check, playbook commenting, multi-file deals).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant