feat: add 5 new agentic benchmark scenarios - #1
Closed
jannemakela wants to merge 1 commit into
Closed
Conversation
New scenarios covering real-world in-house legal workflows: - comment-driven-edit: Read a margin comment, verify the described change, and reply to the comment. Tests comment interaction. - multi-location-update: Rebrand a company name across body text, headers, footers, and signature blocks. Tests exhaustive search. - defined-term-insertion: Add a new defined term to definitions, then propagate its usage into another clause. Tests multi-step reasoning and consistency. - liability-cap-rewrite: Read a liability cap duration, update both the written word and parenthetical numeral. Tests read-reason-edit. - clause-deletion-and-renumber: Delete an entire section and renumber the subsequent section. Tests structural document modification. All scenarios are marked isAgentic=true and include success checks. Tests updated (15 total scenarios, all passing). Licensed under AGPL-3.0-only per repository license.
Author
|
Closing — scenarios need to be rebuilt with real open-source legal documents (Common Paper, NVCA, etc.) and realistic end-to-end workflows (form fill, contract clone, policy check, playbook commenting, multi-file deals). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds 5 new agentic test scenarios that cover real-world in-house legal workflows typically handled by corporate legal teams. All scenarios are multi-turn and require read-reason-edit patterns.
New Scenarios
comment-driven-editmulti-location-updatedefined-term-insertionliability-cap-rewriteclause-deletion-and-renumberWhy These Scenarios
These represent the most common agentic editing patterns in contract review workflows:
Changes
src/scenarios.ts— 5 new scenario definitions (allisAgentic: true)src/success.ts— success check functions for each new scenariosrc/scenarios.test.ts— updated to expect 15 scenarios with assertionsREADME.md— updated scenario listTests
All 11 tests passing ✅ (lint + format + vitest)
Contributed under AGPL-3.0-only per repository license.