Website · Paper · Contribute
Run the ready ScopeBench Lite Docker tasks from the repository root:
uvx harbor run -c configs/paired_lite.json \
--ve 'ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}'The setup and run guide explains Docker image access, verifier credentials, and job settings for the ready dataset.
An agent can finish a security task by taking a step it was explicitly told to avoid. ScopeBench is an open benchmark for AI security agents that measures technical capability and scope adherence, pairing a capability task with a scoped task in the same environment. Only the instructions change.
The capability task tests whether the agent can complete the objective, while the scope task places an authorization boundary across every known route to that same objective. A successful finding can still violate scope.
All three benchmarks share an evaluation protocol and contribution process, with new tasks under tasks/ and the pilot scenarios retained in ScopeBench Lite.
Web application and API assessments test boundaries around tenants and data.
Tasks · Environments and build guide
Cloud scenarios cover identities, resources, and control planes, with authorization boundaries across accounts and services.
Tasks · Environment and provider guide
This benchmark covers Linux hosts, internal networks, Windows, and Active Directory.
Tasks · Linux build guide · Windows/AD build guide
Help build a realistic scenario. We also welcome task repairs and improvements to environments and tooling, following the same contribution process so reviewers can assess each change and its evidence.
| Step | Where to start |
|---|---|
| Propose | Read the task proposal rubric, browse main environments and Lite scenarios, and open a Task Proposals discussion for early feedback. |
| Build | Follow the contribution guide and your benchmark's build guide. Store shared scenarios in environments/ and new task pairs in tasks/. |
| Submit | Open a pull request with validation and run evidence for maintainer review. |
Report a broken task. Use the issue form to describe the failure or follow the repair guide to fix it, with tooling checks and image publication covered in the CI guide.