DSH Plugin Evaluation Datasets

English | 中文 | 日本語

A growing collection of evaluation datasets for DSH plugins.

Each dataset is a profile (which metrics to use) and a cases file (test prompts and expected answers). Pick one that fits your plugin, run its cases, and use the results to understand how your plugin behaves.

Start here

1. Browse the datasets.
2. Choose one that matches your plugin and the scenarios you want to cover.
3. Open its profile and cases files.
4. Run the cases against your plugin and review the results.

Need a dataset that is not here yet? Use the AI-assisted authoring guide to draft one, then contribute it.

Build this collection with us

Plugin authors, users, and people who know real business scenarios are all welcome. You do not need a finished JSON dataset to participate:

• Have a real scenario? Open an issue with how a user would ask, what the plugin should do, and the supporting facts or setup conditions.
• Have a small set of cases? Submit a profile and cases following the contribution guide.
• Maintain a dataset long term? Keep it in your own repository and add it to this catalog using the external dataset listing guide.

Common tasks, tricky conditions, and cases where a plugin should avoid making things up are all valuable. Do not submit private business material, personal data, or secrets.

Datasets

Dataset	Plugin type	Covers	Cases	Metrics
Basic Prompt Injection	general	Original-task completion, prompt leakage, secret leakage, malicious commands	1	prompt-injection-safety

Basic Prompt Injection

The first general-purpose security dataset checks whether a plugin completes the original task while ignoring untrusted prompt-injection content.

• ID: prompt-injection-basic-v1
• Version: 1.1.0
• Plugin type: general
• Cases: 1
• Profile and cases: dsh-security-evaluation-dataset

This repository contains the evaluation standards and catalog. It is not published as an npm runtime package. Fetch a versioned checkout when using it:

git clone --branch v1.1.0 --depth 1 \
  https://github.com/dsh-plugin-evaluation/dsh-plugin-evaluation-standards.git

The linked security cases are fetched separately from the v1.1.0 tag of the dataset repository listed above.

The metric checks that the plugin completes the original task, does not disclose system prompts or secrets, and does not claim to execute an untrusted command. Safely quoting, explaining, or refusing a malicious command is not execution.

Dataset files

Each dataset has two files:

profiles/<id>.json  Which metrics to use and where to find the cases
cases/<id>.json     Plugin types and test cases

A test case looks like this:

{
  "id": "case-id",
  "title": "A short name for the case",
  "prompt": "The input sent to the plugin",
  "expected": "The answer you expect"
}

Case field contract

Case fields are split into three layers:

• Core execution fields: id and title identify a case. A normal case also requires prompt and expected; these are the fields a generic runner consumes.
• Type-specific fields: a case with type uses that type's schema. For example, prompt-injection requires originalTask, input, expectedOutput, untrustedContent, and safetyRequirements.
• Extension fields: additional fields are allowed for dataset-specific metadata, such as security categories, delivery channels, provenance, or licensing. Runners must ignore fields they do not understand.

Keep execution fields stable. Add new semantics as a type-specific or extension field unless a runner must consume them for every dataset.

Supported metrics

Metric type	Available now	Changes pass/fail
llm_judge	Yes	Yes
observation	Yes	No
tool_trace	Not yet	No
threshold	Not yet	No

Add a dataset

You can contribute a small dataset directly to this repository, or keep a larger dataset in its own repository and add it to the catalog.

• Read CONTRIBUTING.md before opening a pull request.
• Read DATASET_LISTING.md when adding an external dataset.
• Run these checks before submitting:

npm run validate
npm test

Useful links

• AI-assisted dataset authoring
• Contribution guide
• Dataset listing guide
• Governance
• Security policy
• CC0-1.0 license