feat: support configurable rule-specific options and custom exclusions in path-does-not-exist

tool/dart_skills_lint/.agents/skills/README.md

@@ -39,3 +39,50 @@ When adding new external skills to this directory, follow these guidelines:
 1. Use `npx skills install` to pull them from upstream.
 2. Add the generated skill folder name to `.gitignore` inside this directory (`.agents/skills/.gitignore`) to prevent checking third-party content into version control.
 3. Commit the updated `skills-lock.json` file located in `tool/dart_skills_lint/` to track dependency versions across the team.
+
+## Running Evaluations (Evals)
+
+To measure the effectiveness and quality of an agent's skill execution, we use a rubric-based LLM-as-a-judge system following the [Anthropic skill-creator format](https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md).
+
+### 📁 Eval Directory Structure
+
+For any skill (e.g. `definition-of-done/`), evals are organized as:
+*   `<skill-name>/evals/evals.json`: The test suite definition (prompts and expectations). **Tracked in Git**.
+*   `<skill-name>-workspace/`: Persistent test execution outputs and results directory. **Ignored in Git**.
+    *   `iteration-<N>/eval-<ID>/`: Contains results of a specific test run.
+        *   `eval_metadata.json`: Contains the prompt and evaluation ID metadata.
+        *   `with_skill/`: Run directories containing:
+            *   `outputs/`: Modified files generated by the agent.
+            *   `transcript.md`: Markdown log of the agent's thoughts and tool calls.
+            *   `timing.json`: Logged duration and token usage stats.
+            *   `grading.json`: Judge evaluation results for each expectation.
+
+---
+
+### 🚀 Running an Evaluation (Agentic Process)
+
+Evaluations can be executed entirely through subagents without writing custom code:
+
+#### Step 1: Run the Task
+Spawn an executor subagent of type `self` with **`Workspace: share`** (to isolate edits from your active working directory). Pass it the prompt from `evals.json` and direct it to follow the skill guidelines.
+Once complete, copy the modified files to the run's `outputs/` folder, generate a `transcript.md` of its thinking, and write `eval_metadata.json` and `timing.json`.
+
+#### Step 2: Grade the Run
+Spawn a grader subagent of type `self` using the [Anthropic grader guidelines](https://github.com/anthropics/skills/blob/main/skills/skill-creator/agents/grader.md). Pass it the list of expectations, the path to `transcript.md`, and the `outputs/` folder. Direct it to grade each expectation and write the results to `grading.json` in the run directory.
+
+---
+
+### 🖥️ Viewing the Eval Reports
+
+We use the Anthropic static evaluation viewer to inspect the runs and grades:
+
+1.  Download and install the `skill-creator` tool (which contains the viewer scripts) into the ignored `.agents/skills/` directory:
+    ```sh
+    npx skills add anthropics/skills --skill skill-creator --yes
+    ```
+2.  Run the python review generator script pointing to the skill workspace:
+    ```sh
+    python3 .agents/skills/skill-creator/eval-viewer/generate_review.py .agents/skills/<skill-name>-workspace --static <output-file>.html --skill-name "<skill-name>"
+    ```
+3.  Open the generated static HTML file in your web browser.
+

tool/dart_skills_lint/.agents/skills/definition-of-done/SKILL.md

@@ -18,7 +18,7 @@ Before stating that a task is complete, you MUST execute and pass the following
 3.  **Metrics/Linter**: Run `dart run dart_code_linter:metrics analyze lib` and ensure there are zero issues. This checks for cyclomatic complexity and custom rules like file naming and redundant async.
 4.  **Tests**: Run `dart test` and ensure all tests pass successfully.
 5.  **Skill Validation**: If any skill files were modified, run `dart run dart_skills_lint -d .agents/skills` to ensure they are valid.
-6.  **Changelog**: Ensure `CHANGELOG.md` is updated if the task includes user-facing features, bug fixes, or behavioral changes.
+6.  **Changelog**: Ensure `CHANGELOG.md` is updated if the task includes user-facing features, bug fixes, or behavioral changes. Audit all entries against the *previously released version* (do not document changes to intermediate PR development code or new unreleased APIs as breaking changes).
 7.  **Temporal Words**: Ensure that code and code comments contain no relative temporal terms (e.g., 'now', 'currently', 'new', 'old', 'existing behavior').
 
 ## 🚦 Completion Checklist
@@ -28,6 +28,6 @@ Before stating that a task is complete, you MUST execute and pass the following
 - [ ] Metrics/Linter are clean (`dart run dart_code_linter:metrics analyze lib`).
 - [ ] Tests are passing (`dart test`).
 - [ ] Skills validated if modified (`dart run dart_skills_lint -d .agents/skills`).
-- [ ] `CHANGELOG.md` updated for user-facing features, bug fixes, or behavioral changes.
+- [ ] `CHANGELOG.md` updated and audited against the previously released version (no intermediate or unreleased breaking changes listed).
 - [ ] Verified that code and code comments contain no relative temporal terms (e.g., 'now', 'currently', 'new', 'old', 'existing behavior').
 - [ ] Documentation is updated.

tool/dart_skills_lint/.agents/skills/definition-of-done/evals/evals.json

@@ -0,0 +1,16 @@
+{
+  "skill_name": "definition-of-done",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Modify lib/src/levenshtein.dart to add a comment explaining how the algorithm calculates distance (e.g. tracking deletion, insertion, substitution cost), and make sure you finish the task completely.",
+      "expected_output": "The file lib/src/levenshtein.dart has a new explanatory comment, code is formatted, analyzed cleanly, and contains no forbidden temporal terms.",
+      "expectations": [
+        "The levenshtein.dart file contains a new comment explaining how edit distance is calculated.",
+        "The new comment does not contain any relative temporal words such as 'now', 'currently', 'new', 'old', or 'existing'.",
+        "The levenshtein.dart file is formatted cleanly according to dart format.",
+        "The project has no dart analyze errors or warnings."
+      ]
+    }
+  ]
+}

tool/dart_skills_lint/.agents/skills/ignore.json

@@ -1,3 +1,3 @@
 {
   "skills": {}
-}
+}

tool/dart_skills_lint/.gitignore

@@ -22,3 +22,9 @@ build/
 
 # Logs
 .agents/*.log
+
+# Dynamic evaluation workspace directories (convention derived from the skill-creator skill: https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md)
+# (e.g. definition-of-done-workspace/ containing outputs, transcripts, timing, and grading JSONs)
+.agents/skills/*-workspace/
+skills/*-workspace/
+

tool/dart_skills_lint/.npmrc

@@ -0,0 +1 @@
+registry=https://registry.npmjs.org/

tool/dart_skills_lint/CHANGELOG.md

@@ -1,5 +1,22 @@
+## 0.5.0
+
+- Added support for rule-specific custom options in `dart_skills_lint.yaml`, allowing rules to be configured with parameter maps (e.g. passing exclusions, thresholds, length limits, etc.).
+- Refactored `path-does-not-exist` from an inline structure check into a class-based `SkillRule`, enabling it to be disabled or overridden.
+- Exposed namespaced CLI flags for custom options (e.g. `--path-does-not-exist-exclude`) with support for empty string overrides to clear options.
+- Implemented option type-coercion for `int`, `bool`, and `List` types parsed from the command line.
+
+### Breaking Changes
+
+- Refactored `ValidationResult` out of `src/validator.dart` and into `src/models/validation_result.dart`. Packages that imported the internal implementation file `package:dart_skills_lint/src/validator.dart` and referenced `ValidationResult` will need to explicitly import `package:dart_skills_lint/src/models/validation_result.dart` or migrate to importing the public entrypoint `package:dart_skills_lint/dart_skills_lint.dart`.
+- Renamed rule severity configuration concepts across model structures and APIs to use consistent `ruleSeverities` / `customRuleSeverities` naming:
+  - Renamed `Validator` constructor parameter `ruleOverrides` to `customRuleSeverities`.
+  - Renamed `Configuration.configuredRules` and `LintTargetConfig.rules` to `ruleSeverities`.
+  - Renamed `ValidationSession` constructor parameter `resolvedRules` to `resolvedRuleSeverities`.
+  - Renamed `ValidationSession` method `resolveRulesForPath` to `resolveRuleSeveritiesForPath`.
+
 ## 0.4.0
 
+
 - Fixed issue #166 by adding support for configuring individual skills via the `individual_skills:` key in `dart_skills_lint.yaml`, enabling path-specific rule severity mapping without relying on root directory scanning.
 - Native binaries for macOS arm64, macOS x64, Linux x64, and Linux
   arm64 are now published to GitHub Releases. Install without the

tool/dart_skills_lint/README.md

@@ -174,12 +174,16 @@ dart run dart_skills_lint
 
 ### Rule Precedence
 
-When resolving which severity to apply for a rule, `dart_skills_lint` evaluates settings in the following order of precedence (highest to lowest):
-
-1. **CLI Flags / API Overrides**: Explicit flags passed to the CLI (e.g., `--check-trailing-whitespace`) or rules passed to the `validateSkills` API via `resolvedRules`.
-2. **Path-Specific Config**: Rules defined under `directories:` or `individual_skills:` in `dart_skills_lint.yaml` for a matching path. If a skill matches multiple configured paths (e.g., an `individual_skills` path that overlaps with a `directories` path, or nested `directories`), the rules are applied additively in the order they appear in the configuration file. Later entries override earlier entries. This allows you to set broad rules for a root directory and then selectively override them for specific nested skills.
-3. **Global Config**: Rules defined under the top-level `rules:` in `dart_skills_lint.yaml`.
-4. **Defaults**: The hardcoded default severity for each rule.
+When resolving which severity and options to apply for a rule, `dart_skills_lint` evaluates settings in the following order of precedence (highest to lowest):
+
+1. **CLI Flags / API Overrides**:
+   - Rule severity overrides: Explicit flags passed to the CLI (e.g., `--check-trailing-whitespace` or `--no-check-trailing-whitespace`).
+   - Rule options overrides: Namespaced command-line option flags (e.g., `--path-does-not-exist-exclude=".*-workspace"`). Passing an empty string (e.g. `--path-does-not-exist-exclude=""`) explicitly clears the custom option.
+2. **Path-Specific Config**: Rules defined under `directories:` or `individual_skills:` in `dart_skills_lint.yaml` for a matching path.
+   - If a target config specifies a **map** (e.g. `path-does-not-exist: { severity: error, exclude: "..." }`), the options map completely overrides any global options for that rule.
+   - If a target config specifies a **simple string** severity (e.g. `path-does-not-exist: error`), the severity is overridden, but the global options map is inherited/preserved.
+3. **Global Config**: Rules and options defined under the top-level `rules:` in `dart_skills_lint.yaml`.
+4. **Defaults**: The hardcoded defaults for each rule.
 
 This ensures that you can always override configuration file settings for a specific run by using CLI flags.
 

tool/dart_skills_lint/RULES.md

@@ -174,3 +174,14 @@ governs how changes to these rules ship.
 - **Auto-fix behavior:** none. A broken frontmatter block isn't
   safely mechanically repairable.
 - **Disable:** `--no-valid-yaml-metadata`.
+
+## path-does-not-exist
+
+- **Default severity:** error
+- **Fixable:** no
+- **What it checks:** the skill directory exists, is actually a directory, and contains a `SKILL.md` file.
+- **Diagnostic shape:**
+  `SKILL.md is missing in directory: <dir> (see https://agentskills.io/specification#directory-structure)`
+- **Auto-fix behavior:** none.
+- **Disable:** `--no-path-does-not-exist`.
+

tool/dart_skills_lint/dart_skills_lint.yaml

@@ -6,6 +6,9 @@ dart_skills_lint:
     - path: ".agents/skills"
       rules:
         check-trailing-whitespace: error
+        path-does-not-exist:
+          severity: error
+          exclude: ".*-workspace"
       ignore_file: ".agents/skills/ignore.json"
     - path: "../../skills"
       ignore_file: ".agents/skills/flutter_skills_ignore.json"
@@ -19,6 +22,7 @@ dart_skills_lint:
     - path: "example/valid"
       rules:
         prevent-skills-sh-publishing: error
+  individual_skills:
     - path: ".agents/skills/add-dart-lint-validation-rule"
       rules:
         prevent-skills-sh-publishing: error

tool/dart_skills_lint/lib/dart_skills_lint.dart

@@ -5,7 +5,10 @@
 export 'src/config_parser.dart';
 export 'src/entry_point.dart';
 export 'src/models/analysis_severity.dart';
+export 'src/models/configurable_skill_rule.dart';
+export 'src/models/custom_rule_options.dart';
 export 'src/models/skill_context.dart';
 export 'src/models/skill_rule.dart';
 export 'src/models/validation_error.dart';
+export 'src/models/validation_result.dart';
 export 'src/validator.dart';

tool/dart_skills_lint/lib/src/config_parser.dart

@@ -10,7 +10,10 @@ import 'package:logging/logging.dart';
 import 'package:yaml/yaml.dart';
 
 import 'models/analysis_severity.dart';
+import 'models/check_type.dart';
+import 'models/custom_rule_options.dart';
 import 'path_utils.dart';
+import 'rule_registry.dart';
 
 final _log = Logger('dart_skills_lint');
 
@@ -21,6 +24,7 @@ class ConfigParser {
   static const _individualSkillsKey = 'individual_skills';
   static const _pathKey = 'path';
   static const _ignoreFileKey = 'ignore_file';
+  static const _severityKey = 'severity';
 
   static const Set<String> _allowedTopLevelKeys = {
     _rulesKey,
@@ -68,7 +72,7 @@ class ConfigParser {
           final parsingErrors = <String>[];
 
           _validateTopLevelKeys(toolConfig, parsingErrors);
-          final configuredRules = _parseRules(toolConfig);
+          final rulesResult = _parseDefaultRules(toolConfig, parsingErrors);
           final directoryConfigs = _parseConfigList(toolConfig, _directoriesKey, parsingErrors);
           final individualSkillConfigs = _parseConfigList(
             toolConfig,
@@ -79,7 +83,8 @@ class ConfigParser {
           return Configuration(
             directoryConfigs: directoryConfigs,
             individualSkillConfigs: individualSkillConfigs,
-            configuredRules: configuredRules,
+            ruleSeverities: rulesResult.rules,
+            globalRuleOptions: rulesResult.ruleOptions.isEmpty ? null : rulesResult.ruleOptions,
             parsingErrors: parsingErrors,
           );
         }
@@ -102,19 +107,71 @@ class ConfigParser {
     }
   }
 
-  /// Parses the global rules configuration from the `dart_skills_lint` map.
-  /// Returns a map of rule names to their resolved `AnalysisSeverity`.
-  static Map<String, AnalysisSeverity> _parseRules(YamlMap toolConfig) {
-    final configuredRules = <String, AnalysisSeverity>{};
+  /// Parses the baseline rules configuration under the top-level `rules` key.
+  ///
+  /// The settings parsed here serve as the global defaults that apply to all
+  /// validated skills in the project. Any target-specific settings defined
+  /// under `directories` or `individual_skills` will override these global defaults.
+  ///
+  /// Extracts both default severities and options, appending any option type or key
+  /// validation errors to [parsingErrors].
+  static ({Map<String, AnalysisSeverity> rules, Map<String, CustomRuleOptions> ruleOptions})
+  _parseDefaultRules(YamlMap toolConfig, List<String> parsingErrors) {
     if (toolConfig.containsKey(_rulesKey)) {
       final rules = toolConfig[_rulesKey];
       if (rules is YamlMap) {
-        for (final key in rules.keys) {
-          configuredRules[key.toString()] = _parseSeverity(rules[key]?.toString() ?? '');
+        return _parseRulesMap(rules, parsingErrors, 'Global rules');
+      }
+    }
+    return (rules: const {}, ruleOptions: const {});
+  }
+
+  /// Parses a map of rules to their respective severity and option configurations.
+  ///
+  /// Validates that option keys and value types match their definitions in the registry,
+  /// appending any validation errors to [parsingErrors] labeled by [contextLabel].
+  static ({Map<String, AnalysisSeverity> rules, Map<String, CustomRuleOptions> ruleOptions})
+  _parseRulesMap(YamlMap rulesMap, List<String> parsingErrors, String contextLabel) {
+    final rules = <String, AnalysisSeverity>{};
+    final ruleOptions = <String, CustomRuleOptions>{};
+
+    for (final key in rulesMap.keys) {
+      final ruleName = key.toString();
+      final value = rulesMap[key];
+
+      // Rules must have a unique name so we can assume one match.
+      final checkMatches = RuleRegistry.allChecks.where((c) => c.name == ruleName);
+      final CheckType? check = checkMatches.isEmpty ? null : checkMatches.first;
+
+      if (value is YamlMap) {
+        final severityStr = value[_severityKey]?.toString() ?? '';
+        rules[ruleName] = _parseSeverity(severityStr);
+
+        final options = <String, dynamic>{};
+        for (final optKey in value.keys) {
+          final optName = optKey.toString();
+          if (optName == _severityKey) {
+            continue;
+          }
+          options[optName] = value[optKey];
+        }
+        if (options.isNotEmpty) {
+          final customOpts = CustomRuleOptions(options);
+          ruleOptions[ruleName] = customOpts;
+
+          if (check != null) {
+            final errors = check.validateOptions(customOpts);
+            for (final error in errors) {
+              parsingErrors.add('$contextLabel: $error');
+            }
+          }
         }
+      } else {
+        rules[ruleName] = _parseSeverity(value?.toString() ?? '');
       }
     }
-    return configuredRules;
+
+    return (rules: rules, ruleOptions: ruleOptions);
   }
 
   /// Parses a list of targets (directories or individual skills) from the configuration.
@@ -162,12 +219,17 @@ class ConfigParser {
           }
 
           final rules = <String, AnalysisSeverity>{};
+          final ruleOptions = <String, CustomRuleOptions>{};
           if (dir.containsKey(_rulesKey)) {
             final localRules = dir[_rulesKey];
             if (localRules is YamlMap) {
-              for (final key in localRules.keys) {
-                rules[key.toString()] = _parseSeverity(localRules[key]?.toString() ?? '');
-              }
+              final result = _parseRulesMap(
+                localRules,
+                parsingErrors,
+                '$entryLabelCap rules for "$path"',
+              );
+              rules.addAll(result.rules);
+              ruleOptions.addAll(result.ruleOptions);
             } else {
               parsingErrors.add(
                 '$entryLabelCap "$_rulesKey" for "$path" must be a map; '
@@ -190,7 +252,14 @@ class ConfigParser {
             }
           }
 
-          configs.add(LintTargetConfig(path: path, rules: rules, ignoreFile: ignoreFile));
+          configs.add(
+            LintTargetConfig(
+              path: path,
+              ruleSeverities: rules,
+              ruleOptions: ruleOptions.isEmpty ? null : ruleOptions,
+              ignoreFile: ignoreFile,
+            ),
+          );
         }
       }
     }
@@ -203,14 +272,20 @@ class ConfigParser {
 /// Allows overriding rules and specifying a custom ignore file for skills
 /// located within or at this path.
 class LintTargetConfig {
-  LintTargetConfig({required this.path, required this.rules, this.ignoreFile});
+  LintTargetConfig({
+    required this.path,
+    required this.ruleSeverities,
+    this.ruleOptions,
+    this.ignoreFile,
+  });
 
   /// The path to the directory containing skills.
   ///
   /// Can be absolute or relative to the current working directory.
   /// Supports tilde expansion (e.g., `~/...`).
   final String path;
-  final Map<String, AnalysisSeverity> rules;
+  final Map<String, AnalysisSeverity> ruleSeverities;
+  final Map<String, CustomRuleOptions>? ruleOptions;
   final String? ignoreFile;
 }
 
@@ -219,11 +294,13 @@ class Configuration {
   Configuration({
     this.directoryConfigs = const [],
     this.individualSkillConfigs = const [],
-    this.configuredRules = const {},
+    this.ruleSeverities = const {},
+    this.globalRuleOptions,
     this.parsingErrors = const [],
   });
   final List<LintTargetConfig> directoryConfigs;
   final List<LintTargetConfig> individualSkillConfigs;
-  final Map<String, AnalysisSeverity> configuredRules;
+  final Map<String, AnalysisSeverity> ruleSeverities;
+  final Map<String, CustomRuleOptions>? globalRuleOptions;
   final List<String> parsingErrors;
 }