Skip to content

Add agent instruction validators#2245

Open
ather-techie wants to merge 3 commits into
github:mainfrom
ather-techie:add-agent-instruction-validators
Open

Add agent instruction validators#2245
ather-techie wants to merge 3 commits into
github:mainfrom
ather-techie:add-agent-instruction-validators

Conversation

@ather-techie

@ather-techie ather-techie commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • agents/ (220 *.agent.md) and instructions/ (189 *.instructions.md) were the two largest flat content sets in the repo without a validator or CI gate, unlike skills (validate-skills.mjs) and plugins (validate-plugins.mjs).
  • Add eng/validate-agents.mjs and eng/validate-instructions.mjs, modeled on validate-skills.mjs's frontmatter-parsing and error-reporting pattern. They enforce the required fields from AGENTS.md / docs/README.*.md as blocking errors (description on agents; description + applyTo on instructions) and flag recommended-but-optional conventions (filename casing, name / model / tools on agents) as non-blocking warnings.
  • Wire them up as npm run agent:validate / npm run instruction:validate, documented in AGENTS.md alongside the existing skill:validate / plugin:validate commands. Like those two, they default to a full-corpus scan when run with no file arguments.
  • Run them in CI as new steps in validate-readme.yml (alongside the existing plugin:validate step) rather than as a separate workflow. skill-check.yml already triggers on agents/** changes and reports agent files as "detected but skipped" (vally only lints SKILL.md content); a second, separately-triggered workflow silently validating the same files would have shown up as a confusing, seemingly-contradictory pair of checks on the same PR. validate-readme.yml already triggers on both agents/** and instructions/**, so this keeps content-integrity checks in one place.
  • Fix 17 pre-existing frontmatter gaps the new validators surfaced (1 agent and 16 instruction files missing description and/or applyTo, mostly the dataverse-python-* series) so both validators pass cleanly across the full corpus today.

Test plan

  • npm run agent:validate - passes on the full corpus of 220 agent files
  • npm run instruction:validate - passes on the full corpus of 189 instruction files
  • npm run plugin:validate - unaffected, still passes
  • validate-readme.yml YAML validated for syntax
  • Verified the file-args CLI path each validator supports: exits 0 on a clean file, exits 1 with the correct error on a broken one

@ather-techie ather-techie requested a review from aaronpowell as a code owner July 8, 2026 04:33
@github-actions github-actions Bot added agent PR touches agents instructions PR touches instructions new-submission PR adds at least one new contribution workflow PR touches workflow automation labels Jul 8, 2026
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔒 PR Risk Scan Results

Scanned 22 changed file(s).

Severity Count
🔴 High 0
🟠 Medium 22
ℹ️ Info 0
Severity Rule File Line Match
🟠 unpinned-version-indicator eng/validate-agents.mjs 110 if (errors.length > 0) {
🟠 unpinned-version-indicator eng/validate-agents.mjs 134 cliArgs.length > 0
🟠 unpinned-version-indicator eng/validate-instructions.mjs 112 if (errors.length > 0) {
🟠 unpinned-version-indicator eng/validate-instructions.mjs 138 cliArgs.length > 0
🟠 unpinned-version-indicator instructions/codexer.instructions.md 45 - If your function is >50 lines, you're doing it wrong
🟠 unpinned-version-indicator instructions/codexer.instructions.md 298 - Any function >50 lines = rewrite or reject
🟠 unpinned-version-indicator instructions/codexer.instructions.md 303 - Nested loops >3 levels = refactor now
🟠 unpinned-version-indicator instructions/codexer.instructions.md 409 - If a library has >1000 GitHub stars and recent commits, it's probably safe
🟠 unpinned-version-indicator instructions/dataverse-python-advanced-features.instructions.md 143 WHERE a.creditlimit > 50000
🟠 unpinned-version-indicator instructions/dataverse-python-best-practices.instructions.md 182 # SDK automatically uses CreateMultiple for arrays > 1 record
🟠 unpinned-version-indicator instructions/dataverse-python-best-practices.instructions.md 228 WHERE creditlimit > 50000
🟠 unpinned-version-indicator instructions/dataverse-python-file-operations.instructions.md 55 **When to use:** Large videos, databases, archives > 128 MB
🟠 unpinned-version-indicator instructions/dataverse-python-file-operations.instructions.md 112 print(f"Using chunked upload (file > 128 MB)")
🟠 unpinned-version-indicator instructions/dataverse-python-pandas-integration.instructions.md 88 filter="creditlimit > 50000",
🟠 unpinned-version-indicator instructions/dataverse-python-pandas-integration.instructions.md 464 filter="creditlimit > 50000", # Server-side filter
🟠 unpinned-version-indicator package.json 44 "ajv": "^8.20.0",
🟠 unpinned-version-indicator package.json 45 "ajv-formats": "^3.0.1",
🟠 unpinned-version-indicator package.json 46 "@​​microsoft/vally": "^0.6.0",
🟠 unpinned-version-indicator package.json 47 "all-contributors-cli": "^6.26.1"
🟠 unpinned-version-indicator package.json 50 "js-yaml": "^5.2.0",
🟠 unpinned-version-indicator package.json 51 "vfile": "^6.0.3",
🟠 unpinned-version-indicator package.json 52 "vfile-matter": "^5.0.1"

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔍 Vally Lint Results

⚠️ Warnings or advisories found

Scope Checked
Skills 0
Agents 1
Total 1
Severity Count
❌ Errors 0
⚠️ Warnings 0
ℹ️ Advisories 1

Summary

Level Finding
ℹ️ Vally currently lints SKILL.md content. Agent files were detected but skipped:
Full linter output
### Agent files detected (not linted by vally)
ℹ️ Vally currently lints SKILL.md content. Agent files were detected but skipped:
agents/declarative-agents-architect.agent.md

@github-actions github-actions Bot removed the new-submission PR adds at least one new contribution label Jul 8, 2026
@aaronpowell

Copy link
Copy Markdown
Contributor

This looks like a pretty good starting point at improving the consistency of agents and instructions (previously we'd been relying on CCR to do these suggestions).

The only real question is where we draw the line on the fields that should be validated in the frontmatter. For example VS Code has a lot that are not part of the base design. This is because there's never really been a true spec for Agents or Instructions.

Because there's been some updates to files, the readme needs to be updated, and then we can merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent PR touches agents instructions PR touches instructions workflow PR touches workflow automation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants