Skip to content

Feat/verification pilot nist ai rmf - #216

Open
a-moskvin wants to merge 2 commits into
GenAI-Security-Project:mainfrom
a-moskvin:feat/verification-pilot-nist-ai-rmf
Open

a-moskvin wants to merge 2 commits into
GenAI-Security-Project:mainfrom
a-moskvin:feat/verification-pilot-nist-ai-rmf

Conversation

@a-moskvin

Copy link
Copy Markdown

**Depends on #215 ** (data model and validator). Until that merges, this diff also shows its commits; the pilot itself is 3 files.

What this PR changes

Pilot for the verification-methods data model (#191): 10 draft methods for NIST AI RMF 1.0.

  • data/verification-methods.json: VM-0001–VM-0010
  • data/verification-links/nist-ai-rmf.json: 10 links, 6 control-level and 4 row-level (ASI01, ASI02, ASI03, ASI10)
  • CHANGELOG.md

Type of change

  • New mapping file
  • Update to existing mapping (content, controls, CVE refs)
  • Bug fix (broken link, typo, incorrect cross-ref)
  • New recipe (shared/RECIPES.md)
  • New tool (shared/TOOLS.md)
  • Infrastructure (scripts, CI, templates)
  • Translation (i18n/)

Source / evidence

Checklist

Content

  • Follows the file template structure (header comment, H1, Why section, quick-reference table, audience
    tags, detailed per-entry mappings, references, changelog)
  • Severity ratings consistent with AIVSS / OWASP definitions in shared/SEVERITY.md
  • Cross-references are bidirectional — if this file mentions Agentic_X.md, that file mentions this one back
  • All referenced vulnerability IDs are valid (LLM01–LLM10, ASI01–ASI10, DSGAI01–DSGAI21)
  • License header present: CC BY-SA 4.0

Links & data

  • All internal .md links resolve to real files
  • All external URLs return 200 (checked manually or via lychee)
  • data/schema.json compatible (if adding a new entry type)

Project hygiene

  • npm run build passes: 0 errors, no verification warnings
  • npm test passes
  • Changelog entry added at bottom of every modified file (YYYY-MM-DD format)
  • CHANGELOG.md updated if this is a new mapping file (include in correct version section)
  • README.md counts updated if file count changed (badge + summary table + section heading + repo tree)
  • Ran node scripts/validate.js --file <path> locally and it passes

For new mapping files only

  • Added to correct section in README.md mapping table with "Standout content" description
  • Added to CROSSREF.md primary frameworks column where relevant
  • Framework coverage matrix in README.md updated (✅ for the new cell)
  • File named correctly: SourceList_Framework.md (e.g., Agentic_SAMM.md)

Notes for reviewers

  • Scope: 10 selected rows from existing NIST AI RMF 1.0 mapping.
  • Sources: 8 methods are adapted from the NPW Agentic AI Control Catalogue v2.4.0 (CC BY-SA 4.0). Each cites the control id and URL, with the original wording in source.text. VM-0004 and VM-0009 are project-authored (source: null).
  • Review state: all methods and links are draft. They were drafted with AI assistance (Claude) and reviewed and edited by me. I am the contributor, and NPW is my company, so I have not marked anything reviewed. Independent SME review is welcome: a reviewer's name goes into reviewed_by in a follow-up. The links are also draft, i.e. no reviewed_by.

Disclosure: NPW (New Pacific Way Ltd.) is my company; eight of the ten methods derive from its catalogue.

@emmanuelgjr

Copy link
Copy Markdown
Contributor

Thanks, @a-moskvin. The maintainer has decided to hold this PR until #215 and #219 land. Here's what the review found and what would move it forward.

What checks out

  • All 8 NPW citations resolve: H06, A01, J05, I07, J04, C02, E09 and I01 exist in catalogue v2.4.0, and each source.text matches the control's "Verification method" field word for word. The CC BY-SA 4.0 licence is stated on the site.
  • Every expected_result is observable, and nothing is marked reviewed. Thank you for the disclosure in the PR body.

Before it can merge

  1. Rebase after data: re-transcribe the NIST AI RMF registry from AI RMF 1.0 #219. The pilot was written against main's registry text, which data: re-transcribe the NIST AI RMF registry from AI RMF 1.0 #219 corrects. MS-2.6 and MP-5.1 change materially; on the corrected text VM-0009 and VM-0010 actually fit better. Please re-read all six targets against data: re-transcribe the NIST AI RMF registry from AI RMF 1.0 #219's descriptions.

  2. Put the disclosure in the repo. GOVERNANCE.md (lines 97–102) asks for disclosed conflicts of interest to be noted in the changelog entry. Right now the ownership of NPW appears only in the PR body.

  3. Re-check two links:

    • VM-0005 → GV-1.7: isolation and re-entry reads as containment (NPW files I07 under "Containment & Incident Response"), while GV-1.7 is about decommissioning.
    • VM-0008 → MP-4.1: signature and SBOM checks test component integrity, while MP-4.1 is about documenting how technology and legal/IP risks are mapped.

    VM-0002 → GV-1.6 (agent identity vs inventory) is also worth a second look. These are flags for a reviewer, not rulings.

  4. Prefer row-level links. Most of these methods verify an agentic implementation rather than the GOVERN/MAP subcategory in general, so row-level links would overstate less than control-level ones.

  5. Reference the NIST AI RMF Playbook. It publishes Suggested Actions and Transparency & Documentation questions for each subcategory. It isn't examine/interview/test procedures, so this isn't the same overlap as AIUC-1 on feat(schema): optional verification_method field + Agentic_AIUC1 pilot #186. Still, citing the Playbook entry alongside each method would anchor it to NIST's own guidance.

A small licence point to confirm with NPW: the site licenses "control statements" under CC BY-SA 4.0, and it's worth confirming that the verification-method field is included.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants