diff --git a/.opencode/agents/validator.md b/.opencode/agents/validator.md index 96c48e6d..3be00395 100644 --- a/.opencode/agents/validator.md +++ b/.opencode/agents/validator.md @@ -24,6 +24,7 @@ Before validating a finding, read: - `.opencode/skills/finding-format/SKILL.md` - `.opencode/skills/exploit-validation/SKILL.md` - `.opencode/skills/sandbox-validation/SKILL.md` +- If present, `itemdb/notes/threat-model.md` — the operational threat model from Phase 1b. Use it to ensure validation planning respects documented attacker capabilities, non-capabilities, trust boundaries, existing controls, and open assumptions. - relevant files under `itemdb/notes/` - the assigned finding - relevant source files under `src/` @@ -117,13 +118,14 @@ Multiple methods may be combined. 2. Extract the exact vulnerability claim. 3. Review the counter-analysis. 4. Identify what evidence would confirm or reject the claim. -5. Inspect the relevant source code. -6. Prepare the sandbox. -7. Execute the validation plan or improve it. -8. Record commands and outputs. -9. Store evidence under `itemdb/evidence//`. -10. Update the finding. -11. Move the finding to the correct status directory if needed. +5. When the threat model exists, review how the finding's validation plan aligns with documented attacker capabilities, trust boundaries, existing controls, and open assumptions. Refine the plan if it assumes capabilities contradicted by the threat model. +6. Inspect the relevant source code. +7. Prepare the sandbox. +8. Execute the validation plan or improve it. +9. Record commands and outputs. +10. Store evidence under `itemdb/evidence//`. +11. Update the finding. +12. Move the finding to the correct status directory if needed. ## Evidence directory diff --git a/.opencode/skills/exploit-validation/SKILL.md b/.opencode/skills/exploit-validation/SKILL.md index 53872678..fc16719a 100644 --- a/.opencode/skills/exploit-validation/SKILL.md +++ b/.opencode/skills/exploit-validation/SKILL.md @@ -30,6 +30,7 @@ Read: - the assigned finding - relevant source files under `src/` - sandbox documentation under `sandbox/` +- If present, `itemdb/notes/threat-model.md` — the operational threat model from Phase 1b ## Outputs @@ -104,13 +105,14 @@ For the assigned finding: 1. Read the finding completely. 2. Understand the hypothesis and counter-analysis. 3. Identify the minimal proof needed. -4. Inspect relevant source code. -5. Determine the best validation method. -6. Prepare the sandbox. -7. Execute validation steps. -8. Capture commands, inputs, outputs, logs, and observations. -9. Decide whether the finding is confirmed, rejected, or unresolved. -10. Update the finding and evidence directory. +4. When the threat model is present, review how the finding's claimed attacker profile, trust boundary, and impact align with the threat model's documented capabilities, non-capabilities, and existing controls. +5. Inspect relevant source code. +6. Determine the best validation method. +7. Prepare the sandbox. +8. Execute validation steps. +9. Capture commands, inputs, outputs, logs, and observations. +10. Decide whether the finding is confirmed, rejected, or unresolved. +11. Update the finding and evidence directory. ## Evidence directory diff --git a/prompts/phase-4-validate.md b/prompts/phase-4-validate.md index 6da7cbfd..431801d6 100644 --- a/prompts/phase-4-validate.md +++ b/prompts/phase-4-validate.md @@ -37,6 +37,7 @@ Read the following files (all paths are relative to the project/workspace root): - the assigned finding - relevant source files under `src/` - sandbox documentation under `sandbox/` +- If present, `itemdb/notes/threat-model.md` — operational threat model from Phase 1b: assets, attacker capabilities and non-capabilities, trust-boundary summary, existing controls, abuse-path themes, risk calibration, and open assumptions. Use it to plan realistic validation attempts. Use additional target-specific skills only if they clearly apply. @@ -59,13 +60,19 @@ Do not create unrelated findings. 2. Extract the exact vulnerability claim. 3. Review the existing counter-analysis. 4. Identify what evidence would confirm or reject the claim. -5. Inspect relevant source files. -6. Prepare the sandbox under `sandbox/`. -7. Execute the validation plan or improve it. -8. Capture commands, inputs, outputs, logs, and observations. -9. Store evidence under `itemdb/evidence//`. -10. Update the finding. -11. Move the finding to the correct status directory if needed. +5. When available, consult `itemdb/notes/threat-model.md` before finalizing the validation approach: + - Attacker capabilities: validate against the described attacker profile. Do not design a validation that assumes capabilities explicitly excluded as non-capabilities. + - Trust boundaries: ensure the validation crosses documented trust boundaries. A finding that does not cross a documented boundary may still be valid but requires additional justification. + - Existing controls: identify controls that may block or narrow the validation path and factor them into the plan. + - Affected assets: map the finding's claimed impact to documented assets and security objectives. + - Open assumptions: note any threat-model assumptions that change exploitability or validation feasibility, and record them in the evidence. +6. Inspect relevant source files. +7. Prepare the sandbox under `sandbox/`. +8. Execute the validation plan or improve it. +9. Capture commands, inputs, outputs, logs, and observations. +10. Store evidence under `itemdb/evidence//`. +11. Update the finding. +12. Move the finding to the correct status directory if needed. ## Evidence requirements @@ -191,6 +198,7 @@ At the end, summarize: - result: CONFIRMED / REJECTED / UNRESOLVED, - evidence files created, - finding file moved or updated, +- threat-model assumptions that materially affected validation strategy or evidence interpretation, - open questions for the user (same content as in the run summary), - re-run prompt hints (same content as in the run summary; use `PROMPT_EXTRA` / `PROMPT_EXTRA_FILE` snippets), - remaining limitations. diff --git a/templates/evidence-readme.md b/templates/evidence-readme.md index b0d5deef..3b4e881c 100644 --- a/templates/evidence-readme.md +++ b/templates/evidence-readme.md @@ -36,6 +36,19 @@ Include: - relevant configuration, - date of validation. +# Threat-model assumptions (if applicable) + +When the validation result was materially affected by assumptions from the threat model, document them here. + +Examples: + +- Attacker capability constrained by threat-model non-capabilities +- Trust boundary documented in `itemdb/notes/threat-model.md` that shaped the validation path +- Existing control from the threat model that blocked or narrowed validation +- Open assumption from the threat model that affected exploitability assessment + +If the threat model did not affect the result, this section may be omitted. + # Commands executed List the exact commands executed. diff --git a/tests/test_phase_1_prompts_threat_model.py b/tests/test_prompts_threat_model.py similarity index 55% rename from tests/test_phase_1_prompts_threat_model.py rename to tests/test_prompts_threat_model.py index 01240bc3..eb341b07 100644 --- a/tests/test_phase_1_prompts_threat_model.py +++ b/tests/test_prompts_threat_model.py @@ -13,6 +13,12 @@ def _read_prompt(name: str) -> str: return path.read_text(encoding="utf-8") +def _read_opencode(path_from_root: str) -> str: + path = ROOT / path_from_root + assert path.is_file(), f"{path} does not exist" + return path.read_text(encoding="utf-8") + + def test_phase_1c_recon_prompt_exists() -> None: path = ROOT / "prompts" / "phase-1c-recon.md" assert path.is_file(), f"{path} does not exist" @@ -100,3 +106,71 @@ def test_phase_1_recon_md_removed() -> None: def test_phase_1b_codeql_recon_md_removed() -> None: path = ROOT / "prompts" / "phase-1b-codeql-recon.md" assert not path.exists(), f"{path} should have been renamed" + + + +# --------------------------------------------------------------------------- +# Phase 4 validator agent and skill — threat-model.md integration +# --------------------------------------------------------------------------- + +def test_validator_agent_references_threat_model() -> None: + content = _read_opencode(".opencode/agents/validator.md") + assert "itemdb/notes/threat-model.md" in content + + +def test_validator_agent_uses_conditional_language() -> None: + content = _read_opencode(".opencode/agents/validator.md") + content_lower = content.lower() + assert ( + "when available" in content_lower + or "when present" in content_lower + or "if present" in content_lower + ) + + +def test_exploit_validation_skill_references_threat_model() -> None: + content = _read_opencode(".opencode/skills/exploit-validation/SKILL.md") + assert "itemdb/notes/threat-model.md" in content + + +def test_exploit_validation_skill_mentions_attacker_capabilities() -> None: + content = _read_opencode(".opencode/skills/exploit-validation/SKILL.md") + content_lower = content.lower() + assert "attacker" in content_lower + assert "non-capabilities" in content_lower + + +# --------------------------------------------------------------------------- +# Phase 4 — threat-model.md integration +# --------------------------------------------------------------------------- + +def test_phase_4_explicitly_references_threat_model_when_present() -> None: + content = _read_prompt("phase-4-validate.md") + assert "itemdb/notes/threat-model.md" in content + + +def test_phase_4_uses_conditional_when_available_language() -> None: + content = _read_prompt("phase-4-validate.md") + content_lower = content.lower() + assert ( + "when this file is available" in content_lower + or "when available" in content_lower + or "when present" in content_lower + or "if present" in content_lower + ) + + +def test_phase_4_mentions_attacker_capabilities_and_non_capabilities() -> None: + content = _read_prompt("phase-4-validate.md") + assert "attacker capabilit" in content.lower() + assert "non-capabilities" in content.lower() + + +def test_phase_4_mentions_trust_boundaries_in_validation_context() -> None: + content = _read_prompt("phase-4-validate.md") + assert "trust boundar" in content.lower() + + +def test_phase_4_mentions_existing_controls() -> None: + content = _read_prompt("phase-4-validate.md") + assert "existing controls" in content.lower()