Merge pull request #108 from dreadnode/cap-1168-add-ast-grep-to-web-security-runtime-and-integrate-with-jxscout-static-analysis-skills

feat(web-security): add ast-grep to runtime for AST-based JS static a…

Author: GangGreenTemperTatum

capabilities/web-security/capability.yaml

@@ -1,14 +1,15 @@
 schema: 1
 name: web-security
-version: "1.3.0"
+version: "1.4.0"
 description: >
   Web application penetration testing with 70+ attack technique playbooks
   covering request smuggling, cache poisoning, SSRF, SSTI, DOM
   vulnerabilities, authentication bypasses, parser differentials,
   AEM/Sling exploitation, GraphQL, OAuth, and client-side attacks.
   Includes HTTP client tooling with OOB callbacks via interactsh, Caido
   and Burp proxy integration via MCP, browser automation via
-  agent-browser, JS static analysis via jxscout, protobuf inspection via
+  agent-browser, JS static analysis via jxscout, AST-based code pattern
+  search via ast-grep, protobuf inspection via
   protoscope, credential management, DNS rebinding, blind SQLi
   extraction, bug bounty scope lookup via bbscope, HackerOne program
   recon and report submission, SecurityContext vulnerability context
@@ -160,6 +161,8 @@ checks:
     command: 'test -f "$HOME/git/archivealchemist/archive-alchemist.py"'
   - name: exiftool
     command: command -v exiftool
+  - name: ast-grep
+    command: command -v ast-grep
   - name: jxscout
     command: command -v jxscout-pro-v2
 
@@ -208,6 +211,7 @@ keywords:
   - browser-automation
   - burp-suite
   - jxscout
+  - ast-grep
   - interactsh
   - oob-callbacks
   - securitycontext

capabilities/web-security/docker/Dockerfile.runtime

@@ -22,6 +22,7 @@
 #   - kiterunner          (API-aware content discovery)
 #   - surf                (SSRF target identification)
 #   - pacu                (AWS exploitation framework)
+#   - ast-grep            (AST-based code pattern search via tree-sitter)
 #
 # Tools with bundled SDK/MCP integration (require a running instance
 # reachable by network — the client library and MCP server are included):
@@ -119,7 +120,8 @@ RUN pip install --no-cache-dir \
         "fastmcp>=2.0" \
         "httpx>=0.28" \
         "caido-sdk-client" \
-        "pacu"
+        "pacu" \
+        "ast-grep-cli"
 
 # ── fireprox (AWS API Gateway IP rotation) ──────────────────────────
 # Installed to a predictable path; requires AWS credentials at runtime.

capabilities/web-security/scripts/install_tools.sh

@@ -119,6 +119,11 @@ fi
 npm install -g agent-browser
 agent-browser install || true
 
+# -- ast-grep (AST-based code pattern search) ---------------------------------
+# Tree-sitter based structural code matching for JS/TS/HTML. Lightweight
+# alternative to semgrep for pattern matching (no taint analysis).
+pip install --break-system-packages ast-grep-cli
+
 # -- waymore (Wayback Machine recon) -----------------------------------------
 pip install --break-system-packages waymore
 

capabilities/web-security/skills/dom-vulnerability-static-analysis/SKILL.md

@@ -63,8 +63,45 @@ For DOMPurify: check version and config -- see `dompurify-mxss-bypass` skill.
 
 ## 4. AST-based analysis (large codebases)
 
-- **Semgrep**: `semgrep --config p/javascript` for DOM XSS rules
-- **CodeQL**: `javascript/ql/src/Security/CWE-079` for taint tracking
+`rg` patterns are fragile for code constructs — they miss variations in whitespace, nesting, and comments. Use AST-based tools for structural matching.
+
+### ast-grep (installed in runtime)
+
+ast-grep uses tree-sitter for structural code matching. Patterns use real code syntax with `$VAR` wildcards — far more accurate than regex for function calls, assignments, and nested expressions.
+
+```bash
+# innerHTML / outerHTML sinks — catches any receiver expression
+ast-grep -p '$EL.innerHTML = $VAR' -l js src/
+ast-grep -p '$EL.outerHTML = $VAR' -l js src/
+
+# insertAdjacentHTML
+ast-grep -p '$EL.insertAdjacentHTML($POS, $CONTENT)' -l js src/
+
+# eval-family sinks
+ast-grep -p 'eval($INPUT)' -l js src/
+ast-grep -p 'new Function($BODY)' -l js src/
+
+# URL assignment sinks
+ast-grep -p 'location.href = $VAR' -l js src/
+ast-grep -p 'location.assign($URL)' -l js src/
+ast-grep -p 'window.open($TARGET)' -l js src/
+
+# postMessage handlers (multi-line aware)
+ast-grep -p 'window.addEventListener("message", $HANDLER)' -l js src/
+
+# React dangerouslySetInnerHTML
+ast-grep -p 'dangerouslySetInnerHTML={{$VAR}}' -l tsx src/
+
+# jQuery .html() sink
+ast-grep -p '$EL.html($CONTENT)' -l js src/
+```
+
+Add `--json` to any command for structured output with file paths, line/column positions, and captured metavariable values. This is useful for programmatic triage or feeding results into jxscout custom analyzers.
+
+### Additional AST tools (if operator-installed)
+
+- **Semgrep**: `semgrep --config p/javascript` for DOM XSS rules (includes taint tracking in Pro)
+- **CodeQL**: `javascript/ql/src/Security/CWE-079` for data-flow taint analysis
 - **ESLint**: `no-unsanitized/property`, `no-unsanitized/method`
 
 ## 5. Report findings

capabilities/web-security/skills/jxscout-custom-analyzers/SKILL.md

@@ -20,12 +20,13 @@ The `JXSCOUT_PROJECT_NAME` environment variable must be set. The project setting
 
 ## Choosing the right analyzer type
 
-For JS/TS code pattern matching (function calls, assignments, data flow, control structures), **prefer AST-based analysis via script analyzers** (e.g. semgrep) over regex. Regex works for simple literal patterns -- hardcoded URLs, API keys, specific strings -- but is fragile for code constructs because it misses variations in whitespace, formatting, nesting, and comments.
+For JS/TS code pattern matching (function calls, assignments, data flow, control structures), **prefer AST-based analysis via script analyzers** over regex. Regex works for simple literal patterns -- hardcoded URLs, API keys, specific strings -- but is fragile for code constructs because it misses variations in whitespace, formatting, nesting, and comments.
 
 Rule of thumb:
 - **Regex**: simple string patterns that don't depend on code structure
 - **Derived**: narrowing down an existing match kind by value
-- **Script (semgrep)**: anything involving code semantics -- function calls, assignments, data flow, missing checks
+- **Script (ast-grep)**: anything involving code semantics -- function calls, assignments, data flow, missing checks. Installed in the runtime, use this by default.
+- **Script (semgrep)**: same use case as ast-grep, with additional taint tracking in semgrep Pro. Only available if operator-installed.
 
 ## Analyzer types
 
@@ -113,9 +114,87 @@ The script receives file paths on stdin (one per line) and must output a JSON ar
 
 `$JXSCOUT_PROJECT_DIR` and `$JXSCOUT_PROJECT_NAME` are available as environment variables in all scripts.
 
-## Using semgrep for JS analysis
+## Using ast-grep for JS analysis (installed in runtime)
 
-When semgrep is available and you need to find JS/TS code patterns, prefer it over regex. Semgrep uses AST-based matching which is far more accurate for code patterns (function calls, assignments, data flow) than regular expressions.
+ast-grep uses tree-sitter for AST-based structural code matching. Patterns use real code syntax with `$VAR` wildcards -- far more accurate than regex for function calls, assignments, and nested expressions. Prefer ast-grep over regex for script analyzers.
+
+### 1. Create the wrapper script
+
+Create `scripts/ast_grep_to_jxscout.sh` in your project directory. This script converts ast-grep JSON output to the jxscout match format:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+PATTERN=""
+KIND=""
+LANG="javascript"
+
+while [[ $# -gt 0 ]]; do
+  case $1 in
+    -p|--pattern) PATTERN="$2"; shift 2 ;;
+    -k|--kind) KIND="$2"; shift 2 ;;
+    -l|--lang) LANG="$2"; shift 2 ;;
+    *) echo "Usage: $0 -p|--pattern <pattern> -k|--kind <kind> [-l|--lang <lang>] (file paths on stdin)" >&2; exit 1 ;;
+  esac
+done
+
+[[ -n "$PATTERN" ]] || { echo "Missing -p|--pattern <pattern>" >&2; exit 1; }
+[[ -n "$KIND" ]] || { echo "Missing -k|--kind <kind>" >&2; exit 1; }
+
+FILES=()
+while IFS= read -r line || [[ -n "$line" ]]; do
+  [[ -n "$line" ]] && FILES+=("$line")
+done
+
+[[ ${#FILES[@]} -gt 0 ]] || { echo "No file paths on stdin" >&2; exit 1; }
+
+output=""
+for f in "${FILES[@]}"; do
+  AG_JSON=$(ast-grep -p "$PATTERN" -l "$LANG" --json "$f" 2>/dev/null || echo "[]")
+  while IFS= read -r result; do
+    one=$(echo "$result" | jq -c \
+      --arg kind "$KIND" \
+      '{kind: $kind, value: .text, start: {line: (.range.start.line + 1), column: .range.start.column}, end: {line: (.range.end.line + 1), column: .range.end.column}}')
+    output="${output}${one}"$'\n'
+  done < <(echo "$AG_JSON" | jq -c '.[]? // empty')
+done
+
+if [[ -z "$output" ]]; then
+  echo "[]"
+else
+  echo "$output" | jq -cs 'if . == null then [] else . end'
+fi
+```
+
+Make it executable: `chmod +x scripts/ast_grep_to_jxscout.sh`
+
+**Note:** ast-grep uses 0-based line numbers; the script adds 1 to align with jxscout's 1-based convention.
+
+### 2. Add to project settings
+
+```json
+{
+  "analyzer": {
+    "custom_analyzers": {
+      "innerHTML_sink": {
+        "enabled": true,
+        "type": "script",
+        "file_types": ["js", "reversed_source"],
+        "script": "$JXSCOUT_PROJECT_DIR/scripts/ast_grep_to_jxscout.sh --pattern '$EL.innerHTML = $VAR' --kind innerHTML_sink"
+      }
+    }
+  }
+}
+```
+
+### 3. Multiple patterns
+
+For analyzers that need several patterns (e.g. all eval-family sinks), create one analyzer per pattern with a shared match kind, or write a wrapper that runs ast-grep multiple times and merges the output.
+
+## Using semgrep for JS analysis (if operator-installed)
+
+When semgrep is available and you need taint tracking or data-flow analysis beyond structural matching, use it as a script analyzer. Semgrep is not installed in the runtime by default.
 
 ### 1. Create a semgrep rule
 
@@ -304,7 +383,7 @@ jxscout auto-reloads when `settings.jsonc` changes, so the view updates immediat
 
 ## Workflow for adding a custom analyzer
 
-1. Decide the analyzer type: regex for simple literal patterns, derived for filtering existing match kinds, script for code-semantic analysis (prefer semgrep for JS/TS when available)
+1. Decide the analyzer type: regex for simple literal patterns, derived for filtering existing match kinds, script for code-semantic analysis (prefer ast-grep for JS/TS structural matching; use semgrep if operator-installed and taint tracking is needed)
 2. Add the configuration to `settings.jsonc` under `analyzer > custom_analyzers`
 3. Test with `analyze` on a file that should match (create a temp test file if needed, then delete it)
 4. **Always** add the new match kind to the VS Code matches view -- **first ensure the full `vscode_extension` block is in `settings.jsonc`** (get it from `print-full-project-settings` if missing), then append your entry