From b355f380627e7a3d4f9db039eeb11aeb692fb2f1 Mon Sep 17 00:00:00 2001 From: Martin Zeman Date: Sun, 24 May 2026 12:51:31 +0200 Subject: [PATCH] =?UTF-8?q?feat:=20LocalModel=20skill=20=E2=80=94=20privac?= =?UTF-8?q?y-preserving=20local=20LLM=20classifier?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- skills/LocalModel/SKILL.md | 70 +++++++ skills/LocalModel/scripts/classify.py | 194 ++++++++++++++++++++ skills/LocalModel/scripts/detect-backend.sh | 31 ++++ 3 files changed, 295 insertions(+) create mode 100644 skills/LocalModel/SKILL.md create mode 100755 skills/LocalModel/scripts/classify.py create mode 100755 skills/LocalModel/scripts/detect-backend.sh diff --git a/skills/LocalModel/SKILL.md b/skills/LocalModel/SKILL.md new file mode 100644 index 0000000..ff642d7 --- /dev/null +++ b/skills/LocalModel/SKILL.md @@ -0,0 +1,70 @@ +--- +name: LocalModel +version: 0.1.0 +description: "Run prompts against a local LLM backend (LM Studio, Ollama, or MLX) with privacy-preserving output guardrails — drop reasoning_content, enforce structured output schemas, and post-redact identifiers. USE WHEN sending sensitive content to a model, classifying private data on-machine, scanning dotfiles or configs for leaks via local LLM, content audit that must not touch cloud APIs." +sources: + - https://lmstudio.ai/docs/local-server + - https://github.com/ollama/ollama/blob/main/docs/api.md + - https://github.com/ml-explore/mlx-examples/tree/main/llms/mlx_lm +--- + +# LocalModel + +Invoke a locally hosted LLM (LM Studio, Ollama, or MLX) with structured output and three layers of leak prevention. All three backends speak the OpenAI-compatible `POST /v1/chat/completions` interface; the skill provides a single wrapper that probes for whichever is running and applies the same safety rails to every backend. + +The skill exists because **chain-of-thought reasoning models leak in `reasoning_content`** even when the prompt forbids reproducing values. Nemotron, DeepSeek-R1, QwQ, and similar models emit their working notes through a separate response field that is not subject to the same constraints as the final answer. Naive wrappers that fall back to `reasoning_content` when `content` is empty (e.g. when the token budget is exhausted mid-thought) print raw secrets directly to logs. + +## Workflow Routing + +| Trigger | Companion / script | +| ------------------------------------------------------------------------ | ---------------------------------------------------------- | +| User asks to scan, classify, or audit sensitive content via a local LLM | [scripts/classify.py](scripts/classify.py) | +| User asks which local backend is running, or wants to probe availability | [scripts/detect-backend.sh](scripts/detect-backend.sh) | +| User asks for raw inference (no privacy constraints) | Use `classify.py --no-redact --raw` flag | + +## Backends + +| Backend | Default port | Auth | Endpoint | +| --------- | -----------: | ----------------------------- | ------------------------- | +| LM Studio | 1234 | Bearer token (`LMSTUDIO_API_KEY` in `.env`) | `/v1/chat/completions` | +| Ollama | 11434 | none | `/v1/chat/completions` (OpenAI-compat) or `/api/chat` (native) | +| MLX | 8080 | none (default `mlx_lm.server`)| `/v1/chat/completions` | + +`scripts/detect-backend.sh` probes each port in that order and returns the first responding backend's identifier on stdout. + +## Three layers of leak prevention + +**Layer 1 — prompt-side classification, not reproduction.** The system prompt asks the model for *categories* + counts, never raw values. Output schema is enforced: a markdown table with `Category | Count | Risk`. Free-text answers are rejected by the wrapper. + +**Layer 2 — drop `reasoning_content` entirely.** The wrapper reads only `choices[0].message.content`. If `content` is empty (token budget exhausted before the final answer emitted), the wrapper reports `[scan failed: model produced no final content]` and discards `reasoning_content` unread. Falling back to reasoning is forbidden because chain-of-thought emits raw values mid-thought regardless of prompt constraints. + +**Layer 3 — post-redaction regex sweep on the model's output.** Even with the first two layers, the wrapper runs a final pass that masks: + +- IPv4 literals: `\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b` → `` +- IPv6 literals: `\b([0-9a-fA-F]{1,4}:){2,7}[0-9a-fA-F]{1,4}\b` → `` +- Email-like: `\b[\w.+-]+@[\w-]+\.[\w.-]+\b` → `` +- FQDN-like (any TLD ≥ 2 chars): `\b[a-z0-9][\w-]*\.[a-z]{2,}(?:\.[a-z]{2,})*\b` → `` +- Short identifiers with at least one digit (a common machine-name pattern): `\b[a-z]{2,4}-?\d+[a-z0-9-]*\b` → ``. The leading-letter cap and required digit avoid over-matching English hyphenated words like `keep-private`. + +## When NOT to use this skill + +Structural audits (counting how many `Host`, `HostName`, `IdentityFile`, `ProxyJump` directives exist in an SSH config) are deterministic grep work. The LLM adds no signal and creates leak risk. Prefer: + +```sh +for directive in Host HostName User Port IdentityFile ProxyJump ProxyCommand Match Include; do + count=$(grep -ciE "^\s*${directive}\s" "$file") + [ "$count" -gt 0 ] && printf " %-16s %s\n" "$directive" "$count" +done +``` + +The LLM earns its place when the question is semantic ("does this file expose infrastructure topology?", "is this comment hint a leak vector?") and the answer can be expressed as a category + count. + +## Constraints + +- **Never read `reasoning_content`** from the response. Treat it as if the field does not exist. If `content` is empty, the scan failed; do not patch with reasoning. +- **Never echo raw model output to chat** before running it through the Layer 3 redaction pass. +- **Always set a tight `max_tokens` budget** when the goal is a fixed-schema reply (e.g. 200 for a category table). High budgets give reasoning models more room to leak. +- **Reject free-text responses.** If the model output does not match the structured schema declared in the prompt, treat it as a failed scan and do not display the raw text. +- **Auth via env var, not flag.** `LMSTUDIO_API_KEY` reads from `~/.env` or the process environment. Never pass keys as positional arguments visible in `ps aux`. +- **Backend detection is deterministic.** `detect-backend.sh` probes ports in a fixed order (LM Studio 1234, Ollama 11434, MLX 8080) and stops at the first responder. No DNS, no env-var fallback that could be spoofed. +- **Tiny models may emit nothing useful.** Reasoning models with ≤ 8B parameters often spend the entire token budget on chain-of-thought and produce empty `content`. Prefer non-reasoning models (e.g. `qwen2.5-7b-instruct`, `gpt-oss-20b`) for classification tasks. diff --git a/skills/LocalModel/scripts/classify.py b/skills/LocalModel/scripts/classify.py new file mode 100755 index 0000000..b7920d6 --- /dev/null +++ b/skills/LocalModel/scripts/classify.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +"""Send a file or stdin to a local LLM for classification with three layers +of leak prevention. See sibling SKILL.md for the safety rationale. + +Layer 1: prompt-side schema (categories + counts only, never raw values) +Layer 2: drop reasoning_content; if content is empty, report failed scan +Layer 3: regex post-redaction on the model's output + +Usage: + classify.py # default classification prompt + classify.py --prompt PROMPT # custom system prompt + classify.py --model MODEL # override auto-selected model + classify.py --backend lmstudio|ollama|mlx # override auto-detect + classify.py --raw # skip Layer 3 redaction (caller asserts the prompt is safe) + cat file | classify.py # read from stdin +""" + +import argparse +import json +import os +import pathlib +import re +import subprocess +import sys +import urllib.request +import urllib.error + + +BACKENDS = { + "lmstudio": {"url": "http://localhost:1234/v1/chat/completions", "auth_env": "LMSTUDIO_API_KEY"}, + "ollama": {"url": "http://localhost:11434/v1/chat/completions", "auth_env": None}, + "mlx": {"url": "http://localhost:8080/v1/chat/completions", "auth_env": None}, +} + + +DEFAULT_SYSTEM = """You are a security auditor. Given a file fragment, enumerate the CATEGORIES of sensitive identifiers it contains. + +Output rules (MANDATORY): +- Reply with ONLY a markdown table. No prose before or after. +- Columns: Category | Count | Risk (low|medium|high) +- NEVER echo any raw values: no hostnames, no IPs, no usernames, no port numbers, no key fingerprints, no path fragments, no comments. +- Categories to look for: Host aliases, HostName values, IP addresses, User identifiers, Port numbers, IdentityFile paths, ProxyJump targets, custom commands, comments hinting at topology. +- Add one final row: `Recommendation | | n/a`. +""" + + +def load_env_var(name: str) -> str | None: + """Read VAR from process env, falling back to ~/.env.""" + val = os.environ.get(name) + if val: + return val + env_path = pathlib.Path.home() / ".env" + if not env_path.exists(): + return None + for line in env_path.read_text().splitlines(): + line = line.strip() + if line.startswith(f"{name}="): + return line.split("=", 1)[1].strip().strip('"').strip("'") + return None + + +def detect_backend() -> str: + """Run the sibling detect-backend.sh and return its stdout.""" + script = pathlib.Path(__file__).parent / "detect-backend.sh" + result = subprocess.run([str(script)], capture_output=True, text=True) + if result.returncode != 0: + sys.exit("no local backend responding (LM Studio:1234, Ollama:11434, MLX:8080 all down)") + return result.stdout.strip() + + +def pick_model(backend: str) -> str: + """Return a sensible default model for the backend by querying its /v1/models.""" + base = BACKENDS[backend]["url"].rsplit("/chat/completions", 1)[0] + "/models" + headers = {} + if BACKENDS[backend]["auth_env"]: + key = load_env_var(BACKENDS[backend]["auth_env"]) + if key: + headers["Authorization"] = f"Bearer {key}" + req = urllib.request.Request(base, headers=headers) + try: + with urllib.request.urlopen(req, timeout=5) as resp: + body = json.loads(resp.read()) + except urllib.error.URLError as exc: + sys.exit(f"failed to query {base}: {exc}") + models = [m["id"] for m in body.get("data", [])] + if not models: + sys.exit(f"backend {backend} reports no loaded models") + # Prefer non-reasoning models for classification (reasoning models leak in CoT). + # Heuristic: ids containing 'instruct', 'chat', or 'gpt-oss' first. + for keyword in ("instruct", "chat", "gpt-oss"): + for m in models: + if keyword in m.lower(): + return m + return models[0] + + +def call_model(backend: str, model: str, system: str, user: str, max_tokens: int = 400) -> str: + """POST to the backend's chat completions endpoint. Return content only — + NEVER reasoning_content, per Layer 2 of the leak-prevention design.""" + payload = { + "model": model, + "messages": [ + {"role": "system", "content": system}, + {"role": "user", "content": user}, + ], + "temperature": 0, + "max_tokens": max_tokens, + } + headers = {"Content-Type": "application/json"} + if BACKENDS[backend]["auth_env"]: + key = load_env_var(BACKENDS[backend]["auth_env"]) + if key: + headers["Authorization"] = f"Bearer {key}" + req = urllib.request.Request( + BACKENDS[backend]["url"], + data=json.dumps(payload).encode(), + headers=headers, + ) + try: + with urllib.request.urlopen(req, timeout=300) as resp: + body = json.loads(resp.read()) + except urllib.error.HTTPError as exc: + detail = exc.read().decode(errors="replace")[:500] + sys.exit(f"backend returned {exc.code}: {detail}") + except urllib.error.URLError as exc: + sys.exit(f"backend unreachable: {exc}") + msg = body["choices"][0]["message"] + content = msg.get("content") or "" + # Layer 2: do NOT fall back to reasoning_content. If content is empty, + # the scan failed and reasoning is discarded unread. + if not content.strip(): + return "[scan failed: model produced no final content; reasoning_content discarded unread]" + return content + + +REDACT_PATTERNS = [ + (re.compile(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b"), ""), + (re.compile(r"\b(?:[0-9a-fA-F]{1,4}:){2,7}[0-9a-fA-F]{1,4}\b"), ""), + (re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"), ""), + # FQDN: at least one dot, lowercase TLD ≥ 2 chars + (re.compile(r"\b[a-z0-9][\w-]*\.[a-z]{2,}(?:\.[a-z]{2,})*\b", re.IGNORECASE), ""), + # Short machine-name pattern: 2-4 lowercase letters, optional hyphen, then + # at least one digit (so hyphenated English words like "keep-private" do + # not over-match). + (re.compile(r"\b[a-z]{2,4}-?\d+[a-z0-9-]*\b"), ""), +] + + +def redact(text: str) -> str: + """Layer 3: mask anything that looks like a leaked identifier.""" + for pattern, replacement in REDACT_PATTERNS: + text = pattern.sub(replacement, text) + return text + + +def main(): + parser = argparse.ArgumentParser(description="Local-LLM classifier with leak prevention") + parser.add_argument("file", nargs="?", help="file to classify (omit to read stdin)") + parser.add_argument("--prompt", help="custom system prompt (default: schema-based categorization)") + parser.add_argument("--backend", choices=list(BACKENDS), help="force a backend (default: auto-detect)") + parser.add_argument("--model", help="override model id (default: backend's first non-reasoning model)") + parser.add_argument("--max-tokens", type=int, default=400) + parser.add_argument("--raw", action="store_true", help="skip Layer 3 redaction (caller asserts safety)") + args = parser.parse_args() + + if args.file: + content = pathlib.Path(args.file).read_text() + filename = pathlib.Path(args.file).name + else: + content = sys.stdin.read() + filename = "" + + backend = args.backend or detect_backend() + model = args.model or pick_model(backend) + system = args.prompt or DEFAULT_SYSTEM + + reply = call_model( + backend=backend, + model=model, + system=system, + user=f"File: {filename}\n\n```\n{content}\n```", + max_tokens=args.max_tokens, + ) + if not args.raw: + reply = redact(reply) + + print(f"backend: {backend}") + print(f"model: {model}") + print() + print(reply) + + +if __name__ == "__main__": + main() diff --git a/skills/LocalModel/scripts/detect-backend.sh b/skills/LocalModel/scripts/detect-backend.sh new file mode 100755 index 0000000..fbf93a7 --- /dev/null +++ b/skills/LocalModel/scripts/detect-backend.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Probe local LLM backends in deterministic order. Print the first responder's +# identifier ("lmstudio" | "ollama" | "mlx") to stdout, exit 0. If none respond, +# print "none" and exit 2. +# +# Usage: detect-backend.sh +# +# Authentication: LM Studio requires a Bearer token. The probe sends without +# auth and accepts HTTP 401 as a "backend reachable" signal (the auth check +# itself proves the server is up). Ollama and MLX accept no-auth probes. + +set -u + +probe() { + local name="$1" url="$2" + local code + code=$(curl -sSo /dev/null -w '%{http_code}' --max-time 2 "$url" 2>/dev/null || echo "000") + case "$code" in + # 200: model list returned; 401: auth required, server up (LM Studio); + # 404: endpoint exists at a different path but server up (some Ollama versions). + 200|401|404) echo "$name"; return 0 ;; + *) return 1 ;; + esac +} + +probe lmstudio "http://localhost:1234/v1/models" && exit 0 +probe ollama "http://localhost:11434/v1/models" && exit 0 +probe mlx "http://localhost:8080/v1/models" && exit 0 + +echo "none" +exit 2