diff --git a/AGENTS.md b/AGENTS.md index da35a1b..45e3277 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,7 +75,8 @@ Keep each lab narrow, local-file-based, reviewable, and easy to validate with sa ## Current release state -- Latest stable release: `v0.2.0` +- Latest stable release: `v0.3.0` - `v0.1.0`: first credible mini-lab, centered on `linux-auth-observe` - `v0.2.0`: second credible mini-lab, adding `linux-socket-observe` -- Unreleased: `linux-permission-observe`, the 408-to-security bridge, and `linux-process-observe` +- `v0.3.0`: 408-to-security bridge, `linux-permission-observe`, `linux-process-observe`, and the process-diff to telemetry-lab JSONL adapter +- Unreleased: follow-up hardening only; do not infer a fifth mini-lab from this release diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b20441..4328154 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,30 +6,29 @@ All notable changes to this project will be documented in this file. ### Added -- Added `projects/linux-permission-observe` for deterministic file mode/ownership, group, and sudoers drift evidence. -- Added `notes/408-to-linux-security.md` to map operating-system concepts to security evidence. -- Added `projects/linux-process-observe` for saved procfs identity, process/socket linking, normalized diffs, and Markdown reports. -- Added the `stacknil.system-evidence.v1` envelope contract for process evidence artifacts. -- Added `notes/process-evidence-schema.md` to document process identity and evidence caveats. -- Added `docs/reviewer-brief.md` as a short external-review entry point. -- Added `notes/network-state-to-detection-thinking.md` to connect socket state diffs with detection review questions. -- Added repository-level docs and notes index pages. -- Added related-notes links from each mini-lab README to its supporting notes. -- Added `.gitignore` rules for local Python test artifacts and generated mini-lab output files. - ### Changed -- Expanded repository navigation and validation commands for the permission and process mini-labs while keeping `v0.2.0` as the latest stable release. -- Updated repository agent guidance with permission/process input boundaries and explicit non-goals. -- Added changelog and docs-directory links to the root README and docs index. -- Added local validation commands to the root README. -- Updated the root README with a reviewer brief link. -- Updated repository agent guidance to reflect the current two-mini-lab scope and boundaries. - ### Fixed -- Tightened reviewer brief wording to avoid implying CI coverage where only local pytest coverage is currently documented. -- Made the reviewer brief quick-run path separator platform-neutral. +## [v0.3.0] - 2026-08-09 + +408-to-Security Bridge + +### Added + +- Introduced `projects/linux-permission-observe` for deterministic file mode/ownership, group, and sudoers drift evidence. +- Added `notes/408-to-linux-security.md` to map operating-system concepts to security evidence. +- Introduced `projects/linux-process-observe` for saved procfs identity, process/socket linking, normalized diffs, and Markdown reports. +- Added the `stacknil.system-evidence.v1` envelope contract for process evidence artifacts. +- Added a process-diff adapter that emits telemetry-lab-compatible JSONL with stable evidence metadata. +- Added `stacknil.system-evidence.telemetry.v1` to identify the adapter mapping contract separately from the source evidence contract. +- Added adapter golden regression, malformed input, PID fallback, and CLI coverage. + +### Documentation + +- Added `notes/process-evidence-schema.md` and the v0.3.0 release notes. +- Updated repository navigation and reviewer guidance for four stable mini-labs. +- Documented the process diff -> telemetry-lab JSONL bridge without adding a fifth mini-lab. ## [v0.2.0] - 2026-05-20 @@ -67,6 +66,7 @@ First Credible Mini-Lab - Added release notes in `docs/release-v0.1.0.md` - Added root README release entry and latest release link -[Unreleased]: https://github.com/stacknil/systems-foundations/compare/v0.2.0...HEAD +[Unreleased]: https://github.com/stacknil/systems-foundations/compare/v0.3.0...HEAD +[v0.3.0]: https://github.com/stacknil/systems-foundations/releases/tag/v0.3.0 [v0.2.0]: https://github.com/stacknil/systems-foundations/releases/tag/v0.2.0 [v0.1.0]: https://github.com/stacknil/systems-foundations/releases/tag/v0.1.0 diff --git a/README.md b/README.md index bf5664f..58d997e 100644 --- a/README.md +++ b/README.md @@ -10,11 +10,11 @@ The goal is to keep each lab narrow, deterministic, and easy to inspect end to e - [`projects/linux-auth-observe`](projects/linux-auth-observe/README.md): Linux auth evidence mini-lab for exported journald JSON lines and distro auth syslog files. This was the first stable mini-lab in `v0.1.0`. - [`projects/linux-socket-observe`](projects/linux-socket-observe/README.md): local Linux networking state mini-lab for `ss` plus selected `iproute2` snapshots. It builds one normalized snapshot artifact and generates a Markdown diff between two snapshots. -- [`projects/linux-permission-observe`](projects/linux-permission-observe/README.md): saved Linux permission state mini-lab for file ownership/mode, group membership, and sudoers drift. This work is currently unreleased. -- [`projects/linux-process-observe`](projects/linux-process-observe/README.md): saved procfs identity and `ss` context mini-lab that links processes to listening sockets and network endpoints. This work is currently unreleased. +- [`projects/linux-permission-observe`](projects/linux-permission-observe/README.md): saved Linux permission state mini-lab for file ownership/mode, group membership, and sudoers drift. Released in `v0.3.0`. +- [`projects/linux-process-observe`](projects/linux-process-observe/README.md): saved procfs identity and `ss` context mini-lab that links processes to listening sockets and network endpoints. It also adapts process diffs into telemetry-lab-compatible JSONL without adding another mini-lab. Released in `v0.3.0`. -Latest stable release: [v0.2.0](https://github.com/stacknil/systems-foundations/releases/latest) -Latest release notes: [v0.2.0](docs/release-v0.2.0.md) +Latest stable release: [v0.3.0](https://github.com/stacknil/systems-foundations/releases/latest) +Latest release notes: [v0.3.0](docs/release-v0.3.0.md) Changelog: [CHANGELOG.md](CHANGELOG.md) Docs index: [docs/README.md](docs/README.md) Reviewer brief: [docs/reviewer-brief.md](docs/reviewer-brief.md) @@ -45,6 +45,8 @@ cd ../linux-process-observe python -m pytest -q ``` +The process lab's optional cross-repository bridge is local-file based: `process_diff.json` -> `telemetry_events.jsonl` -> telemetry-lab's existing event/window workflow. + ## Repository Shape - `projects/`: focused mini-labs diff --git a/docs/README.md b/docs/README.md index 5fd69c0..2db3eeb 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,5 +9,6 @@ Repository-level documents for reviewers, releases, and project context. ## Release Notes - [Changelog](../CHANGELOG.md) +- [v0.3.0 release notes](release-v0.3.0.md) - [v0.2.0 release notes](release-v0.2.0.md) - [v0.1.0 release notes](release-v0.1.0.md) diff --git a/docs/release-v0.3.0.md b/docs/release-v0.3.0.md new file mode 100644 index 0000000..7f979f3 --- /dev/null +++ b/docs/release-v0.3.0.md @@ -0,0 +1,72 @@ +# v0.3.0 Release Notes + +## Title + +408-to-Security Bridge + +## Summary + +`systems-foundations` now has four small, deterministic Linux/systems foundations mini-labs. This release adds the permission and process evidence paths and connects the process diff artifact to the existing telemetry-lab event contract without adding a fifth mini-lab. + +The release remains local-file based and reviewable: + +- permission state becomes normalized evidence and a Markdown drift report +- saved procfs and `ss` context become process snapshots, socket links, a diff, and a report +- `process_diff.json` can be adapted into telemetry-lab-compatible JSONL events + +## Included In v0.3.0 + +- `projects/linux-permission-observe` +- `notes/408-to-linux-security.md` +- `projects/linux-process-observe` +- `notes/process-evidence-schema.md` +- the `stacknil.system-evidence.v1` evidence envelope +- the `linux-process-observe adapt` command +- adapter golden output and malformed-input coverage + +## Adapter Contract + +The adapter reads an existing `process_diff.json` and writes one JSON object per line with the required telemetry-lab fields: + +| system-evidence diff | telemetry-lab event | +| --- | --- | +| `observed_at` | `timestamp` | +| process `added/removed/modified` | `event_type=process_added/process_removed/process_modified` | +| socket-link `added/removed` | `event_type=socket_link_added/socket_link_removed` | +| `process_id` | `source` | +| executable or formatted socket endpoint | `target` | +| `added/removed/modified` | `status` | +| snapshot comparison observation semantics | `metadata.time_semantics=snapshot_diff_observed_at` | +| adapter mapping contract | `metadata.adapter_contract=stacknil.system-evidence.telemetry.v1` | + +Each event includes deterministic metadata with the source evidence schema and the versioned adapter mapping contract, plus source, host, record type, identity, record index, and field changes. An unlinked socket uses a deterministic `host_id:pid:` source fallback when a PID is available. + +`metadata.time_semantics` is `snapshot_diff_observed_at`. The event `timestamp` +is the diff observation time, not an inferred process-start, process-exit, or +socket-occurrence time. A window containing several adapter rows therefore +represents evidence deltas observed in one snapshot comparison; it does not +prove that those system activities happened together. + +Run the bridge with: + +```bash +python -m linux_process_observe adapt \ + --input output/diff/process_diff.json \ + --output output/diff/telemetry_events.jsonl +``` + +The output can be supplied as `input_path` to telemetry-lab's existing window workflow. That downstream repository owns its window, deduplication, and investigation demo artifacts; this release does not duplicate those workflows or change their schemas. + +## Validation Status + +- All four mini-lab pytest suites pass locally. +- `linux-process-observe` covers adapter golden output, malformed diff records, unlinked socket source fallback, and CLI error reporting. +- The adapter output satisfies telemetry-lab's required `timestamp`, `event_type`, `source`, `target`, and `status` event fields. + +## Non-Goals + +- no fifth mini-lab +- no live procfs crawling or real-time monitoring +- no `/proc/net/tcp` parsing, pcap, raw sockets, or packet sockets +- no auditd parser +- no database, network service, web UI, cloud dependency, or EDR agent behavior diff --git a/docs/reviewer-brief.md b/docs/reviewer-brief.md index 5472d8a..53b0329 100644 --- a/docs/reviewer-brief.md +++ b/docs/reviewer-brief.md @@ -12,13 +12,15 @@ Current stable labs: - `projects/linux-auth-observe` for normalizing Linux auth evidence, filtering it, and generating short Markdown summaries - `projects/linux-socket-observe` for turning saved `ss` and `iproute2` snapshots into normalized JSON and Markdown diffs +- `projects/linux-permission-observe` for turning saved file, group, and sudoers records into normalized permission drift artifacts +- `projects/linux-process-observe` for saved procfs identity, process/socket links, normalized diffs, and a telemetry-lab JSONL adapter ## Reviewer Evidence - Reproducible command: `python -m linux_auth_observe normalize --input tests/fixtures/ubuntu_auth.log --source auto --year 2026 --timezone Asia/Shanghai --output output/events.jsonl` -- Deterministic outputs: normalized auth JSONL, parse-error JSONL, auth summaries, socket snapshot JSON, and socket diff Markdown reports. -- Tests: local pytest coverage for parsers, filtering, summaries, CLI workflows, golden regression artifacts, and malformed input handling. -- Release evidence: versioned mini-lab release notes for `v0.1.0` and `v0.2.0`. +- Deterministic outputs: normalized auth JSONL, parse-error JSONL, auth summaries, socket snapshot JSON, socket diff Markdown reports, permission drift artifacts, process evidence envelopes, and telemetry-lab-compatible JSONL. +- Tests: local pytest coverage for parsers, filtering, summaries, CLI workflows, golden regression artifacts, malformed input handling, process/socket diffs, and adapter mapping. +- Release evidence: versioned mini-lab release notes for `v0.1.0`, `v0.2.0`, and `v0.3.0`. - Non-goals: live monitoring, packet capture, `/proc/net/tcp` parsing, `audit.log` support, databases, or offensive functionality. ## Quick run @@ -32,6 +34,10 @@ python -m linux_auth_observe summary --input output/events.jsonl --output output cd ../linux-socket-observe python -m pip install -e ".[dev]" python -m linux_socket_observe snapshot --ss tests/fixtures/baseline/ss.txt --ip-addr tests/fixtures/baseline/ip_addr.json --ip-link tests/fixtures/baseline/ip_link.json --ip-neigh tests/fixtures/baseline/ip_neigh.json --ip-link-stats tests/fixtures/baseline/ip_link_stats.txt --output output/baseline.json + +cd ../linux-process-observe +python -m pip install -e ".[dev]" +python -m linux_process_observe adapt --input tests/golden/diff/process_diff.json --output output/telemetry_events.jsonl ``` ## Sample output @@ -52,7 +58,7 @@ python -m linux_socket_observe snapshot --ss tests/fixtures/baseline/ss.txt --ip - Linux evidence normalization and schema discipline - CLI workflows that are deterministic and reviewer-friendly - the ability to turn low-level system state into stable artifacts -- foundations work that supports later telemetry and monitoring repos +- a bounded evidence bridge into the existing telemetry-lab event contract ## Safety / boundaries @@ -65,8 +71,8 @@ python -m linux_socket_observe snapshot --ss tests/fixtures/baseline/ss.txt --ip - supported input families are intentionally selective - `linux-auth-observe` does not cover `audit.log` - `linux-socket-observe` does not do live capture or traffic analysis -- labs are separate mini-projects, not one unified system +- labs are separate mini-projects; the process adapter is a file-format bridge, not a unified runtime ## Next milestone -Add the next small Linux or systems-state mini-lab while keeping the same evidence-first, reviewer-friendly shape. +Keep hardening the four existing labs and their evidence contracts without adding a fifth mini-lab by default. diff --git a/projects/linux-process-observe/README.md b/projects/linux-process-observe/README.md index 0084f47..32bbe94 100644 --- a/projects/linux-process-observe/README.md +++ b/projects/linux-process-observe/README.md @@ -31,6 +31,7 @@ proc/ | `process_socket_links.json` | Listening sockets and network endpoints linked to process evidence when possible | | `process_diff.json` | Added, removed, or modified processes plus added or removed process/socket links | | `report.md` | Reviewer-friendly Markdown summary of the normalized diff | +| `telemetry_events.jsonl` | telemetry-lab-compatible events adapted from `process_diff.json` | All JSON artifacts use the same envelope: @@ -73,6 +74,18 @@ python -m linux_process_observe diff \ --output-dir output/diff ``` +To bridge the diff into the existing telemetry-lab event contract: + +```bash +python -m linux_process_observe adapt \ + --input output/diff/process_diff.json \ + --output output/diff/telemetry_events.jsonl +``` + +The adapter output has the required `timestamp`, `event_type`, `source`, `target`, and `status` fields. A process change maps to `process_added`, `process_removed`, or `process_modified`; a process/socket link change maps to `socket_link_added` or `socket_link_removed`. The process ID is the event source, the executable or endpoint is the target, and the diff change type is the status. Each row also keeps deterministic evidence metadata for traceability. `metadata.adapter_contract` is `stacknil.system-evidence.telemetry.v1`, separate from the source `metadata.evidence_schema`; `metadata.time_semantics` is `snapshot_diff_observed_at`: `timestamp` is when the snapshot comparison was observed, not an inferred process or socket occurrence time. + +The JSONL can be supplied as `input_path` to telemetry-lab's existing `run window` configuration to produce its normal window features, alerts, summary, and run manifest. telemetry-lab's demo-specific deduplication and investigation workflows remain in that repository; this lab does not add a second copy of those commands or a fifth mini-lab. + ## Identity And Link Semantics - `process_id` is `host_id:pid:start_time_ticks`; PID alone is not treated as durable identity because Linux can reuse it. @@ -85,6 +98,7 @@ python -m linux_process_observe diff \ ## Validation Status Pytest covers procfs parsing, `ss` parsing, process identity, socket linking, malformed inputs, timezone normalization, golden artifacts, diffs, reports, and the CLI workflow. +The adapter adds golden JSONL coverage, an unlinked-socket source fallback test, malformed diff coverage, and CLI error reporting coverage. ## Non-Goals diff --git a/projects/linux-process-observe/src/linux_process_observe/adapters.py b/projects/linux-process-observe/src/linux_process_observe/adapters.py new file mode 100644 index 0000000..9e70f7c --- /dev/null +++ b/projects/linux-process-observe/src/linux_process_observe/adapters.py @@ -0,0 +1,154 @@ +from __future__ import annotations + +from pathlib import Path +from typing import Any +import json + +from .models import EVIDENCE_SCHEMA, EvidenceEnvelope + + +_CHANGE_TYPES = frozenset(("added", "removed", "modified")) +_EVENT_TYPES = { + "process_change": "process", + "process_socket_link_change": "socket_link", +} +ADAPTER_CONTRACT = "stacknil.system-evidence.telemetry.v1" +_TIME_SEMANTICS = "snapshot_diff_observed_at" +_SOCKET_FIELDS = ( + "protocol", + "state", + "local_address", + "local_port", + "remote_address", + "remote_port", +) + + +def build_telemetry_events(diff: EvidenceEnvelope) -> list[dict[str, Any]]: + """Map a process diff envelope to telemetry-lab-compatible JSONL records.""" + if diff.schema != EVIDENCE_SCHEMA: + raise ValueError(f"unsupported evidence schema: {diff.schema}") + if diff.source != "procfs+ss": + raise ValueError("process diff must use source=procfs+ss") + + return [ + _map_record(diff, record, index) + for index, record in enumerate(diff.records, start=1) + ] + + +def write_telemetry_events(events: list[dict[str, Any]], path: str | Path) -> None: + """Write deterministic one-event-per-line telemetry JSONL.""" + output_path = Path(path) + if output_path.exists() and output_path.is_dir(): + raise ValueError(f"output path is a directory: {output_path}") + output_path.parent.mkdir(parents=True, exist_ok=True) + with output_path.open("w", encoding="utf-8", newline="\n") as handle: + for event in events: + handle.write( + json.dumps( + event, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ) + + "\n" + ) + + +def _map_record( + diff: EvidenceEnvelope, + record: dict[str, Any], + index: int, +) -> dict[str, Any]: + record_type = _required_string(record, "record_type", index) + event_prefix = _EVENT_TYPES.get(record_type) + if event_prefix is None: + raise ValueError(f"record {index} has unsupported record_type={record_type}") + + change_type = _required_string(record, "change_type", index) + if change_type not in _CHANGE_TYPES: + raise ValueError(f"record {index} has unsupported change_type={change_type}") + + identity = _required_string(record, "identity", index) + selected = _selected_record(record, change_type, index) + if event_prefix == "process": + target = _required_string(selected, "executable", index) + else: + target = _socket_target(selected, index) + + source = _source_value(diff.host_id, selected) + changes = record.get("changes", {}) + if not isinstance(changes, dict): + raise ValueError(f"record {index} changes must be an object") + + return { + "timestamp": diff.observed_at, + "event_type": f"{event_prefix}_{change_type}", + "source": source, + "target": target, + "status": change_type, + "metadata": { + "adapter_contract": ADAPTER_CONTRACT, + "change_type": change_type, + "evidence_schema": diff.schema, + "evidence_source": diff.source, + "host_id": diff.host_id, + "identity": identity, + "process_id": selected.get("process_id"), + "record_index": index, + "record_type": record_type, + "time_semantics": _TIME_SEMANTICS, + "changes": changes, + }, + } + + +def _selected_record( + record: dict[str, Any], + change_type: str, + index: int, +) -> dict[str, Any]: + key = "before" if change_type == "removed" else "after" + selected = record.get(key) + if not isinstance(selected, dict): + raise ValueError(f"record {index} {change_type} change requires an object in {key}") + return selected + + +def _source_value(host_id: str, record: dict[str, Any]) -> str: + process_id = record.get("process_id") + if isinstance(process_id, str) and process_id.strip(): + return process_id + + pid = record.get("pid") + if isinstance(pid, int) and not isinstance(pid, bool): + return f"{host_id}:pid:{pid}" + return f"{host_id}:unlinked" + + +def _socket_target(record: dict[str, Any], index: int) -> str: + values = { + field: _required_value(record, field, index) for field in _SOCKET_FIELDS + } + return ( + f"{values['protocol']} {values['state']} " + f"{values['local_address']}:{values['local_port']} -> " + f"{values['remote_address']}:{values['remote_port']}" + ) + + +def _required_string(record: dict[str, Any], field: str, index: int) -> str: + value = record.get(field) + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"record {index} missing non-empty {field}") + return value.strip() + + +def _required_value(record: dict[str, Any], field: str, index: int) -> str | int: + value = record.get(field) + if isinstance(value, bool) or not isinstance(value, (int, str)): + raise ValueError(f"record {index} missing valid {field}") + if isinstance(value, str) and not value.strip(): + raise ValueError(f"record {index} missing valid {field}") + return value diff --git a/projects/linux-process-observe/src/linux_process_observe/cli.py b/projects/linux-process-observe/src/linux_process_observe/cli.py index 6873a83..2204253 100644 --- a/projects/linux-process-observe/src/linux_process_observe/cli.py +++ b/projects/linux-process-observe/src/linux_process_observe/cli.py @@ -5,6 +5,7 @@ from pathlib import Path import sys +from .adapters import build_telemetry_events, write_telemetry_events from .diff import build_diff_envelope from .report import build_markdown_report from .snapshot import EvidenceInputError, build_snapshot_artifacts, load_envelope, write_envelope @@ -29,6 +30,14 @@ def build_parser() -> ArgumentParser: diff_parser.add_argument("--after-links", required=True, help="later process_socket_links.json") diff_parser.add_argument("--output-dir", required=True, help="directory for process_diff.json and report.md") diff_parser.set_defaults(handler=_handle_diff) + + adapt_parser = subparsers.add_parser( + "adapt", + help="map process_diff.json to telemetry-lab-compatible JSONL events", + ) + adapt_parser.add_argument("--input", required=True, help="process_diff.json") + adapt_parser.add_argument("--output", required=True, help="output telemetry JSONL path") + adapt_parser.set_defaults(handler=_handle_adapt) return parser @@ -87,6 +96,36 @@ def _handle_diff(args: Namespace) -> int: return 0 +def _handle_adapt(args: Namespace) -> int: + try: + diff = load_envelope(args.input, input_name="process-diff") + events = build_telemetry_events(diff) + write_telemetry_events(events, args.output) + except EvidenceInputError as exc: + _print_error("adapt", exc) + return 1 + except OSError as exc: + _print_error( + "adapt", + EvidenceInputError("output", str(args.output), exc.__class__.__name__, str(exc)), + ) + return 1 + except ValueError as exc: + _print_error( + "adapt", + EvidenceInputError( + "process-diff", + str(args.input), + exc.__class__.__name__, + str(exc), + ), + ) + return 1 + + print(f"adapt wrote {len(events)} telemetry events", file=sys.stderr) + return 0 + + def _print_error(command: str, error: EvidenceInputError) -> None: print( f"error command={command} input={error.input_name} path={error.path} " diff --git a/projects/linux-process-observe/tests/golden/diff/telemetry_events.jsonl b/projects/linux-process-observe/tests/golden/diff/telemetry_events.jsonl new file mode 100644 index 0000000..0870bc2 --- /dev/null +++ b/projects/linux-process-observe/tests/golden/diff/telemetry_events.jsonl @@ -0,0 +1,7 @@ +{"event_type":"process_added","metadata":{"adapter_contract":"stacknil.system-evidence.telemetry.v1","change_type":"added","changes":{},"evidence_schema":"stacknil.system-evidence.v1","evidence_source":"procfs+ss","host_id":"lab-host","identity":"lab-host:330:3300","process_id":"lab-host:330:3300","record_index":1,"record_type":"process_change","time_semantics":"snapshot_diff_observed_at"},"source":"lab-host:330:3300","status":"added","target":"/usr/bin/python3.11","timestamp":"2026-07-05T00:05:00Z"} +{"event_type":"process_removed","metadata":{"adapter_contract":"stacknil.system-evidence.telemetry.v1","change_type":"removed","changes":{},"evidence_schema":"stacknil.system-evidence.v1","evidence_source":"procfs+ss","host_id":"lab-host","identity":"lab-host:220:2200","process_id":"lab-host:220:2200","record_index":2,"record_type":"process_change","time_semantics":"snapshot_diff_observed_at"},"source":"lab-host:220:2200","status":"removed","target":"/opt/example/bin/app-worker","timestamp":"2026-07-05T00:05:00Z"} +{"event_type":"process_modified","metadata":{"adapter_contract":"stacknil.system-evidence.telemetry.v1","change_type":"modified","changes":{"argv":{"after":["/usr/bin/dash","-c","sleep 30"],"before":["/opt/example/bin/task-runner","--once"]},"executable":{"after":"/usr/bin/dash","before":"/opt/example/bin/task-runner"},"name":{"after":"sh","before":"task-runner"}},"evidence_schema":"stacknil.system-evidence.v1","evidence_source":"procfs+ss","host_id":"lab-host","identity":"lab-host:230:2300","process_id":"lab-host:230:2300","record_index":3,"record_type":"process_change","time_semantics":"snapshot_diff_observed_at"},"source":"lab-host:230:2300","status":"modified","target":"/usr/bin/dash","timestamp":"2026-07-05T00:05:00Z"} +{"event_type":"socket_link_added","metadata":{"adapter_contract":"stacknil.system-evidence.telemetry.v1","change_type":"added","changes":{},"evidence_schema":"stacknil.system-evidence.v1","evidence_source":"procfs+ss","host_id":"lab-host","identity":"lab-host:120:1200|tcp|ESTAB|192.0.2.10:22|198.51.100.21:51000","process_id":"lab-host:120:1200","record_index":4,"record_type":"process_socket_link_change","time_semantics":"snapshot_diff_observed_at"},"source":"lab-host:120:1200","status":"added","target":"tcp ESTAB 192.0.2.10:22 -> 198.51.100.21:51000","timestamp":"2026-07-05T00:05:00Z"} +{"event_type":"socket_link_added","metadata":{"adapter_contract":"stacknil.system-evidence.telemetry.v1","change_type":"added","changes":{},"evidence_schema":"stacknil.system-evidence.v1","evidence_source":"procfs+ss","host_id":"lab-host","identity":"lab-host:330:3300|tcp|LISTEN|0.0.0.0:8080|0.0.0.0:*","process_id":"lab-host:330:3300","record_index":5,"record_type":"process_socket_link_change","time_semantics":"snapshot_diff_observed_at"},"source":"lab-host:330:3300","status":"added","target":"tcp LISTEN 0.0.0.0:8080 -> 0.0.0.0:*","timestamp":"2026-07-05T00:05:00Z"} +{"event_type":"socket_link_removed","metadata":{"adapter_contract":"stacknil.system-evidence.telemetry.v1","change_type":"removed","changes":{},"evidence_schema":"stacknil.system-evidence.v1","evidence_source":"procfs+ss","host_id":"lab-host","identity":"lab-host:120:1200|tcp|ESTAB|192.0.2.10:22|198.51.100.20:50000","process_id":"lab-host:120:1200","record_index":6,"record_type":"process_socket_link_change","time_semantics":"snapshot_diff_observed_at"},"source":"lab-host:120:1200","status":"removed","target":"tcp ESTAB 192.0.2.10:22 -> 198.51.100.20:50000","timestamp":"2026-07-05T00:05:00Z"} +{"event_type":"socket_link_removed","metadata":{"adapter_contract":"stacknil.system-evidence.telemetry.v1","change_type":"removed","changes":{},"evidence_schema":"stacknil.system-evidence.v1","evidence_source":"procfs+ss","host_id":"lab-host","identity":"lab-host:220:2200|tcp|LISTEN|127.0.0.1:9000|0.0.0.0:*","process_id":"lab-host:220:2200","record_index":7,"record_type":"process_socket_link_change","time_semantics":"snapshot_diff_observed_at"},"source":"lab-host:220:2200","status":"removed","target":"tcp LISTEN 127.0.0.1:9000 -> 0.0.0.0:*","timestamp":"2026-07-05T00:05:00Z"} diff --git a/projects/linux-process-observe/tests/test_adapter.py b/projects/linux-process-observe/tests/test_adapter.py new file mode 100644 index 0000000..c06a1a1 --- /dev/null +++ b/projects/linux-process-observe/tests/test_adapter.py @@ -0,0 +1,128 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from linux_process_observe.adapters import ADAPTER_CONTRACT, build_telemetry_events +from linux_process_observe.cli import main +from linux_process_observe.models import EvidenceEnvelope +from linux_process_observe.snapshot import load_envelope + + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +GOLDEN = PROJECT_ROOT / "tests" / "golden" + + +def test_process_diff_maps_to_stable_telemetry_events() -> None: + diff = load_envelope(GOLDEN / "diff" / "process_diff.json", input_name="process-diff") + + actual = build_telemetry_events(diff) + expected = _load_jsonl(GOLDEN / "diff" / "telemetry_events.jsonl") + + assert actual == expected + assert { + "timestamp", + "event_type", + "source", + "target", + "status", + } <= actual[0].keys() + assert {event["metadata"]["time_semantics"] for event in actual} == { + "snapshot_diff_observed_at" + } + assert {event["metadata"]["adapter_contract"] for event in actual} == { + ADAPTER_CONTRACT + } + assert {event["timestamp"] for event in actual} == {diff.observed_at} + + +def test_adapter_uses_pid_fallback_for_unlinked_socket_context() -> None: + diff = EvidenceEnvelope( + schema="stacknil.system-evidence.v1", + source="procfs+ss", + host_id="lab-host", + observed_at="2026-07-05T00:05:00Z", + records=[ + { + "record_type": "process_socket_link_change", + "change_type": "added", + "identity": "pid=68|udp|UNCONN|0.0.0.0:68|0.0.0.0:*", + "before": None, + "after": { + "record_type": "process_socket_link", + "linked": False, + "process_id": None, + "pid": 68, + "protocol": "udp", + "state": "UNCONN", + "local_address": "0.0.0.0", + "local_port": 68, + "remote_address": "0.0.0.0", + "remote_port": "*", + }, + "changes": {}, + } + ], + ) + + event = build_telemetry_events(diff)[0] + + assert event["source"] == "lab-host:pid:68" + assert event["target"] == "udp UNCONN 0.0.0.0:68 -> 0.0.0.0:*" + + +def test_adapter_rejects_malformed_diff_record() -> None: + payload = json.loads((GOLDEN / "diff" / "process_diff.json").read_text(encoding="utf-8")) + payload["records"][0]["after"].pop("executable") + diff = EvidenceEnvelope.from_mapping(payload) + + with pytest.raises(ValueError, match="record 1 missing non-empty executable"): + build_telemetry_events(diff) + + +def test_cli_adapt_writes_jsonl_and_reports_malformed_input(tmp_path: Path, capsys) -> None: + output_path = tmp_path / "telemetry_events.jsonl" + assert ( + main( + [ + "adapt", + "--input", + str(GOLDEN / "diff" / "process_diff.json"), + "--output", + str(output_path), + ] + ) + == 0 + ) + assert _load_jsonl(output_path) == _load_jsonl(GOLDEN / "diff" / "telemetry_events.jsonl") + assert "adapt wrote 7 telemetry events" in capsys.readouterr().err + + malformed_path = tmp_path / "malformed.json" + payload = json.loads((GOLDEN / "diff" / "process_diff.json").read_text(encoding="utf-8")) + payload["records"][0]["after"].pop("executable") + malformed_path.write_text(json.dumps(payload), encoding="utf-8") + assert ( + main( + [ + "adapt", + "--input", + str(malformed_path), + "--output", + str(tmp_path / "not-written.jsonl"), + ] + ) + == 1 + ) + captured = capsys.readouterr() + assert "error command=adapt input=process-diff" in captured.err + assert "type=ValueError" in captured.err + + +def _load_jsonl(path: Path) -> list[dict[str, object]]: + return [ + json.loads(line) + for line in path.read_text(encoding="utf-8").splitlines() + if line.strip() + ]