Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ Keep each lab narrow, local-file-based, reviewable, and easy to validate with sa

## Current release state

- Latest stable release: `v0.2.0`
- Latest stable release: `v0.3.0`
- `v0.1.0`: first credible mini-lab, centered on `linux-auth-observe`
- `v0.2.0`: second credible mini-lab, adding `linux-socket-observe`
- Unreleased: `linux-permission-observe`, the 408-to-security bridge, and `linux-process-observe`
- `v0.3.0`: 408-to-security bridge, `linux-permission-observe`, `linux-process-observe`, and the process-diff to telemetry-lab JSONL adapter
- Unreleased: follow-up hardening only; do not infer a fifth mini-lab from this release
42 changes: 21 additions & 21 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,30 +6,29 @@ All notable changes to this project will be documented in this file.

### Added

- Added `projects/linux-permission-observe` for deterministic file mode/ownership, group, and sudoers drift evidence.
- Added `notes/408-to-linux-security.md` to map operating-system concepts to security evidence.
- Added `projects/linux-process-observe` for saved procfs identity, process/socket linking, normalized diffs, and Markdown reports.
- Added the `stacknil.system-evidence.v1` envelope contract for process evidence artifacts.
- Added `notes/process-evidence-schema.md` to document process identity and evidence caveats.
- Added `docs/reviewer-brief.md` as a short external-review entry point.
- Added `notes/network-state-to-detection-thinking.md` to connect socket state diffs with detection review questions.
- Added repository-level docs and notes index pages.
- Added related-notes links from each mini-lab README to its supporting notes.
- Added `.gitignore` rules for local Python test artifacts and generated mini-lab output files.

### Changed

- Expanded repository navigation and validation commands for the permission and process mini-labs while keeping `v0.2.0` as the latest stable release.
- Updated repository agent guidance with permission/process input boundaries and explicit non-goals.
- Added changelog and docs-directory links to the root README and docs index.
- Added local validation commands to the root README.
- Updated the root README with a reviewer brief link.
- Updated repository agent guidance to reflect the current two-mini-lab scope and boundaries.

### Fixed

- Tightened reviewer brief wording to avoid implying CI coverage where only local pytest coverage is currently documented.
- Made the reviewer brief quick-run path separator platform-neutral.
## [v0.3.0] - 2026-08-09

408-to-Security Bridge

### Added

- Introduced `projects/linux-permission-observe` for deterministic file mode/ownership, group, and sudoers drift evidence.
- Added `notes/408-to-linux-security.md` to map operating-system concepts to security evidence.
- Introduced `projects/linux-process-observe` for saved procfs identity, process/socket linking, normalized diffs, and Markdown reports.
- Added the `stacknil.system-evidence.v1` envelope contract for process evidence artifacts.
- Added a process-diff adapter that emits telemetry-lab-compatible JSONL with stable evidence metadata.
- Added `stacknil.system-evidence.telemetry.v1` to identify the adapter mapping contract separately from the source evidence contract.
- Added adapter golden regression, malformed input, PID fallback, and CLI coverage.

### Documentation

- Added `notes/process-evidence-schema.md` and the v0.3.0 release notes.
- Updated repository navigation and reviewer guidance for four stable mini-labs.
- Documented the process diff -> telemetry-lab JSONL bridge without adding a fifth mini-lab.

## [v0.2.0] - 2026-05-20

Expand Down Expand Up @@ -67,6 +66,7 @@ First Credible Mini-Lab
- Added release notes in `docs/release-v0.1.0.md`
- Added root README release entry and latest release link

[Unreleased]: https://github.com/stacknil/systems-foundations/compare/v0.2.0...HEAD
[Unreleased]: https://github.com/stacknil/systems-foundations/compare/v0.3.0...HEAD
[v0.3.0]: https://github.com/stacknil/systems-foundations/releases/tag/v0.3.0
[v0.2.0]: https://github.com/stacknil/systems-foundations/releases/tag/v0.2.0
[v0.1.0]: https://github.com/stacknil/systems-foundations/releases/tag/v0.1.0
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ The goal is to keep each lab narrow, deterministic, and easy to inspect end to e

- [`projects/linux-auth-observe`](projects/linux-auth-observe/README.md): Linux auth evidence mini-lab for exported journald JSON lines and distro auth syslog files. This was the first stable mini-lab in `v0.1.0`.
- [`projects/linux-socket-observe`](projects/linux-socket-observe/README.md): local Linux networking state mini-lab for `ss` plus selected `iproute2` snapshots. It builds one normalized snapshot artifact and generates a Markdown diff between two snapshots.
- [`projects/linux-permission-observe`](projects/linux-permission-observe/README.md): saved Linux permission state mini-lab for file ownership/mode, group membership, and sudoers drift. This work is currently unreleased.
- [`projects/linux-process-observe`](projects/linux-process-observe/README.md): saved procfs identity and `ss` context mini-lab that links processes to listening sockets and network endpoints. This work is currently unreleased.
- [`projects/linux-permission-observe`](projects/linux-permission-observe/README.md): saved Linux permission state mini-lab for file ownership/mode, group membership, and sudoers drift. Released in `v0.3.0`.
- [`projects/linux-process-observe`](projects/linux-process-observe/README.md): saved procfs identity and `ss` context mini-lab that links processes to listening sockets and network endpoints. It also adapts process diffs into telemetry-lab-compatible JSONL without adding another mini-lab. Released in `v0.3.0`.

Latest stable release: [v0.2.0](https://github.com/stacknil/systems-foundations/releases/latest)
Latest release notes: [v0.2.0](docs/release-v0.2.0.md)
Latest stable release: [v0.3.0](https://github.com/stacknil/systems-foundations/releases/latest)
Latest release notes: [v0.3.0](docs/release-v0.3.0.md)
Changelog: [CHANGELOG.md](CHANGELOG.md)
Docs index: [docs/README.md](docs/README.md)
Reviewer brief: [docs/reviewer-brief.md](docs/reviewer-brief.md)
Expand Down Expand Up @@ -45,6 +45,8 @@ cd ../linux-process-observe
python -m pytest -q
```

The process lab's optional cross-repository bridge is local-file based: `process_diff.json` -> `telemetry_events.jsonl` -> telemetry-lab's existing event/window workflow.

## Repository Shape

- `projects/`: focused mini-labs
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,6 @@ Repository-level documents for reviewers, releases, and project context.
## Release Notes

- [Changelog](../CHANGELOG.md)
- [v0.3.0 release notes](release-v0.3.0.md)
- [v0.2.0 release notes](release-v0.2.0.md)
- [v0.1.0 release notes](release-v0.1.0.md)
72 changes: 72 additions & 0 deletions docs/release-v0.3.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# v0.3.0 Release Notes

## Title

408-to-Security Bridge

## Summary

`systems-foundations` now has four small, deterministic Linux/systems foundations mini-labs. This release adds the permission and process evidence paths and connects the process diff artifact to the existing telemetry-lab event contract without adding a fifth mini-lab.

The release remains local-file based and reviewable:

- permission state becomes normalized evidence and a Markdown drift report
- saved procfs and `ss` context become process snapshots, socket links, a diff, and a report
- `process_diff.json` can be adapted into telemetry-lab-compatible JSONL events

## Included In v0.3.0

- `projects/linux-permission-observe`
- `notes/408-to-linux-security.md`
- `projects/linux-process-observe`
- `notes/process-evidence-schema.md`
- the `stacknil.system-evidence.v1` evidence envelope
- the `linux-process-observe adapt` command
- adapter golden output and malformed-input coverage

## Adapter Contract

The adapter reads an existing `process_diff.json` and writes one JSON object per line with the required telemetry-lab fields:

| system-evidence diff | telemetry-lab event |
| --- | --- |
| `observed_at` | `timestamp` |
| process `added/removed/modified` | `event_type=process_added/process_removed/process_modified` |
| socket-link `added/removed` | `event_type=socket_link_added/socket_link_removed` |
| `process_id` | `source` |
| executable or formatted socket endpoint | `target` |
| `added/removed/modified` | `status` |
| snapshot comparison observation semantics | `metadata.time_semantics=snapshot_diff_observed_at` |
| adapter mapping contract | `metadata.adapter_contract=stacknil.system-evidence.telemetry.v1` |

Each event includes deterministic metadata with the source evidence schema and the versioned adapter mapping contract, plus source, host, record type, identity, record index, and field changes. An unlinked socket uses a deterministic `host_id:pid:<pid>` source fallback when a PID is available.

`metadata.time_semantics` is `snapshot_diff_observed_at`. The event `timestamp`
is the diff observation time, not an inferred process-start, process-exit, or
socket-occurrence time. A window containing several adapter rows therefore
represents evidence deltas observed in one snapshot comparison; it does not
prove that those system activities happened together.

Run the bridge with:

```bash
python -m linux_process_observe adapt \
--input output/diff/process_diff.json \
--output output/diff/telemetry_events.jsonl
```

The output can be supplied as `input_path` to telemetry-lab's existing window workflow. That downstream repository owns its window, deduplication, and investigation demo artifacts; this release does not duplicate those workflows or change their schemas.

## Validation Status

- All four mini-lab pytest suites pass locally.
- `linux-process-observe` covers adapter golden output, malformed diff records, unlinked socket source fallback, and CLI error reporting.
- The adapter output satisfies telemetry-lab's required `timestamp`, `event_type`, `source`, `target`, and `status` event fields.

## Non-Goals

- no fifth mini-lab
- no live procfs crawling or real-time monitoring
- no `/proc/net/tcp` parsing, pcap, raw sockets, or packet sockets
- no auditd parser
- no database, network service, web UI, cloud dependency, or EDR agent behavior
18 changes: 12 additions & 6 deletions docs/reviewer-brief.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,15 @@ Current stable labs:

- `projects/linux-auth-observe` for normalizing Linux auth evidence, filtering it, and generating short Markdown summaries
- `projects/linux-socket-observe` for turning saved `ss` and `iproute2` snapshots into normalized JSON and Markdown diffs
- `projects/linux-permission-observe` for turning saved file, group, and sudoers records into normalized permission drift artifacts
- `projects/linux-process-observe` for saved procfs identity, process/socket links, normalized diffs, and a telemetry-lab JSONL adapter

## Reviewer Evidence

- Reproducible command: `python -m linux_auth_observe normalize --input tests/fixtures/ubuntu_auth.log --source auto --year 2026 --timezone Asia/Shanghai --output output/events.jsonl`
- Deterministic outputs: normalized auth JSONL, parse-error JSONL, auth summaries, socket snapshot JSON, and socket diff Markdown reports.
- Tests: local pytest coverage for parsers, filtering, summaries, CLI workflows, golden regression artifacts, and malformed input handling.
- Release evidence: versioned mini-lab release notes for `v0.1.0` and `v0.2.0`.
- Deterministic outputs: normalized auth JSONL, parse-error JSONL, auth summaries, socket snapshot JSON, socket diff Markdown reports, permission drift artifacts, process evidence envelopes, and telemetry-lab-compatible JSONL.
- Tests: local pytest coverage for parsers, filtering, summaries, CLI workflows, golden regression artifacts, malformed input handling, process/socket diffs, and adapter mapping.
- Release evidence: versioned mini-lab release notes for `v0.1.0`, `v0.2.0`, and `v0.3.0`.
- Non-goals: live monitoring, packet capture, `/proc/net/tcp` parsing, `audit.log` support, databases, or offensive functionality.

## Quick run
Expand All @@ -32,6 +34,10 @@ python -m linux_auth_observe summary --input output/events.jsonl --output output
cd ../linux-socket-observe
python -m pip install -e ".[dev]"
python -m linux_socket_observe snapshot --ss tests/fixtures/baseline/ss.txt --ip-addr tests/fixtures/baseline/ip_addr.json --ip-link tests/fixtures/baseline/ip_link.json --ip-neigh tests/fixtures/baseline/ip_neigh.json --ip-link-stats tests/fixtures/baseline/ip_link_stats.txt --output output/baseline.json

cd ../linux-process-observe
python -m pip install -e ".[dev]"
python -m linux_process_observe adapt --input tests/golden/diff/process_diff.json --output output/telemetry_events.jsonl
```

## Sample output
Expand All @@ -52,7 +58,7 @@ python -m linux_socket_observe snapshot --ss tests/fixtures/baseline/ss.txt --ip
- Linux evidence normalization and schema discipline
- CLI workflows that are deterministic and reviewer-friendly
- the ability to turn low-level system state into stable artifacts
- foundations work that supports later telemetry and monitoring repos
- a bounded evidence bridge into the existing telemetry-lab event contract

## Safety / boundaries

Expand All @@ -65,8 +71,8 @@ python -m linux_socket_observe snapshot --ss tests/fixtures/baseline/ss.txt --ip
- supported input families are intentionally selective
- `linux-auth-observe` does not cover `audit.log`
- `linux-socket-observe` does not do live capture or traffic analysis
- labs are separate mini-projects, not one unified system
- labs are separate mini-projects; the process adapter is a file-format bridge, not a unified runtime

## Next milestone

Add the next small Linux or systems-state mini-lab while keeping the same evidence-first, reviewer-friendly shape.
Keep hardening the four existing labs and their evidence contracts without adding a fifth mini-lab by default.
14 changes: 14 additions & 0 deletions projects/linux-process-observe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ proc/
| `process_socket_links.json` | Listening sockets and network endpoints linked to process evidence when possible |
| `process_diff.json` | Added, removed, or modified processes plus added or removed process/socket links |
| `report.md` | Reviewer-friendly Markdown summary of the normalized diff |
| `telemetry_events.jsonl` | telemetry-lab-compatible events adapted from `process_diff.json` |

All JSON artifacts use the same envelope:

Expand Down Expand Up @@ -73,6 +74,18 @@ python -m linux_process_observe diff \
--output-dir output/diff
```

To bridge the diff into the existing telemetry-lab event contract:

```bash
python -m linux_process_observe adapt \
--input output/diff/process_diff.json \
--output output/diff/telemetry_events.jsonl
```

The adapter output has the required `timestamp`, `event_type`, `source`, `target`, and `status` fields. A process change maps to `process_added`, `process_removed`, or `process_modified`; a process/socket link change maps to `socket_link_added` or `socket_link_removed`. The process ID is the event source, the executable or endpoint is the target, and the diff change type is the status. Each row also keeps deterministic evidence metadata for traceability. `metadata.adapter_contract` is `stacknil.system-evidence.telemetry.v1`, separate from the source `metadata.evidence_schema`; `metadata.time_semantics` is `snapshot_diff_observed_at`: `timestamp` is when the snapshot comparison was observed, not an inferred process or socket occurrence time.

The JSONL can be supplied as `input_path` to telemetry-lab's existing `run window` configuration to produce its normal window features, alerts, summary, and run manifest. telemetry-lab's demo-specific deduplication and investigation workflows remain in that repository; this lab does not add a second copy of those commands or a fifth mini-lab.

## Identity And Link Semantics

- `process_id` is `host_id:pid:start_time_ticks`; PID alone is not treated as durable identity because Linux can reuse it.
Expand All @@ -85,6 +98,7 @@ python -m linux_process_observe diff \
## Validation Status

Pytest covers procfs parsing, `ss` parsing, process identity, socket linking, malformed inputs, timezone normalization, golden artifacts, diffs, reports, and the CLI workflow.
The adapter adds golden JSONL coverage, an unlinked-socket source fallback test, malformed diff coverage, and CLI error reporting coverage.

## Non-Goals

Expand Down
Loading