feat(privacy): Privacy & Secret Protection Engine

AGENTS.md

@@ -13,7 +13,7 @@ control plane, or runtime AI filter.
 
 - `src/Bower.Contracts`: immutable event contracts and serialization.
 - `src/Bower.PolicyEngine`: deterministic, explainable event selection.
-- `src/Bower.Redaction`: local data minimisation and intelligent sensitive-data detection before persistence.
+- `src/Bower.Redaction`: Privacy & Secret Protection Engine — modular detectors (AU identifiers, financial, secrets, crypto), policy actions, privacy metadata; redacts before persistence.
 - `src/Bower.Persistence`: SQLite queue, cursors, delivery state and evidence.
 - `src/Bower.Core`: source-to-policy-to-queue orchestration.
 - `src/Bower.Sdk`: semantic developer API and transports.

Bower.sln

@@ -51,6 +51,8 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bower.Dcr", "src\Bower.Dcr\
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bower.Agent.Aws", "src\Bower.Agent.Aws\Bower.Agent.Aws.csproj", "{A240528B-2792-4CEB-8A84-1CB773CA4BEC}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bower.Benchmarks", "tests\Bower.Benchmarks\Bower.Benchmarks.csproj", "{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -61,6 +63,18 @@ Global
 		Release|x86 = Release|x86
 	EndGlobalSection
 	GlobalSection(ProjectConfigurationPlatforms) = postSolution
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Debug|x64.ActiveCfg = Debug|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Debug|x64.Build.0 = Debug|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Debug|x86.Build.0 = Debug|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Release|Any CPU.Build.0 = Release|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Release|x64.ActiveCfg = Release|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Release|x64.Build.0 = Release|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Release|x86.ActiveCfg = Release|Any CPU
+		{B7E1C9A0-4D2F-4A8B-9E3C-1F6A2B8D0C4E}.Release|x86.Build.0 = Release|Any CPU
 		{1A4F182D-F153-4529-A20B-E0551C82734C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
 		{1A4F182D-F153-4529-A20B-E0551C82734C}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{1A4F182D-F153-4529-A20B-E0551C82734C}.Debug|x64.ActiveCfg = Debug|Any CPU

docs/architecture/overview.md

@@ -7,7 +7,10 @@ validation or policy decision.
 ## Trust boundaries
 
 1. Source records are untrusted, bounded input.
-2. Redaction removes known dangerous fields before typed parsing and persistence.
+2. Privacy & Secret Protection Engine (`Bower.Redaction`) removes field-name
+   secrets and scans string values for regulated AU identifiers, credentials and
+   crypto material before typed parsing and persistence. See
+   [privacy engine](../privacy/privacy-secret-engine.md).
 3. Policy Engine accepts only approved semantic types with required context.
 4. SQLite queue is tenant-controlled durable state; delivery leases recover after
    crashes and acknowledged rows remain auditable until retention.

docs/privacy/privacy-secret-engine.md

@@ -0,0 +1,161 @@
+# Privacy & Secret Protection Engine
+
+Bower's privacy engine inspects every event **after parse and before
+persistence / normalisation**. It is a core control for preventing accidental
+leakage of regulated Australian identifiers, credentials and cryptographic
+material into downstream SIEM platforms.
+
+```
+Raw Event → Parser → Privacy & Secret Engine → Normalisation → Output
+                         ├── Detect
+                         ├── Validate
+                         ├── Classify
+                         └── Apply Policy
+```
+
+## Design principles
+
+| Principle | How |
+| --- | --- |
+| Deterministic | Compiled regex, checksums, structure, entropy — **no AI at runtime** |
+| Streaming / per-event | Events processed independently; safe under concurrent callers |
+| Extensible | `ISensitiveDetector` plugins; provider API keys via pattern registry |
+| Configurable | Global default action + per-detector overrides |
+| Fail closed | Invalid / oversized payloads → redaction failure → quarantine |
+
+## Actions
+
+| Action | Effect |
+| --- | --- |
+| `Allow` | Leave value unchanged (still recorded in metadata if detected) |
+| `Remove` | Delete property (field-name) or replace span with empty |
+| `Replace` | Replace with configured placeholder |
+| `Mask` | Partial reveal (e.g. last 4 digits, email local-part) |
+| `Sha256` | `sha256:<hex>` |
+| `Hmac` | `hmac-sha256:<hex>` (requires 32+ byte key; else SHA-256) |
+| `Encrypt` | AES-GCM (`enc:aesgcm:…`); requires 32-byte key; else remove |
+| `AlertOnly` | Detect only; do not rewrite |
+
+## Default policy (production-oriented)
+
+| Detector | Default |
+| --- | --- |
+| Field-name secrets | Remove |
+| TFN | SHA-256 |
+| CRN / Medicare / IHI / Passport / Licence / DVA | Mask |
+| ABN / ACN | Allow |
+| Credit card | Remove |
+| JWT / cloud secrets / API keys / PEM / DB / env | Remove |
+| Email / phone / DOB | Mask |
+| Security markings | AlertOnly (opt-in) |
+
+Opt-in (disabled by default): IP, hostname, username, residential address, GPS,
+security markings.
+
+## Metadata
+
+When findings exist, the engine attaches (never with original values):
+
+```json
+{
+  "privacy": {
+    "detected": ["au.tfn", "secret.jwt", "id.email"],
+    "actions": {
+      "au.tfn": "SHA256",
+      "secret.jwt": "Removed",
+      "id.email": "Masked"
+    }
+  }
+}
+```
+
+`SecurityEventEnvelope.Privacy` maps the same shape.
+
+## Detector modules
+
+### Australian identifiers
+
+| Id | Validation |
+| --- | --- |
+| `au.tfn` | ATO 8/9-digit checksum |
+| `au.crn` | Pattern (9 digits + letter) |
+| `au.medicare` | Medicare checksum + issue digit |
+| `au.ihi` | `800360` + Luhn |
+| `au.passport` | Format + contextual label |
+| `au.driver-licence` | State-heuristic formats (context gated) |
+| `au.abn` | ABN mod-89 |
+| `au.acn` | ACN check digit |
+| `au.dva` | DVA pattern |
+
+### Financial
+
+`fin.credit-card` (Luhn + network), `fin.bsb-account`, `fin.iban` (mod-97),
+`fin.swift-bic`, `fin.payid`.
+
+### Identity
+
+`id.email`, `id.phone.au`, `id.phone.intl`, `id.dob`, plus optional
+`id.address`, `id.gps`, `id.ip`, `id.hostname`, `id.username`.
+
+### Secrets & crypto
+
+AWS, Azure, Entra, GCP, JWT, OAuth, provider API keys (OpenAI, Anthropic,
+GitHub, GitLab, Slack, Stripe, Twilio, Cloudflare, Atlassian, Datadog,
+PagerDuty, Okta, MongoDB, Snowflake), Kubernetes, Docker, database connection
+strings, environment variable assignments, PEM/PKCS8/SSH/PGP/X.509.
+
+### Classification
+
+`class.security-marking` — OFFICIAL, PROTECTED, SECRET, TOP SECRET,
+CABINET-IN-CONFIDENCE (opt-in).
+
+## Configuration (code)
+
+```csharp
+var defaults = PrivacyPolicy.CreateDefault();
+var policy = new PrivacyPolicy
+{
+    DefaultAction = defaults.DefaultAction,
+    DetectorActions = new Dictionary<string, PrivacyAction>(defaults.DetectorActions)
+    {
+        [DetectorIds.Tfn] = PrivacyAction.Hmac,
+        [DetectorIds.Email] = PrivacyAction.Mask,
+        [DetectorIds.Abn] = PrivacyAction.Allow
+    },
+    HmacKey = hmacKey32PlusBytes,
+    EnabledOptInDetectors = new HashSet<string> { DetectorIds.IpAddress },
+    EmitMetadata = true
+};
+
+IEventRedactor redactor = new JsonEventRedactor(policy);
+// or
+var engine = new PrivacyEngine(policy, customDetectors);
+```
+
+## Extension points
+
+1. Implement `ISensitiveDetector` (value scan) and/or `IFieldNameDetector`.
+2. Pass detectors into `PrivacyEngine` constructor **or** extend
+   `DetectorCatalog.CreateDefaultValueDetectors()`.
+3. Set per-id actions / disable flags on `PrivacyPolicy`.
+4. Future (not in this milestone): WASM plugins, org-specific learning,
+   AI-assisted policy recommendation (config-time only).
+
+## Performance
+
+- Per-event JSON walk; detectors only run on string leaves.
+- Overlapping matches resolved once (prefer longer, then earlier).
+- Replacements applied right-to-left.
+- Micro-benchmarks: `tests/Bower.Benchmarks` (Stopwatch; no extra packages).
+
+## Tests
+
+- Unit tests under `tests/Bower.UnitTests` cover checksums and major detectors.
+- Existing `JsonEventRedactor` / `SensitiveDataDetector` tests remain green via
+  compatibility facades.
+
+## Related
+
+- `src/Bower.Redaction/AGENTS.md` — contributor invariants
+- `src/Bower.Core/SecurityEventProcessor.cs` — redaction before enqueue
+}

src/Bower.Collector/Program.cs

@@ -34,7 +34,7 @@
     ?? Environment.MachineName;
 
 builder.Services.AddSingleton<IClock, SystemClock>();
-builder.Services.AddSingleton<IEventRedactor, JsonEventRedactor>();
+builder.Services.AddSingleton<IEventRedactor>(_ => new JsonEventRedactor());
 builder.Services.AddSingleton<IDurableEventStore>(services =>
     new SqliteEventStore(
         databasePath,

src/Bower.Contracts/SecurityEventEnvelope.cs

@@ -48,6 +48,20 @@ public sealed record SecurityEventEnvelope
     public IReadOnlyDictionary<string, string>? Labels { get; init; }
 
     public IReadOnlyDictionary<string, JsonElement>? Attributes { get; init; }
+
+    /// <summary>
+    /// Sanitisation summary produced by the Privacy &amp; Secret Protection Engine.
+    /// Never contains original sensitive values.
+    /// </summary>
+    public PrivacyContext? Privacy { get; init; }
+}
+
+/// <summary>Additive privacy metadata attached after detection and policy application.</summary>
+public sealed record PrivacyContext
+{
+    public required IReadOnlyList<string> Detected { get; init; }
+
+    public required IReadOnlyDictionary<string, string> Actions { get; init; }
 }
 
 public sealed record ApplicationContext

src/Bower.Redaction/AGENTS.md

@@ -0,0 +1,24 @@
+# Privacy & Secret Protection Engine
+
+Runtime redaction must stay **deterministic**. No AI, no network lookups, no
+arbitrary code from configuration.
+
+## Invariants
+
+- Redact **before** persistence. Treat redaction failure as a security failure.
+- Never log original secrets, credentials, tokens, or unrestricted bodies.
+- Privacy metadata may list detector ids and actions only — never original values.
+- Default-deny field-name secrets (password, token, body, headers, …).
+- Prefer checksum validation (TFN, ABN, ACN, Medicare, IHI, Luhn) to cut false positives.
+- New detectors implement `ISensitiveDetector` and register via `DetectorCatalog`
+  or constructor injection — do not hard-code detector lists inside `PrivacyEngine`.
+- Keep detectors independently enable/disable-able through `PrivacyPolicy`.
+- Opt-in detectors (IP, hostname, username, address, GPS, markings) stay off unless enabled.
+- Add unit tests for every new detector (positive + negative / invalid checksum).
+- Add or update micro-benchmarks when changing hot paths.
+
+## Layout
+
+- `Privacy/` — engine, policy, interfaces, catalog, applicator
+- `Validation/` — checksum algorithms
+- `Detectors/` — modular detector implementations by domain