Skip to content

feat(pilot): design-partner pilot program — framework, kit, ragctl pilot, case study (Step 7.4) #174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
officialCodeWork merged 1 commit into from
Jun 9, 2026
Merged

feat(pilot): design-partner pilot program — framework, kit, ragctl pilot, case study (Step 7.4) #174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 16 additions & 5 deletions TRACKER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,14 @@

| | |
|---|---|
| **Last updated** | 2026-06-08 |
| **Current phase** | Phase 7 — Pilot, Harden, GA (**3 / 10 steps**) — Phases 0–6 ✅ complete |
| **Overall** | **77 / 84 steps** — Phases 0–6 complete |
| **Next action** | **Step 7.4 — Design-partner pilots**: onboard 2–3 design partners across distinct verticals; per-pilot success criteria + weekly KPIs; case study documented. |
| **Last updated** | 2026-06-10 |
| **Current phase** | Phase 7 — Pilot, Harden, GA (**4 / 10 steps**) — Phases 0–6 ✅ complete |
| **Overall** | **78 / 84 steps** — Phases 0–6 complete |
| **Next action** | **Step 7.5 — Documentation site**: Docusaurus / MkDocs site; API reference generated from OpenAPI; quickstart + concept + how-to + ops guides; versioned docs; link-check + spell-check in CI. |

**Recently shipped**

- **7.4** ✅ Design-partner pilots — the pilot program shipped as a **repo-backed machine** (signing the referenceable partners is the external GTM deliverable, like the 7.3 pentest): an operator runbook + five fill-in templates (`docs/pilots/`), a **customer-support/KB kit** (`docs/pilots/customer-support/` — sample corpus with synthetic PII + a planted injection probe; domain-calibrated criteria), and **`ragctl pilot`** (`onboard` renders the per-tenant config + checklist; `report` assembles a weekly-KPI dashboard + **PASS/FAIL** verdict from the platform's own feedback / drift / cost signal components — the same ones behind `GET /v1/status/*`). KPIs are **pulled, not self-reported**. A worked case study runs it end-to-end on the kit with real `ragctl pilot report` output (satisfaction **+0.733**, **0/5** drift, cost **ok** → **PASS**) + the PII-redaction + injection-blocked security demo. No `dist`/SPI/config change. [#174](https://github.com/officialCodeWork/AgentContextOS/pull/174)
- **7.3** ✅ Red-team / security — turns the governance stack into an **adversarial probe gate** across four classes (prompt injection / PII egress / ACL bypass / tenant escape) under a `redteam` marker + a first-class **`redteam-gate`** CI job (`task redteam`). Closes the verified injection gap: new **`rag-injection`** package — a pluggable `InjectionDetector` (dependency-free `HeuristicInjectionDetector`, regexes anchored on attack *grammar* so benign prose isn't flagged) + `PromptInjectionGuard.inspect` that drops hijack chunks **before the LLM**, paired with `INJECTION_RESISTANT_SYSTEM_PROMPT` + `build_user_message` so untrusted context is **fenced data in the user turn, never a system-trust position** (fixes the OpenAI-chat surface that put context in a `system` message); wired on `/v1/query` + `/v1/chat/completions` + MCP; off by default (`cfg.injection`); PII-free `injection.blocked` event. Deterministic gate: a ≥ 500 known + ≥ 500 generated corpus (`eval/redteam_v0/`) hits **96.6 % block** (≥ 95 % bar) at **0 false positives**, plus the no-system-position invariant end-to-end — building the corpus *hardened the detector* (~20 missed phrasings). New PII-egress probe over `PiiPolicyEngine` (zero leakage, second-detector verified); `pip-audit` is the CVE gate; the external pentest is a documented process item. Injection types stay internal (no attacker signal, `dist/schemas`/`openapi` untouched). [#166](https://github.com/officialCodeWork/AgentContextOS/pull/166)
- **7.2** ✅ Chaos engineering — a deterministic in-process **kill-matrix gate** (`eval/gateway_chaos_v0/kill_matrix.py` + `tests/perf/test_chaos_kill_matrix.py`) that extends 7.1 from the three retrieval backends to the **full hot-path set** (vector/keyword/graph/embedder/retrieval_cache/reranker/llm): kill each backend in turn (100% unavailable) behind the real breakers + fallback, drive `/v1/query`, and assert **no single failure 5xx-es the gateway** (no 5xx, on-path retrieval breaker opens, expected degraded shape; a seeded keyword corpus + real `hydrate` make rerank/generate actually run). **Chaos fixed what it found** — the matrix exposed that a down **retrieval cache** or **reranker** 5xx-ed, so the gateway gained two minimal **degrade-open** guards (`gateway.cache.degraded` → miss on `/v1/query` + `/v1/retrieve`; `gateway.rerank.degraded` → retrieval-only, honouring `RerankPipeline`'s "caller decides" contract); LLM + embedder already degraded. **LitmusChaos cluster manifests** (`infra/chaos/`: gateway pod-delete + backend `pod-network-loss`/`latency` with httpProbe acceptance) as the cluster runbook + `task chaos-kill`. No `dist/`/SPI/config change. [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165)
- **7.1** ✅ Load + chaos testing — a **chaos-under-load** CI gate (`eval/gateway_chaos_v0/`): drives the in-process gateway under concurrent load while injecting backend faults (`FaultSpec` + `Chaos{Vector,Keyword,Graph}RetrievalBackend` SPI wrappers behind real breakers) and asserts **graceful degradation** — no 5xx, 100% success, the failing backend's breaker opens (validating the Phase-4 breakers + fallback, builds no new resilience); `tests/perf/test_chaos_under_load.py` (timing-independent, `perf`-marked) + `task chaos-test`. An extended **Locust v1 suite** (weighted read/write mix + varied queries + a ramp `LoadTestShape`) + documented **acceptance targets** (≥ 1000 RPS sustained, e2e p99 < 500 ms) as a cluster runbook — [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164)
Expand Down Expand Up @@ -804,7 +805,7 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en
| 7.1 | Load testing | ✅ | [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164) — chaos-under-load CI gate (`eval/gateway_chaos_v0` fault backends + harness; `tests/perf/test_chaos_under_load.py`; graceful degradation: no 5xx + breaker opens); Locust v1 suite (varied-query mix + ramp shape); acceptance targets + runbook |
| 7.2 | Chaos engineering | ✅ | [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165) — in-process **kill-matrix** gate (kill each hot-path backend → no 5xx, breaker opens, expected degraded shape); two new gateway **degrade-open** guards (retrieval cache + reranker); LitmusChaos cluster manifests (`infra/chaos/`) + `task chaos-kill` |
| 7.3 | Red-team | ✅ | [#166](https://github.com/officialCodeWork/AgentContextOS/pull/166) — adversarial probe gate (injection / PII / ACL / tenant-escape) under a `redteam` marker + `redteam-gate` CI job; new **`rag-injection`** guard (≥ 95 % block, no untrusted chunk in a system-trust position) wired on every answer surface; PII-egress probe; `task redteam` |
| 7.4 | Design partner onboarding | ⏳ | 2–3 design partners; feedback incorporated; case study documented |
| 7.4 | Design partner onboarding | ✅ | [#174](https://github.com/officialCodeWork/AgentContextOS/pull/174) — repo-backed pilot program: operator runbook + 5 templates (`docs/pilots/`) + ADR-0046; **customer-support/KB kit** (corpus w/ PII + planted injection probe; domain criteria); **`ragctl pilot`** `onboard` + `report` (weekly KPIs + PASS/FAIL verdict from feedback/drift/cost signal components, KPIs pulled not self-reported); worked case study (real `pilot report` → PASS). Signing partners = external GTM item |
| 7.5 | Documentation site | ⏳ | Docusaurus / MkDocs site; API reference generated from OpenAPI; quickstart guides |
| 7.6 | Marketplace listings | ⏳ | AWS / Azure / GCP Marketplace AMI / Helm listings |
| 7.7 | Packaging & distribution | ⏳ | PyPI publish (`rag-platform`), npm publish (`@ragplatform/sdk`), Docker Hub images |
Expand Down Expand Up @@ -839,6 +840,15 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en
- **PII-egress probe** (`test_pii_egress.py`): synthetic high-PII corpora through the 6.5 `PiiPolicyEngine` — `block` denies, `redact`/`mask` strip every span (verified by an *independent* second detector), over answer-text + `list[Chunk]` shapes
- **Scope:** the injection guard + the four-class probe gate + `task redteam` + the `redteam-gate` CI job. Dependency-scan CVE gating is the existing `pip-audit` `audit` job; the **external pentest** is a documented human-process item (`guides/red-team.md`), not code. Injection types are package-local + internal (no attacker signal on the wire; `dist/schemas`/`openapi` untouched). **Deferred:** a real ML injection classifier behind the seam, per-tenant pattern tuning, nightly staging runs. ~23 new tests (12 package-local + 11 red-team) on top of the existing suite; all gates green (ruff, mypy --strict 329 files, RAG001, policy-coverage, schema/openapi-drift, log-schema/event-registry). [ADR-0045](docs/adr/ADR-0045-red-team-security.md), [guides/red-team.md](docs/guides/red-team.md), [reference/injection.md](docs/reference/injection.md)

### 7.4 — Design-partner pilots ✅ [#174](https://github.com/officialCodeWork/AgentContextOS/pull/174)

- Step 7.4 is a **GTM/process step made checkable**: signing 3 referenceable design partners is an external human deliverable (like the 7.3 external pentest), so the repo ships the **machine** — a repeatable program whose KPIs are **pulled from the platform's own signals**, not self-reported. Delivered as one PR sliced 7.4a–d.
- **7.4a — framework.** The operator runbook ([guides/design-partner-pilots.md](docs/guides/design-partner-pilots.md)): pilot lifecycle (qualify → onboard → run → review → graduate), roles, onboarding-as-configured-deployment, the KPI→signal mapping, the intake → triage → incorporate → close feedback loop, acceptance (signed criteria green ≥ 3 consecutive weeks). Five fill-in templates under `docs/pilots/templates/` (onboarding-checklist / success-criteria / weekly-kpi / feedback-log / case-study). [ADR-0046](docs/adr/ADR-0046-design-partner-pilots.md).
- **7.4b — customer-support/KB kit.** `docs/pilots/customer-support/` — a 3-doc sample corpus calibrated to exercise each criterion: an HR handbook carrying **synthetic PII** (egress demo), a product FAQ (grounded answers), and a security overview hiding a **planted prompt-injection payload** (the 7.3 guard must drop it). Domain-calibrated success criteria (faithfulness 0.92, a deflection KPI); framework stays vertical-extensible (legal / engineering kits deferred).
- **7.4c — `ragctl pilot` tooling.** `pilot onboard` renders the per-tenant `rag.yaml` block + onboarding checklist; `pilot report` seeds a representative stream into the platform's own signal components (`rag_feedback` aggregate / `rag_drift` registry / `rag_observability.CostTracker` — the same ones behind `GET /v1/status/{feedback,drift,cost}`) and prints the weekly-KPI dashboard + a **PASS/FAIL verdict** (satisfaction ≥ +0.5, zero drifting monitors, cost not elevated); quality + latency cross-ref `ragctl eval` / `perf`. Seed-then-report (no infra), the same shape as `ragctl feedback` / `drift`. 3 new ragctl tests. [reference/pilot.md](docs/reference/pilot.md).
- **7.4d — worked case study.** [docs/pilots/customer-support/case-study.md](docs/pilots/customer-support/case-study.md) runs the framework end-to-end on the kit with **real `ragctl pilot report` output** (satisfaction **+0.733**, **0/5** monitors drifting, cost **ok** → **PASS**) + the PII-redaction + injection-blocked security demonstration — framed honestly as a representative pilot (a real partner substitutes their corpus + users).
- **Scope:** the framework + the first vertical kit + the `ragctl pilot` reader + the worked case study. **No new package / core or wire type / governed SPI call / `dist` change** — `pilot` is a pure `ragctl` reader over existing signal components, so the policy-coverage + schema/openapi-drift gates need no new entry. **Deferred / external:** sourcing + signing the real referenceable partners; the legal + internal-engineering kits; per-pilot golden-set *generation* tooling; a pilot KPI card in the admin console. All gates green (ruff, mypy --strict, RAG001, link-check). [ADR-0046](docs/adr/ADR-0046-design-partner-pilots.md), [guides/design-partner-pilots.md](docs/guides/design-partner-pilots.md), [reference/pilot.md](docs/reference/pilot.md).

---

## PR & Branch History
Expand Down Expand Up @@ -986,6 +996,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else
| [#158](https://github.com/officialCodeWork/AgentContextOS/pull/158) | 2026-06-08 | feat(crypto): zero-downtime key rotation — RotatingKeyManager (Step 6.7d) |
| [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165) | 2026-06-08 | test(perf): chaos kill-matrix gate + cache/rerank degrade-open + LitmusChaos (Step 7.2) |
| [#166](https://github.com/officialCodeWork/AgentContextOS/pull/166) | 2026-06-08 | feat(security): prompt-injection guard (rag-injection) + red-team probe gate (Step 7.3) |
| [#174](https://github.com/officialCodeWork/AgentContextOS/pull/174) | 2026-06-10 | feat(pilot): design-partner pilot program — framework + customer-support kit + ragctl pilot + case study (Step 7.4) |
| #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge |
| #81 | Closed | Dependabot bump — superseded |

Expand Down
Loading
Loading