diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..37bc316 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,218 @@ +# Changelog + +Protocol versions are recorded here with the changed rule, the observation that motivated it, and what remains hypothesis (`protocol/06` §9). Repository changes that do not change the protocol are not listed. + +## Unreleased + +- Research references checked against publisher and index records, with unconfirmed fields marked; review record, change-proposal disposition, hypothesis register and analogy records published (see `docs/review/`, `research/`). + +--- + +## v0.2 — from v0.1 + +The section below is the change record written when v0.2 was produced, kept as written. "This package" refers to the private working package the repository was derived from; see [`docs/provenance.md`](docs/provenance.md). + +### Why v0.2 exists + +v0.1 was intentionally packaged once the conceptual discovery had reached a coherent baseline. + +Two later inputs materially changed that baseline: + +1. a full meta-retrospective of the session that created v0.1; +2. focused research on Shared Conceptual Language and AI terminology anchoring. + +The changes below are therefore treated as a genuine protocol revision, not an appendix-only documentation update. + +--- + +### 1. New core principle — Progressive Commitment + +v0.1 already distinguished: + +`PROBE ≠ CANDIDATE ≠ RECOMMENDATION ≠ DECISION ≠ COMMITMENT` + +v0.2 generalizes the underlying mechanism: + +> **AI may externalize tentative structure early, but the protocol must preserve the distinction between a useful proposal and an accepted model commitment.** + +This applies not only to solutions. + +It also applies to: + +- problem frames, +- conceptual models, +- terminology, +- interpretations, +- research-derived hypotheses, +- recommendations, +- downstream decisions. + +Examples: + +```text +Candidate term → Working term → Canonical-for-now + +Solution probe → Candidate → Recommendation → Decision → Commitment + +Working frame → Accepted frame for current decision horizon +``` + +--- + +### 2. Shared Language becomes first-class + +New normative concern: + +> **Shared conceptual language should be negotiated progressively, not silently normalized by AI.** + +v0.2 adds `protocol/04_SHARED_LANGUAGE_POLICY.md`. + +Key changes: + +- human source meaning may need to be preserved; +- AI-generated model-bearing terms begin as proposals; +- unresolved consequential ambiguity should trigger semantic clarification; +- candidate terminology can be used as a semantic probe; +- important terms should be tested in actual scenarios/use before stabilization; +- "canonical" means `canonical-for-now-and-context`; +- terms may be reopened, split, merged, renamed or deprecated; +- no mandatory full glossary or separate terminology phase is introduced. + +--- + +### 3. `Meaning before naming` → `Meaning before commitment` + +The research challenged a strict rule that AI must delay naming until a concept is already fully understood. + +Early labels can help articulation, memory, distinction-making and reasoning. + +The revised rule is: + +> **Early naming is allowed. Early implicit model commitment is not.** + +--- + +### 4. Human Contribution Model retained, with one refinement + +The five current human contributions remain: + +1. Situated Grounding +2. Framing & Meaning +3. Independent Position / Authorship +4. Judgment & Exceptions +5. Commitment & Accountability + +Shared Language is **not** a sixth contribution. + +It is a cross-cutting coordination mechanism connecting these contributions to shared state and later handoff. + +`Independent Position / Authorship` now explicitly includes **framing-bearing terminology** where AI naming could erase a useful human distinction. + +--- + +### 5. State model expanded + +The working state may now maintain lightweight `concepts` / `shared_language` entries for model-relevant language. + +Minimum candidate fields: + +- current term, +- meaning, +- status, +- context, +- optional human source expression, +- example/boundary, +- optional provenance, +- open issue, +- meaningful rename history. + +This is intentionally selective. + +The trigger is not "new word". + +The trigger is: + +> **Would a different interpretation change requirements, decisions, boundaries, responsibilities, priorities or handoff meaning?** + +--- + +### 6. New interaction distinction — structure for exploration vs convergence + +Evidence Case 02 showed that AI structure has two different roles: + +#### Structure for exploration +- provisional, +- explicitly criticizable, +- exposes assumptions, +- invites counterexamples, +- can be replaced cheaply. + +#### Structure for convergence +- consolidates evidence, +- compares alternatives, +- stabilizes current decisions, +- can become a reference baseline. + +The protocol should not render an exploratory structure with the rhetorical certainty of a converged model. + +--- + +### 7. New Analogy Boundary Check + +Evidence Case 02 showed a failure mode when the prior `Turn` model was initially mapped too directly onto Discovery. + +Whenever a prior framework, architecture, product or methodology is imported as precedent, v0.2 now requires a lightweight check: + +1. What original problem did it solve? +2. What is genuinely similar? +3. What important assumptions differ? +4. What mechanisms transfer? +5. What must explicitly not transfer? + +--- + +### 8. Research now produces Model Delta + +After focused research, the discovery should not stop at "here is what the literature says". + +Record what changed: + +- `REINFORCED` +- `REFINED` +- `CHALLENGED` +- `NEW` +- `NO CHANGE` + +Also record: +- what became more uncertain; +- what decision became possible; +- what hypothesis should remain experimental. + +--- + +### 9. Final stable/public baseline needs whole-package adversarial review + +v0.1 was packaged after focused research but without one independent adversarial review of the whole protocol. + +v0.2 adds this as a pre-release recommendation: + +- contradiction search, +- evidence/overclaiming review, +- missing-boundary review, +- handoff fitness test, +- public-source/citation review. + +--- + +### 10. Evidence architecture improved + +This package now separates: + +- normative protocol; +- curated research synthesis; +- retrospective evidence cases; +- raw research inputs. + +This separation is important for later cumulative method development. + +A claim in a normative file is not evidence for itself. diff --git a/README.md b/README.md index f0608e6..19a8e67 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,8 @@ A protocol that forces the two apart: - **Status changes only through a recorded transition.** Never through repetition, never through a polished restatement. - **The commitment act** — a commitment exists only with its rejected alternative, its rationale, and the condition under which it should be reopened. Anything else is a draft, however confident it sounds. - **Decision rights** — what the AI may do alone, what it may draft, what it may only advise on, and what stays with the human. +- **Meaning before commitment** — a term the AI coins starts as a candidate. Fluent vocabulary is not shared understanding until it has been tested against cases. +- **Challenge before convergence** — a model is not ready to commit to until it has survived a deliberate attempt to break it. (Accepted after review; the protocol text follows in v0.2.1.) The governing rule is inverted from most AI tooling: it optimises for the **next valuable human contribution**, not for the completeness of the AI's answer. @@ -41,20 +43,38 @@ It assumes one human, one AI, and a discovery that ends in a handoff. Teams, mul | `capabilities/` | what an adapter would have to do, and how each capability is evaluated | | `research/` | the synthesis, including a standing section for evidence **against** the premise | | `tools/` | the lint that keeps the protocol layer vendor-free | +| `examples/` | one real commitment and one overruled challenge, recorded in the protocol's shape | +| `docs/review/` | the adversarial review of v0.2 and what happened to each of its change proposals | No adapter. No instrumented run. The conformance matrix — which capability is realisable where — is empty because filling it with guesses would be worse than leaving it blank. Start with [`docs/overview.md`](docs/overview.md). +## Using it today + +There is nothing to install. What you can do now: + +- **Use the protocol as a checklist** against your own AI-assisted discovery: does every commitment have a rejected alternative and a revisit condition, did the AI's vocabulary become yours without anyone deciding it? +- **Read [`examples/naming-decision.md`](examples/naming-decision.md)** to see the two central records on a real decision. +- **Attack it.** Open an issue with a contradiction, an unfalsifiable rule, or a discovery where this would have cost more than it returned. + +## How it was developed + +Two discoveries done with an AI, each analysed afterwards as an evidence case; four focused research reports; a protocol revised once on their basis; an [adversarial review](docs/review/2026-09-adversarial-review-v0.2.md) of that revision with 25 findings and 15 change proposals, each proposal [dispositioned](docs/review/disposition.md); then an observation model and a capability catalogue so that the protocol's claims can be measured at all. [`docs/provenance.md`](docs/provenance.md) has the timeline and says what is still private and why. + +## What's next + +Phase 1 finds out what can actually be observed before anything steers: first, whether a conversational turn can be recorded automatically in Claude Code, then state across sessions, skipped capabilities and the cost of enforcement. The instrument for the first question is built; the run is pending. See [`docs/phase-1-feasibility.md`](docs/phase-1-feasibility.md). + ## What the evidence actually says Two retrospective cases, the second derived from the first, so recurrence between them is dependent rather than replicated. No independent case yet. -The study closest to this protocol's target task — LLM support for problem framing, N=280 — found no improvement in frame quality, a widened gap between experienced and inexperienced practitioners, and lower perceived agency among novices. So the honest claim is not that research supports this: +The study closest to this protocol's target task — LLM support for problem framing, N=280 ([Shin et al., CHI 2025](https://doi.org/10.1145/3706598.3713273)) — found no improvement in frame quality, a widened gap between experienced and inexperienced practitioners, and lower perceived agency among novices. So the honest claim is not that research supports this: > earnedshape is a bet that the **design of the interaction** — not the presence of the AI — decides the outcome. The bet is plausible, unproven, and the project is built so that it can lose. -Open findings against this baseline, including two reported by the project's own lint against its own protocol, are in [`docs/known-findings.md`](docs/known-findings.md). +Every source the research synthesis relies on is listed and checked in [`research/REFERENCES.md`](research/REFERENCES.md), including where the support is thinner than the wording. Open findings against this baseline, including two reported by the project's own lint against its own protocol, are in [`docs/known-findings.md`](docs/known-findings.md). ## Contributing diff --git a/capabilities/CONFORMANCE.md b/capabilities/CONFORMANCE.md index c804f0c..3b7c484 100644 --- a/capabilities/CONFORMANCE.md +++ b/capabilities/CONFORMANCE.md @@ -1,7 +1,7 @@ # Conformance Matrix > **Owns:** per surface, which capabilities are realized, by what mechanism, and whether their effect is observable. -> **Status:** Honestly empty. Every `?` is a question for the Phase 1 feasibility spikes. A filled-in guess here would be the most expensive kind of wrong. +> **Status:** Honestly empty. Every `?` is a question for the [Phase 1 feasibility spikes](../docs/phase-1-feasibility.md). A filled-in guess here would be the most expensive kind of wrong. ## How to read it diff --git a/capabilities/catalogue.md b/capabilities/catalogue.md index 39e5feb..d2f5f77 100644 --- a/capabilities/catalogue.md +++ b/capabilities/catalogue.md @@ -2,6 +2,7 @@ > **Owns:** the capabilities derived from the protocol, what each requires, what each must emit, and how each is evaluated on both levels. > **Status:** Draft. The Core/Extended split is the proposal from change proposal CP-11 and is not yet an accepted protocol decision. +> **References:** change proposals (`CP-nn`) and their current status are in [`docs/review/disposition.md`](../docs/review/disposition.md); review findings (`F-nn`) in [`docs/review/`](../docs/review/2026-09-adversarial-review-v0.2.md); the Case 01 and Case 02 codes point into unpublished retrospectives and are explained in [`docs/provenance.md`](../docs/provenance.md). Phase 1 is described in [`docs/phase-1-feasibility.md`](../docs/phase-1-feasibility.md). > **Vendor-free by construction:** no entry names a harness, a hook, a skill or a file format. `initial_form` is `?` everywhere until the Phase 1 spikes report what is reachable. ## How to read an entry @@ -164,7 +165,7 @@ **eval L1** Influential precedents used with no record. **eval L2** Whether a recorded non-transferable assumption is later caught being imported anyway. **falsified by** Records are produced and no import error is ever caught by one. -**note** The protocol does not currently apply this to its own imports (finding F-18). The capability should be exercised on those four first — the cheapest possible test, on material already at hand. +**note** The protocol does not currently apply this to its own imports (finding F-18). The capability should be exercised on those four first — the cheapest possible test, on material already at hand. Done by hand as change proposal CP-09: [`research/analogy-records.md`](../research/analogy-records.md) — each of the four records produced a finding, none of them cosmetic. --- diff --git a/docs/known-findings.md b/docs/known-findings.md index f04be5b..a5df946 100644 --- a/docs/known-findings.md +++ b/docs/known-findings.md @@ -19,6 +19,13 @@ The rule is about citation provenance, not about one product. A technology-indep **Proposed fix:** move section B out of `protocol/` into project documentation, leaving `07` with protocol decisions and open protocol questions. **Status:** open, and the more consequential of the two. +## KF-3 · `research/` states some claims more strongly than their sources support + +The citation check of September 2026 ([`research/REFERENCES.md`](../research/REFERENCES.md)) found every cited work (a few bibliographic fields remain unconfirmed and are marked there), and found that several items under "strongly supported" in `research/01` §2, and a few statements in `research/02`, rest on practitioner literature, a single study, or the project's own inference. The clearest cases: *language and model can co-evolve* (method literature only), *AI is valuable for articulation, search, synthesis and critique* (articulation and drafting are supported, synthesis rests on one preprint, search and critique are not covered), and *human situated input is epistemically special* (a normative reading of requirements research). + +**Proposed fix:** a v0.2.1 pass over `research/01` §2 that moves these items out of "strongly supported" or narrows their wording, with the change recorded here and in the changelog. +**Status:** open. The claims are marked in place and listed in full in `REFERENCES.md`; the wording itself is unchanged until that pass. + ## What this file is not It is not a bug tracker and not a backlog. It holds findings against the *published baseline* that a reader would otherwise have to discover for themselves — the ones an honest reviewer would raise on first reading. Findings that get fixed leave this file and appear in the changelog instead. diff --git a/docs/phase-1-feasibility.md b/docs/phase-1-feasibility.md new file mode 100644 index 0000000..93c3999 --- /dev/null +++ b/docs/phase-1-feasibility.md @@ -0,0 +1,54 @@ +# Phase 1 — what can be observed, before anything is steered + +> **Status (22 September 2026):** planned; one instrument built, no spike run yet. +> **Purpose:** fill [`capabilities/CONFORMANCE.md`](../capabilities/CONFORMANCE.md) and its event-reachability table with **demonstrations, demonstrated impossibilities or named costs**, never opinions. + +## Why this comes first + +The protocol makes claims about what happens in a discovery: a provisional model draws more correction than a polished one, a recorded commitment survives a session boundary, a challenge changes a decision. None of these can be evaluated unless the events they depend on can be recorded. So the order is fixed: + +1. find out what each environment lets you **observe**; +2. record one baseline discovery with **every steering capability switched off**; +3. only then introduce steering capabilities, one at a time, each against that baseline. + +A capability built before step 1 might be unmeasurable. A steering capability built before step 2 leaves nothing to compare against, and the baseline cannot be reconstructed afterwards. + +## Rules for the spikes + +1. **Spike code is throwaway.** Nothing written for a spike goes into an adapter. A spike that turns into the implementation skips the design step it exists to inform. +2. **One question per spike**, answerable in a sitting. +3. **A negative result is a result.** "Not reachable" fills a cell and demotes a hypothesis, which is worth more than a hurried workaround. +4. **Record cost, not just possibility.** "Reachable, but needs a hosted endpoint" is a different answer from "reachable". +5. **No content leaves the spike.** [`04-recording-boundaries.md`](../protocol/observation-model/04-recording-boundaries.md) applies from the first captured byte. + +## The spikes + +| | Question | Status | +|---|---|---| +| **S-0** | Which ChatGPT surface is the second environment: plain chat, a Project, or a Custom GPT with Actions? An owner decision, not a spike; it shapes S-6 to S-8. | open | +| **S-1** | **Is a conversational turn observable** without the model being asked to report it? Decisive: without it, the criticism-surface hypothesis (`HYP-001`) and turn economics (`HYP-009`) are unmeasurable here and get demoted in writing. | **instrument built and self-tested; run pending** | +| S-2 | Does state written in one session arrive intact in the next, across three boundaries including a compaction? (`CAP-01`) | planned | +| S-3 | Can "capability available" be told apart from "capability used", and can a skip be detected? Without this, evaluation level 1 does not exist. | planned | +| S-4 | What does enforcement cost? The same short discovery three times, for the commitment act (`CAP-04`) as guidance, as a skill, and as a deterministic gate, comparing compliance and interruptions. This is the only chance to measure compliance before hardening removes the signal. | planned | +| S-5 | Can "observe this session, ignore that one" be made reliable and fail-open? | planned | +| S-6 | On the chosen ChatGPT surface: per event kind, is capture automatic, semi-automatic, manual, or impossible? | after S-0 | +| S-7 | If the model is asked to emit structured records, how often does it omit, invent or emit late, compared with a manual log over 30 turns? | after S-0 | +| S-8 | What enforcement level is reachable without hooks? Feeds a recommendation on which discoveries belong in which environment. | after S-0 | +| S-9 | One short real discovery, observation only, producing a complete run record: the baseline. | after S-1 to S-5 | + +Sequencing: S-1 first and alone, because a negative result changes the observation model before anything is built on it. S-2 to S-5 in any order. S-9 last, once the record format has settled; a baseline recorded in a format that then changes is not a baseline. + +## S-1 in more detail + +The instrument is a set of hook scripts for Claude Code (kept in the private working repository, as throwaway spike code). It records one line per hook event — timestamp, event name, session id, the **names** of the payload fields, a length bucket for the human's text — and never any text or value. It prints nothing, because anything a hook prints at a prompt boundary is injected into the model's context, and an instrument that talks changes the conversation it measures. It fails open on every path. + +Two findings came out of building it, before any run: + +- **An interrupted turn and a dropped event leave the same trace.** The stop event does not fire on a user interrupt. So the analysis has three verdicts, not two: *clean*; *explained*, where the only gaps are unanswered human turns and the human checks the count against how often they actually interrupted; and *refused*. The two-verdict version passed a dropped turn in the self-test, which is how the third verdict was found. +- **Recording payload keys rather than assuming them** means the first run reports the harness's actual contract instead of the author's expectation. + +What the run needs: a real conversation of at least 20 exchanges, with tool use, at least one interruption, and ideally long enough to compact. The case that matters most is not in any verdict: whether the stream survives compaction and long sessions, because a discovery is long by definition. + +## Exit + +Phase 1 is done when every cell in the conformance matrix and the event-reachability table holds a demonstration, a demonstrated impossibility or a named cost; when there is a recommendation on which discoveries belong in which environment; and when every hypothesis that turns out unobservable everywhere has been marked as such in the protocol instead of looking like a mechanism awaiting evidence. diff --git a/docs/provenance.md b/docs/provenance.md new file mode 100644 index 0000000..eb3aa33 --- /dev/null +++ b/docs/provenance.md @@ -0,0 +1,67 @@ +# Provenance — where this material comes from + +This repository was derived on 14 September 2026 from a private working repository in which the protocol was developed. Some files here refer to documents, identifiers and projects from that repository. This page says what they are, so a reference is at least understandable where it cannot be followed. + +## How the protocol came about + +| When | What | Published here? | +|---|---|---| +| before Sep 2026 | **Case 01** — a long product discovery with an AI (the recipe tool [Cookframe](https://github.com/kschlt/cookframe)), analysed retrospectively afterwards | no — see below | +| up to 5 Sep 2026 | **Case 02** — a second discovery whose subject was human–AI discovery itself, using Case 01's retrospective as input. It produced protocol v0.1 | no | +| | Four focused research reports: human contribution and interaction design; problem framing, solution probes and fixation; evaluating a discovery protocol; shared conceptual language | as a synthesis in [`research/`](../research/), with checked sources in [`research/REFERENCES.md`](../research/REFERENCES.md) | +| 5 Sep 2026 | **Protocol v0.2**, after a post-hoc shared-language addendum to Case 02 | [`protocol/`](../protocol/), [`CHANGELOG.md`](../CHANGELOG.md) | +| 10 Sep 2026 | Orientation and **adversarial review** of v0.2: 25 findings, 15 change proposals | [`docs/review/`](review/) | +| 10–14 Sep 2026 | **Phase 0**: observation model, capability catalogue, conformance matrix, boundary lint; four change proposals executed | [`protocol/observation-model/`](../protocol/observation-model/), [`capabilities/`](../capabilities/), [`research/hypothesis-register.md`](../research/hypothesis-register.md), [`research/analogy-records.md`](../research/analogy-records.md) | +| 14 Sep 2026 | Name decided, this repository derived | [`examples/naming-decision.md`](../examples/naming-decision.md) | +| from 14 Sep 2026 | **Phase 1** planned: feasibility spikes on what can be observed | [`docs/phase-1-feasibility.md`](phase-1-feasibility.md) | + +Cases 01 and 02 and the v0.1/v0.2 baseline were produced with ChatGPT-family models and a deep-research tool; the review and Phase 0 were done with Claude in Claude Code. The same person is practitioner, protocol author, evidence analyst and evaluator throughout — the central threat to validity the review names. + +## What stays private, and why + +- **The evidence cases** (Case 01, about 2,300 lines in German; Case 02 and its addendum). They are raw retrospectives and need editing for self-containedness and non-circular evidence logic before release. The synthesis in `research/` states that they are unpublished and that this limits what a reader can verify. +- **The raw research reports** (in German). They were produced with a deep-research tool and carry that tool's internal citation markers. The claims this repository relies on have been checked against publisher and index records instead; see `research/REFERENCES.md`. +- **Project-level working documents** about repository setup, naming and how this project relates to the author's other projects. + +## The numbered working documents + +Files here sometimes cite these by number, for example "doc 10" or `09`. + +| No. | Content | Here | +|---|---|---| +| `00` | changelog v0.1 → v0.2 | [`CHANGELOG.md`](../CHANGELOG.md) | +| `08` | handover of the v0.2 package to the next agent | — | +| `09` | orientation and adversarial review of v0.2 | [`docs/review/2026-09-adversarial-review-v0.2.md`](review/2026-09-adversarial-review-v0.2.md) | +| `10` | project setup and evolution loop: the layer cut, repository layout, private run store, fidelity levels, phases | its conclusions are in [`docs/overview.md`](overview.md) | +| `11` | brief for the naming session | — | +| `12` | capability and evaluation model; reconciliation with the author's other projects | its §2 is the basis of [`capabilities/`](../capabilities/) | +| `13`–`15` | portfolio model: how the author's projects relate, what may be shared between them | — | +| `16` | status ledger and change-proposal disposition | [`docs/review/disposition.md`](review/disposition.md) | +| `17` | naming decision and move plan | [`examples/naming-decision.md`](../examples/naming-decision.md) | +| `18` | Phase 1 spike plan | [`docs/phase-1-feasibility.md`](phase-1-feasibility.md) | +| `19` | reconciliation of two parallel Phase 0 drafts | — | +| `20` | derivation record for this repository | — | + +## Identifier schemes + +| IDs | Meaning | Defined in | +|---|---|---| +| `D-nn`, `T-nn`, `P-nn`, `E-nn` | protocol decisions, deferred technical decisions, open protocol questions, evaluation questions | [`protocol/07`](../protocol/07_DECISIONS_AND_OPEN_QUESTIONS.md); `T-11`–`T-13` were proposed in the [adversarial review](review/2026-09-adversarial-review-v0.2.md) | +| `CAP-nn` | capabilities | [`capabilities/catalogue.md`](../capabilities/catalogue.md) | +| `F-nn` | findings of the adversarial review | [`docs/review/`](review/2026-09-adversarial-review-v0.2.md) §3 | +| `CP-nn` | change proposals, `CP-01`–`CP-15` from the review, `CP-16`–`CP-21` from executing CP-10 | [`docs/review/disposition.md`](review/disposition.md) | +| `HYP-nnn` | unified hypothesis register | [`research/hypothesis-register.md`](../research/hypothesis-register.md) | +| `KF-n` | known findings against the published baseline | [`docs/known-findings.md`](known-findings.md) | +| `M-nn`, `M1`–`M8`, `G1`–`G7`, `L1`–`L10`, `AP-nn`, `SC-n`, `EV-n`, "Phase n" | codes inside the Case 01 retrospective: method hypotheses, method elements, guidelines, lessons, anti-patterns | unpublished | +| `E-02-nn`, `H-nn`, `HYP-0n`, `H-SL-01` | codes inside the Case 02 retrospective and its addendum | unpublished | + +## The author's other projects that appear by name + +Named where the protocol borrowed from them or was checked against them. Except for Cookframe they are not public at the time of writing. + +- **Turn** — a design for *adaptive cognitive handoff* between a human and predominantly autonomous implementation agents. Protocol v0.1 imported its model and over-assimilated it; the analogy records trace what that import cost ([`research/analogy-records.md`](../research/analogy-records.md), A-03). +- **aos** — the author's agent workflow system for building software with Claude Code: backlog, session protocol, sealed commits, merge gate. Its hook-based telemetry was the closest existing precedent for the observation model, and it is the intended build workflow for this project's tooling. +- **loomwise** — a Claude Code plugin for human-governed knowledge development ("the AI proposes; the human owns"); the precedent for plugin packaging and behavioural evaluation. +- **CWS** — a project that observes work across projects and proposes methods and tools from it; related in scope, deliberately kept independent of this one. +- **Methodplane** — the umbrella under which these projects may eventually be published together. +- **Cookframe** — the product of the Case 01 discovery: [github.com/kschlt/cookframe](https://github.com/kschlt/cookframe). diff --git a/docs/review/2026-09-adversarial-review-v0.2.md b/docs/review/2026-09-adversarial-review-v0.2.md new file mode 100644 index 0000000..6037fe3 --- /dev/null +++ b/docs/review/2026-09-adversarial-review-v0.2.md @@ -0,0 +1,408 @@ +# Orientation & Adversarial Review — Discovery Pack v0.2 + +> **Published record.** This is the review as written on 10 September 2026 against *Discovery Pack v0.2*, the private working package from which this repository was later derived. It is reproduced unchanged apart from this note, because the point of publishing it is that the findings and change proposals can be checked against what was done with them — see [`disposition.md`](disposition.md). +> File names such as `08_HANDOVER`, `README.md`, `evidence/` or `source-research/` refer to that package, not to this repository. What each of them is, and what the ID schemes (`F-nn`, `CP-nn`, `D-nn`, Case 01/02 codes) mean, is explained in [`docs/provenance.md`](../provenance.md). + +> **Status:** Non-normative review input. Project layer: same as `08_HANDOVER_TO_NEXT_AGENT.md` (project input, not protocol, not evidence, not research). +> **Scope:** Task 1 of the handover only — orientation and whole-baseline adversarial review. +> **Protocol changes made:** none. Every proposed change is stated as an explicit change proposal in §5. +> **Reviewer independence:** partial. This review is by an AI agent, on a baseline produced by an AI–human pair. It is *not* the independent adversarial review that D-21 / §15 requires. See Q6. + +--- + +## 1. What this project is trying to become + +My reading, stated so it can be corrected: + +**The asset is a versioned, technology-independent protocol for one human and one AI doing pre-implementation discovery together, plus an evidence apparatus that lets the protocol be revised on something better than taste.** + +Three things follow from that, and they are what make the project unusual: + +1. **The protocol is not a prompt and not a workflow.** It is a set of semantics — what kinds of objects exist (probes, candidates, decisions, commitments, concepts, assumptions), what status they carry, who may change that status, and what the AI may do without asking. A prompt is a binding-level artifact; the protocol is what a prompt would have to realize. + +2. **The optimization target is inverted relative to normal AI tooling.** Most AI assistance optimizes for the completeness of the AI's answer. This protocol optimizes for *the next high-value human contribution*, and treats a fluent, complete AI answer as a potential loss event (anchoring, agency erosion, silent commitment). The core thesis in `README.md` is a claim about what to *protect*, not about what to automate. + +3. **Method evolution is part of the product.** The evidence layering (`evidence/` ≠ `research/` ≠ `protocol/` ≠ `source-research/`), the changelog, the "what this does not prove" sections and the learning backlogs mean the intended end state is not "a good protocol" but "a protocol that accumulates evidence about itself". + +The intended long-run shape, as far as I can infer it: a public protocol spec with independent version history, one or more reference bindings (Claude Code as first candidate), an evidence archive of instrumented discovery cases, and an evaluation harness that can tell whether a protocol change actually improved anything. + +**What it is trying *not* to become**, also inferred: a multi-agent discovery platform, a methodology brand, a portfolio piece, or a glossary/process bureaucracy. The pack states all four resistances explicitly, which is a good sign. + +**The biggest structural risk to that ambition** is stated in the pack's own words (`README.md`, "Current maturity") and I agree with it: *the next phase should make these mechanisms testable, not merely more elaborate*. v0.2 is currently far more elaborate than it is testable. Almost everything in §2.5 below is a mechanism with a name, a rationale and no instrument. + +--- + +## 2. Layer separation + +The handover asks for five categories. I found that two of them (invariants and evidence-backed claims) only separate cleanly if a third distinction is added: *what is invariant because it is well-supported* vs *what is invariant because it was decided*. I have marked that below. + +### 2.1 Protocol invariants (change only via explicit revision) + +These are the load-bearing commitments. I would not touch any of them on current evidence. + +| Invariant | Source | Why it holds | +|---|---|---| +| The asset is a versioned protocol, separate from any binding | D-01, D-02 | Design decision; supported by the observation that harness limits would otherwise rewrite semantics | +| Five human contributions: Situated Grounding, Framing & Meaning, Independent Position/Authorship, Judgment & Exceptions, Commitment & Accountability | D-05 | Framed as *epistemic/normative roles*, not capability claims — that framing is what makes them defensible | +| Human testimony is evidence, not ground truth | `01_PROTOCOL` §4.1 | Directly supported by RE elicitation research (requirements are co-constructed) | +| Progressive Commitment: proposal ≠ working model ≠ recommendation ≠ decision ≠ commitment | D-08, D-17 | The central mechanism; everything else is instrumentation for it | +| `Probe ≠ Candidate ≠ Recommendation ≠ Decision ≠ Commitment` | D-09 | Same | +| Problem and solution co-evolve; separate *commitment status*, not problem and solution | D-07, §7 | Well supported (Dorst & Cross; RE) | +| Initiative is a property of the move, not a role | D-06 | Mixed-initiative HCI; observed in both cases | +| AI must not silently commit on the human's behalf | §4.5, `03` §7 | Design decision, high confidence | +| Shared Language is first-class and cross-cutting, **not** a sixth contribution | D-14 | Decision + research support | +| `Meaning before commitment`, not meaning before naming | D-16 | Research *changed* this from v0.1 — an actual falsification event, worth preserving | +| AI names for model-relevant concepts start as proposals | D-15 | Supported as risk-justification, not as measured effect | +| "Canonical" means canonical-for-now-and-context | §7 of `04` | DDD + conceptual pacts | +| No single protocol score; quality + agency gates | D-11, D-12 | Well supported by evaluation research | +| Public artifacts are durable; evidence hygiene precedes publication | D-13 | Decision | + +### 2.2 Experimental hypotheses (named, unproven, must not be presented as practice) + +The pack is honest about these, and the honesty is correctly located (handover §4, `01_EVIDENCE` §7, `04` §15, Case 02 HYP-01..05). Consolidated: + +- five-factor adaptive routing as an actual router; +- the exact Independent-Position / human-first trigger; +- explicit `PROBE` labelling and a mandatory probe debrief; +- the shared-language lifecycle `candidate → working → canonical-for-now → reopened`; +- visible Human/AI/External provenance labels; +- a scenario gate before canonicalization; +- persistent external state beating strong implicit memory; +- concept registry vs implicit memory; +- the proposed run-level metrics as predictors of downstream implementation quality; +- exploration-turn brevity; +- bounded-uncertainty readiness; +- the analogy boundary check as a *mechanism* (its diagnosis is evidenced; its ritual is not). + +**Naming problem:** these hypotheses exist in five overlapping ID spaces — `H-01..H-12` (Case 02 §15), `H-SL-01` (addendum), `E1..E10` (`05` §10), `P-01..P-14` / `E-01..E-08` / `T-01..T-10` (`07`), and `M-01..M-12` (Case 01). They partly duplicate each other. See CP-11. + +### 2.3 Evidence-backed claims (external research, not this project's own cases) + +Strong enough to rely on: + +- Human+AI is not automatically synergistic; average synergy vs the better single actor was negative (Vaccaro et al. 2024, 106 experiments). This is the strongest single result in the pack and it justifies the whole "design the coupling" premise. +- Generative AI can increase individual output quality while homogenizing outputs (Doshi & Hauser; de Rooij & Biskjaer meta-analysis). +- AI examples can increase design fixation and reduce divergent output (Wadinambiarachchi et al., CHI 2024). +- LLM use *before* independent ideation reduces ownership and self-efficacy; use *after* is better (CHI 2025 experiment). +- Explanations and confidence scores are not robust moderators of appropriate reliance; cognitive forcing works better but is disliked. +- Requirements are co-constructed, not extracted (Ferrari et al.; only 30–38% traceable to original customer ideas). +- Problem/solution co-evolution (Dorst & Cross). +- Lexical entrainment and human→computer lexical alignment are real; shared language reduces referential cost; conceptual pacts are local and revisable. +- Structured scaffolding can beat generic chat (Supermind Ideator) — but see F-10. + +**Missing from the curated layer, and it matters:** Shin et al. (CHI 2025, N=280) found *no improvement in problem-frame quality* from LLM support, a widened expert/novice gap, and lower perceived agency among novices. This is the study closest to the project's actual target task, and it is the strongest disconfirming evidence in the whole pack. It appears once, in `source-research/04`, and never reaches `research/` or `protocol/`. See F-10 / CP-07. + +### 2.4 Process evidence (this project's own cases — weaker, and dependent) + +- **Case 01 (Cookframe, object-level):** intermediate models as criticism surfaces; human held product/scope decisions; hypothesis-led research beat broad research; external adversarial review at the right moment, then a narrow verification review, then stop; source/canonical/derived separation; explicit deferral. +- **Case 02 (meta-level):** the same criticism-surface pattern; human framing/boundary correction as the highest-value human act; research produced material model change; analogy over-assimilation (Turn); deferral preserved openness; no whole-package adversarial review happened. +- **Case 02A (post-hoc):** rapid uptake of AI/research-originated terminology is observable; causal anchoring is **not** established. The addendum's refusal to retro-fit the evidence is the single most disciplined move in the pack. + +Correct epistemic label, already stated by the pack: **cross-level dependent recurrence, not replication.** + +### 2.5 Implementation requirements (things any binding must provide) + +Extracted from `02` §11 and the invariants — these are genuinely normative for a binding: + +1. Persist explicit commitments across sessions; losing one is a defect. +2. Keep epistemic status (evidence / human input / inference / assumption / uncertainty) distinguishable from decision status. +3. Make status transitions explicit and traceable — in particular, no probe→commitment promotion without a recorded transition. +4. Keep deferred decisions retrievable and reviewable. +5. Surface contradictory active constraints rather than holding both silently. +6. Support reopening/renaming/splitting concepts with rename history where meaningful. +7. Make state maintenance cheap, continuous and *not* human-approved. +8. Produce a handoff from which key decisions and concepts can be reconstructed by a party who was not present (cold-start). +9. Keep protocol semantics separable from harness mechanics (no harness limit silently redefining a rule). + +**Gap:** none of these define *what a commitment physically is* — who writes it, in what form, with what fields, and how it is revoked. That is the smallest missing piece blocking any binding. See F-22 / CP-06. + +### 2.6 Still-open implementation choices (correctly deferred) + +`T-01..T-10` are the right list: name, repo topology, first harness, state serialization, skill/hook/agent architecture, review/handoff surface, evaluator architecture, release workflow, concept storage, provenance visibility. + +I would add three that the pack does not list: + +- **T-11 — Conformance definition.** What makes a run "v0.2-conformant"? Without this, Case 03 cannot claim to test the protocol; it can only claim to be inspired by it. +- **T-12 — Trace/telemetry format.** `05` assumes traces exist. Nothing defines an event. +- **T-13 — Session/context boundary handling.** State exists because context is lost; nothing says what happens at a session boundary, at context limit, or when the model version changes mid-project. + +--- + +## 3. Adversarial review of Protocol v0.2 + +Severity: **H** = fix before Case 03; **M** = fix before public release; **L** = hygiene. + +### 3.1 Contradictions and internal tension + +**F-01 (M) — The mode taxonomy separates exactly what §7 forbids separating.** +`01` §7 says "separate commitment status, not problem and solution", yet §5 names the two exploration modes `PROBLEM_EXPLORATION` and `SOLUTION_EXPLORATION` and `02` §1 makes Discovery Mode the answer to "Where are we?". A reader implementing the protocol will build a phase machine out of the mode names and reintroduce the very sequencing the protocol rejects. `P-01`/`P-06` acknowledge the taxonomy is open, but the protocol still uses it normatively. + +**F-02 (H) — Adversarial challenge is a release ritual, not a convergence gate.** +§15 places independent adversarial review before *public release*. But the best-evidenced convergence mechanism in the entire pack is Case 01's `broad adversarial review → fixes → narrow verification review → stop` (Case 01 Phase 9, G6, L4, M-03, and lesson L5), and Case 02's single most-cited defect (E-02-20) is that this did not happen. §13's justification list for convergence contains seven conditions and *not one of them is "the model survived a deliberate attempt to break it"*. (Precision, added after review: §13's closing sequence does read `compare → expose uncertainty → challenge → decide → commit`, so challenge is not absent from the section — but it appears as a step in a suggested order, not as a condition that justifies convergence, and nothing requires the challenge to come from a party independent of the pair that built the model.) The protocol learned this lesson only as a packaging step. This is the largest substantive gap I found. + +**F-03 (H) — The Independent-Position trigger is undecidable and is assigned to the conflicted party.** +§4.3: "The trigger is whether a strong AI anchor would destroy information worth observing." The AI must evaluate, before producing output, whether its own output would erase a human signal it has not yet seen. That is not evaluable in principle, and the evaluator is the party whose output is the risk. `02` §3.1 gives examples, which helps, but the normative rule remains a self-judgment. This also makes hypothesis E2 untestable as written: you cannot measure compliance with an unobservable trigger. + +**F-04 (M) — Anti-bureaucracy rule vs. the concept record.** +`04` §11 and `02` §6 specify a concept entry with nine fields plus rename history and a five-state lifecycle; `03` §10 and `04` §14 list "vocabulary bureaucracy" as a failure mode; `04` §3 gates entry on a subjective question the AI answers alone. The protocol therefore both mandates the machinery and warns against using it, with no operational threshold. Under-application and over-application are equally consistent with the text. + +**F-05 (L/M) — Concept status is not context-scoped, but canonicity is.** +`04` §9 says a term may legitimately be canonical in Billing and different in Identity. The state schema attaches one `status` and one `context` to one `term`. A term that is canonical-for-now in one context and candidate in another cannot be represented without duplicating the entry, and nothing says duplication is intended. + +**F-06 (M) — "Artifacts are projections of state" is asserted by a baseline that is entirely artifacts and has no state.** +D-10 flags the runtime effect as experimental, which is honest, but `02` §8 states the lifecycle as normative. Meanwhile `02` §8 wants maintenance "mostly invisible" and un-approved, while `02` §11 requires explicit, traceable transitions. Invisible-but-traceable is achievable, but only if the protocol says which mutations are silent and which are events. It does not. + +### 3.2 Redundancy and unnecessary complexity + +**F-07 (M) — Four devices encode one idea.** +Progressive Commitment (`01` §3), structure-for-exploration vs structure-for-convergence (`01` §6), three status dimensions (`02` §7), and half the failure-pattern lists (`03` §10, `04` §14) are all instruments for *"do not let tentative become committed by fluency and repetition"*. The four are not wrong, but they are not orthogonal, and each carries its own vocabulary. A single commitment ladder with typed instantiations (frame / term / solution / research implication) plus one presentation rule would say the same thing with a third of the surface. + +**F-08 (H) — ~30 named runtime categories, none instrumented.** +3 modes + 11 interaction moves + 6 shared-language micro-moves + 5 human contributions + 5 routing dimensions + 3 status ladders (5+6+5 values). A binding must decide, per turn, where it is in all of them. Two specific symptoms: `MAINTAIN` sits in the same taxonomy as `ELICIT` and `DECIDE` although `02` §8 says maintenance should be invisible and un-approved; and `CLARIFY_MEANING` overlaps `REFLECT_SOURCE` / `SEMANTIC_PROBE` from a different list. This is the main reason the protocol is currently hard to implement *and* hard to test: there is no declared core. + +**F-09 (H) — The evaluation design cannot run at the available sample size.** +`05` specifies 7 core signals, 6 shared-language diagnostics, and 10 priority experiments, then §9 defines Keep/Investigate/Revert rules that presuppose comparative inference across runs. With n≈1 practitioner and runs measured in weeks, no Pareto comparison over 7 dimensions will ever be decidable, and E-05 already concedes that practitioner learning confounds run-over-run comparison. As written, the evaluation layer will produce numbers that cannot license its own decision rules. A defensible n=1 design is different in kind: pre-registered predictions, falsification criteria, and a small number of qualitative discriminating observations — not a metric dashboard. + +### 3.3 Unsupported certainty and evidence handling + +**F-10 (H) — The curated layer drops the strongest disconfirming study.** +As noted in §2.3: Shin et al. (CHI 2025, N=280) — no problem-frame quality improvement from LLM support, widened competence gap, lower perceived agency in novices — exists only in the German raw research. `research/01` §2 "Strongly supported directions" instead carries "Structured scaffolding can outperform generic chat" (Supermind Ideator, an *ideation* task). The pack's rule that "a claim in a normative file is not evidence for itself" is respected; the complementary rule — *curation must not filter out the closest counter-evidence* — is not. On current evidence, the honest headline is: **the closest study to this project's target task found no benefit, and the protocol is a bet that interaction design is what makes the difference.** That is a defensible and interesting bet. It should be stated as one. + +**F-11 (M/H) — Two incompatible evidence-strength vocabularies.** +`source-research` uses *hoch / mittel / fundiert-theoretisch / emerging* with explicit definitions. Case 02's ledger uses *High / Medium / Low* where "High" means "clearly observed in this single, self-authored, dependent case". A reader — or a future agent — will read "Evidence strength: High" on E-02-01..E-02-07 as strong evidence for the rule. Case 02 §16 corrects this in prose, 500 lines later. + +**Sharper instance of the same defect, added after review:** `source-research/01` rates the Supermind Ideator / scaffolding evidence *mittel* and explicitly *widersprüchlich*. `research/01` §2 promotes the same evidence to "**Strongly supported** directions". That is strength inflation happening inside the curation step itself, on the claim that most directly supports the project's premise — a more concrete example than the ledger-vocabulary one above, and it belongs with F-10 rather than beside it. + +**F-12 (M) — Zero citations have been independently verified.** +The pack says citations must be verified before publication, which is right, but no file states the current status plainly: every reference in `source-research/` comes from a single Deep Research pass, carries internal `citeturn…` markers, includes several 2026-dated items and at least one arXiv preprint, and at least one entry (Noy & Zhang) has no DOI. Fabricated or drifted citations are a known failure mode of that production method. Until re-opened, *every* literature claim in this pack — including the ones I relied on in §2.3 — is unverified. + +**F-13 (M) — `05` silently narrows the researched metric set.** +`source-research/03` recommends nine core metrics; `05` §3 carries seven, dropping *Discovery Quality Profile* and *Beneficial AI Uptake / Defective AI Resistance* — while `05` §9 still uses "defective-AI resistance" in the Keep/Investigate/Revert rule, i.e. a decision rule referencing a signal the protocol no longer collects. Narrowing may well be right (both dropped metrics need an independent reviewer), but the delta is undocumented. + +### 3.4 Hidden assumptions + +**F-14 (H) — The configuration is never declared: one human, one AI, one long conversation, no third parties.** +No stakeholders, no team, no client, no deadline, no budget, no parallel engineering. This assumption is load-bearing everywhere (turn economics, attention scarcity, decision rights, handoff), and it is nowhere stated. It is most acute in the shared-language layer, whose entire evidence base — Ubiquitous Language, common ground, lexical entrainment, conceptual pacts — is about *multi-party* alignment. In a 1:1 human–AI dyad where one party has no persistent memory and no stake, "shared language" may be a different phenomenon with a different failure profile. + +**F-15 (M/H) — Every safeguard against AI anchoring is executed by the AI.** +Candidate-vs-canonical status, "do not present exploratory structure with convergence certainty", "surface consequential ambiguity", "do not silently commit" — all are model self-restraint against a failure mode (fluency → repetition → implicit commitment) that models are structurally poor at detecting in themselves. The protocol never names a single deterministic enforcement point outside the model. For a binding this is the key design question, and it is invisible in v0.2. + +**F-16 (M) — Discovery is assumed to terminate in a handoff to a different implementation context.** +True in Case 01 and Case 02; not true of most product work, where the same people continue into build. §14's readiness definition and the cold-start test both depend on it. Either declare it as scope, or state what readiness means when discovery flows continuously into implementation. + +**F-17 (M) — Model and harness invariance is assumed.** +The whole baseline was produced with ChatGPT-family models and Deep Research (inferable from Case 01's text; never stated). The protocol's central risk claim is a claim about *model behavior*. Case 03 is likely to run on a different model family in a different harness, which makes model family an uncontrolled variable in the first prospective test. `research/01` §7 lists portability as untested but does not treat it as a design variable. + +**F-18 (H, cheap) — The protocol does not apply its own Analogy Boundary Check to its own imports.** +§11 mandates the check for any imported framework. v0.2 imports at least four: DDD / Ubiquitous Language (multi-party, code-bearing, long-lived teams), common ground & lexical entrainment (human–human dyads, referential tasks), *Turn* (predominantly autonomous implementation agents), and cognitive forcing functions (single-decision AI-advice settings). No analogy record exists for any of them. Case 02 identified analogy over-assimilation as a real failure that already happened once in this project. + +### 3.5 Overfitting to the two dependent cases + +**F-19 (H) — v0.2 is shaped by Case 02, and Case 01's operational layer was dropped without a recorded disposition.** +Case 01 does not merely contain observations. It contains a complete *Discovery Method v0.1*: phases M1–M8 with explicit exits, decision gates G1–G7, learning loops L1–L5, a four-way external-verification typology, stop conditions (including an explicit "probably overthinking" condition), anti-patterns AP-01..AP-10, and a twelve-item learning backlog. Protocol v0.1/v0.2 carries almost none of it: no gates, no stop conditions beyond prose, no review typology, no anti-pattern register, phases replaced by three modes. + +Some of that was surely deliberate (phase machines conflict with co-evolution). But **no file records the disposition**, so a reader cannot tell what was rejected from what was forgotten. Concretely, several dropped items look stronger than things v0.2 kept: + +- G3 *Information Loss Gate* + L3/M-05 *source / canonical / derived* — Case 01 explicitly flags this as a generalizable AI-discovery pattern. v0.2 rediscovered a narrow version (`source_expression`) via external research, and only for terminology. +- G6 *Review Gate* + M-03 *two reviews beat one* → see F-02. +- L9/M-12 *formal stop criterion instead of "it feels round"* → v0.2 has readiness prose, no gate. +- AP-10 / M-10 *artifact version churn* → absent, although the pack is itself a pile of versioned artifacts. + +The net effect: the *object-level* case — the only one that is not about discovery methodology, and therefore the least self-referential evidence available — contributed the least normative content, while the meta-level, self-authored, dependent case contributed the most. + +**F-20 (M) — Mechanisms tuned to one strong-willed expert practitioner.** +"Criticism surface" works because the human in both cases pushes back hard, repeatedly, with domain expertise. Shin et al. suggests the opposite population exists and behaves differently (novices lose agency and accept more). "Framing correction is the highest-value human act" generalizes the same person's strength. "Human attention is scarce" (D-04) is a design driver derived from one person's circumstances. "Deferral is cheap" holds when nobody is waiting on the decision. + +**F-21 (M) — "Discovery" implicitly means greenfield conceptual design.** +Both cases are greenfield. Nothing addresses discovery inside an existing system, with legacy constraints, existing users, migration, or data that already encodes prior decisions — where "reopening the frame" is not cheap. + +### 3.6 Gaps + +**F-22 (H) — The commitment act is undefined.** See §2.5. `02` §11 requires that promotion to commitment need an "explicit transition"; nothing defines the transition, its fields, its owner, or how a commitment is revoked or superseded. + +**F-23 (M/H) — No object for unresolved disagreement or AI dissent.** +Case 01 names *Confirmation Bias im normalen Chat* as an observed problem and AP-02 *Friendly AI Consensus* as an anti-pattern. `03` requires the AI to challenge — but if the human overrules the challenge, nothing records it, and the dissent does not survive into the handoff. This is also the cheapest partial substitute for the missing adversarial gate (F-02). + +**F-24 (M) — No treatment of context limits, session boundaries, cost, or state growth.** +External state exists *because* context is lost, yet nothing specifies what happens at a session boundary, what must be re-loaded, how state is compacted, or what it costs. `05` mentions AI latency once. + +**F-25 (H) — No conformance definition.** See T-11. Without it, "we ran Case 03 under Protocol v0.2" is not a checkable statement, and E1–E10 cannot be attributed to the protocol rather than to the practitioner. + +### 3.7 What holds up well (fair-review counterweight) + +- The evidence layering, and the rule that a normative claim is not evidence for itself. +- Case 02A's refusal to retro-fit shared-language evidence onto a retrospective written before that lens existed. This is better epistemic practice than most published method work. +- Case 02 §16's explicit "what this does not prove" list. +- `Meaning before naming → meaning before commitment` is a documented case of research overturning the project's own prior rule. That is the pack's best evidence that the learning loop is real. +- Refusing a composite score, and refusing "human token share" as an agency proxy. +- The decision-rights asymmetry (`03` §§3–7) is the most implementable part of the protocol and needs the least work. + +--- + +## 4. Priority ranking of findings + +If only five things are fixed before Case 03: **F-02** (adversarial gate), **F-25 + F-22** (conformance + commitment act), **F-08** (declare a core), **F-09** (evaluation that works at n=1), **F-10/F-11** (evidence honesty in the curated layer). + +--- + +## 5. Change proposals + +None of these are applied. Each states rationale, evidence, risk, and when it should be decided. + +### CP-01 (High) — Make adversarial challenge a convergence gate, not only a release gate +**Change:** add to `01` §13 a condition — *"the current model has survived at least one deliberate attempt to break it, by a party or context independent of the pair that built it"* — and add a short review typology to the protocol: exploration review (what perspective is missing) / adversarial review (where does the model break) / verification review (are the named blockers actually closed) / spike (how does the external world behave). Keep §15 as the *release* instance of the same mechanism. +**Rationale/evidence:** Case 01 Phase 9, G6, L4, lessons L5 and M-03; Case 02 E-02-20 (the pack's most-cited own defect). This is the only mechanism supported by both cases *and* by a concrete before/after outcome (broad review → precision, not redesign; narrow review → stop). +**Risk:** ritualization; "independent reviewer" is expensive and may not exist. Mitigate with Case 01's own G6 threshold: review when the model is coherent, concrete enough to attack, and still cheap to change. + +### CP-02 (High) — Remove the problem/solution split from the mode names +**Change:** either (a) rename to `EXPAND` / `CONVERGE` with an attribute `focus: problem | solution | both`, or (b) keep three modes but state explicitly that a mode names the *dominant learning objective of the moment*, never a phase, and that mode switches are expected many times per session. +**Rationale:** removes F-01; (a) also removes one taxonomy level, helping F-08. +**Risk:** loses vocabulary already used in `02` and in Case 02; renaming has a real cost. (b) is the low-cost option. + +### CP-03 (High) — Replace the self-judged Independent-Position trigger with an enumerated event list +**Change:** in §4.3, replace "whether a strong AI anchor would destroy information worth observing" with a small closed list of anchor-risk events, e.g.: first framing of a newly opened object; selection among materially different directions; naming of a framing-bearing concept; statement of a high-impact criterion or priority; any decision the human will have to defend later without the AI present. Keep "not a universal human-first mandate". +**Rationale:** F-03. Also makes E2/H-02 testable — compliance becomes observable. +**Evidence:** the timing evidence (LLM before vs after independent ideation) is about *sequence at specific moments*, which is what a list encodes; it does not support a general "AI holds back" rule. +**Risk:** the list will be wrong at first. That is acceptable — a wrong list is revisable; an unobservable rule is not. + +### CP-04 (Medium) — Collapse the four tentativeness devices into one ladder +**Change:** state Progressive Commitment once, with typed instantiations (frame / term / solution object / research implication) and one presentation rule ("render status honestly: exploratory structure must not carry convergence rhetoric"). Demote §6 to guidance under it; keep `02` §7 as the state encoding of the same ladder. +**Rationale:** F-07. Reduces surface without losing a distinction. +**Risk:** §6's exploration/convergence distinction is genuinely useful; it must survive as a rendering rule, not disappear. + +### CP-05 (High) — Add a recorded-disagreement object +**Change:** add to state and to the handoff a small object: `challenge` — what the AI (or a reviewer) contested, the human's response, resolution status (`accepted / overruled / deferred / unresolved`), and whether it remains open at handoff. +**Rationale:** F-23; Case 01 AP-02 and the confirmation-bias observation. Cheapest available counterweight to a two-party echo chamber, and it makes "the AI challenged" verifiable rather than aspirational. +**Risk:** could become a complaint log. Bound it to high-impact decisions only. + +### CP-06 (High) — Define the commitment act +**Change:** specify that a commitment exists only as an explicit recorded transition carrying: object, owner, date, the alternative rejected, the reason, the evidence/assumption it rests on, and the condition under which it should be revisited. No other route creates a commitment. +**Rationale:** F-22. This is the minimum deterministic anchor a binding needs, and it converts D-08 from a principle into something a hook or file schema can enforce (F-15). +**Risk:** friction if applied to everything — apply only to the `commitment` tier, not to decisions. + +### CP-07 (High) — Reinstate disconfirming evidence in the curated layer +**Change:** add Shin et al. and the negative/null findings to `research/01` §2, and add a standing section *"Evidence against this protocol's premise"* that any future research pass must fill or explicitly mark empty. +**Rationale:** F-10. Also protects the project reputationally: a public method document that omits the closest null result invites exactly the criticism the project is trying to earn immunity from. +**Risk:** none I can see, beyond a less confident-sounding README. That is the point. + +### CP-08 (Medium) — One evidence-strength scale, two columns +**Change:** harmonize the vocabularies across `evidence/` and `research/`, and in evidence ledgers split the single strength column into **observation clarity** (how clearly was this seen in this case) and **evidential weight** (what does it license for the protocol). Re-label Case 02's ledger accordingly. +**Rationale:** F-11. +**Risk:** editing an existing evidence case — do it as an added column and a dated note, never by rewriting the original judgements. + +### CP-09 (Medium, cheap) — Analogy records for the protocol's own imports +**Change:** add an appendix with a completed `analogy` record (per `02` §10) for DDD/Ubiquitous Language, common ground/lexical entrainment, Turn, and cognitive forcing functions. +**Rationale:** F-18; self-application of §11. Also the fastest way to discover whether the shared-language layer is over-imported from multi-party settings (F-14). + +### CP-10 (High) — Case 01 method disposition table +**Change:** add a table mapping every Case 01 method element (M1–M8, G1–G7, L1–L5, AP-01..AP-10, M-01..M-12) to `kept / transformed into X / deliberately dropped because Y / not yet considered`. +**Rationale:** F-19. Without it, nobody can tell rejection from oversight, and the strongest object-level evidence stays stranded in a 2,300-line German document. +**Risk:** may surface more re-additions than expected. That is a benefit, not a cost. + +### CP-11 (High) — Declare Protocol Core vs Protocol Extended, and one hypothesis register +**Change:** split v0.2's content into (a) **Core** — conformance-defining, must be implemented and observable in any binding; and (b) **Extended** — experimental mechanisms explicitly not required for conformance. Merge the five hypothesis ID spaces into one register with a stable ID per hypothesis, its status, its owning document, and its falsification criterion. +**Suggested Core, for discussion:** commitment ladder + recorded commitment act (CP-06); epistemic vs decision status separation; deferred-decision register; persistent state with the nine integrity requirements; decision-rights tiers (`03` §§3–7); adversarial gate (CP-01); recorded disagreement (CP-05); handoff/cold-start artifact. +**Everything else Extended,** including the concept lifecycle, provenance labels, routing dimensions, move taxonomy, probe debrief ritual, and turn economics. +**Rationale:** F-08, F-25, and the ID sprawl noted in §2.2. This single change is what makes a minimal binding and a legitimate Case 03 possible. + +### CP-12 (High) — Define conformance and a minimal trace +**Change:** add `T-11` conformance rules ("a run is v0.2-Core-conformant if …") and `T-12` a minimal event vocabulary (object created / status changed / deferred / challenged / committed / concept reopened / research delta recorded), timestamped and append-only. +**Rationale:** F-25, F-09; converts evaluation from dashboard aspiration to something derivable from a log. + +### CP-13 (Medium) — Redesign evaluation for n=1 +**Change:** replace the run-level metric dashboard as the *primary* instrument with: pre-registered hypotheses (max 5 per case) + predicted observable + falsification criterion, plus three cheap always-on measures (cold-start handoff blockers; decision explainability on 3 sampled decisions; state-integrity defects derivable from the trace). Keep the rest as an explicitly optional diagnostic catalogue. Restate Keep/Investigate/Revert as rules over pre-registered predictions, not over cross-run Pareto comparison. +**Rationale:** F-09, F-13 (and E-05's own concession about practitioner learning). +**Risk:** less impressive-looking evaluation section; far more likely to actually produce a defensible conclusion. + +### CP-14 (Medium) — State the configuration assumption explicitly +**Change:** add to `01` §1 a declared scope: *one human, one AI, single-threaded discovery, terminating in a handoff to a separate implementation context*, plus a short "known untested extensions" list (teams, multiple stakeholders, discovery inside an existing system, continuous discovery→delivery). +**Rationale:** F-14, F-16, F-21. Also makes it honest to import multi-party evidence, by naming the transfer as a transfer. + +### CP-15 (Low/Medium) — Context-scope the concept status; move turn economics to binding guidance +**Change:** (a) make shared-language `status` a per-context field, or state that one entry = one context. (b) `02` §5 (turn economics: "What changed / Why it matters / What is needed next") is a response-format rule — binding guidance, not technology-independent semantics. Same for the YAML blocks, which despite the disclaimer read as a serialization contract. +**Rationale:** F-05, and layer hygiene per D-02. + +--- + +## 6. Questions that genuinely block the next phase + +Only these four actually block. Everything else I can proceed on with a stated assumption. + +**Q1 — Is there a real, available discovery to run as Case 03, and in what domain?** +Not Cookframe, not discovery-method design. If no real project exists in the next weeks, Case 03 becomes a simulation and produces non-evidence — which would make almost all of `05` and the entire learning loop unfalsifiable for another cycle. If none is available, the next phase must be designed differently (retrospective re-analysis of Case 01 under v0.2 semantics, or a deliberately small bounded case), and I would want to know that before proposing anything. + +**Q2 — Who is v1 for: you alone, or third-party practitioners?** +This decides whether the protocol must be teachable and conformance-checkable (CP-11/CP-12 become mandatory, and F-20's population question becomes central), or whether it can remain a personal instrument that is merely *published*. It also decides how much of the public-project work in handover §6 is real work at all. + +**Q3 — Is the harness/model change part of the experiment or a confound to control?** +The baseline was produced with ChatGPT-family models; Case 03 will presumably run in Claude Code. If model family is uncontrolled, some v0.2 mechanisms may be answering a failure profile that a different model exhibits differently (F-17). Options: accept and document; run Case 03 on the original family; or run a small paired probe. + +**Q4 — What is publishable from `evidence/`, and in which language?** +Case 01 is a 2,300-line German document about a real product (Cookframe) with named third-party services. Is the product yours; is any of it confidential; is translation in scope; may it be published as an edited case or only as internal evidence? This gates handover task 3 and also gates CP-10's usefulness to any external reader. + +### Decisions needed soon, where I will assume a default unless corrected + +- **Enforcement posture.** I will assume: *deterministic external enforcement for status transitions and state integrity; advisory/prompt-level for everything else.* (F-15.) +- **Q6 — reviewer independence.** §15/D-21 require an independent adversarial review. This document is not that: same technology class, no separate evidence access. I will assume "independent" means *a different session with no access to this conversation, ideally a different model family, and at least one human reader* — but if you intend §15 to be satisfiable only by a human reviewer, say so, because that changes the release plan. +- **Effort/cadence.** No time, budget or cadence constraints appear anywhere in the pack; I will assume a solo practitioner working in bounded sessions and will size proposals accordingly. + +--- + +## 7. Recommended next concrete phase + +**Recommendation: do not proceed to handover tasks 2–5 as written. Run one phase whose only goal is to make v0.2 testable, and defer all public-project design until after Case 03.** + +Rationale, in the protocol's own terms: repo topology, project name, README architecture and binding architecture are *commitments*, and the evidence that should inform them does not exist yet. Handover §6 schedules them before the first prospective evidence. Committing them now is precisely the `plausible abstraction → polished formulation → repeated use → accidental commitment` sequence that `01` §3 warns about. The pack's own maturity note agrees: *make these mechanisms testable, not merely more elaborate.* + +### Phase P1 — "Make v0.2 testable" (three workstreams, timeboxed) + +**W1 — v0.2.1 consolidation (no new mechanisms).** +Apply the accepted change proposals only. Specifically: CP-11 (Core/Extended split + one hypothesis register), CP-12 (conformance + minimal trace), CP-06 (commitment act), CP-01 (adversarial gate), CP-05 (recorded disagreement), CP-14 (declared configuration), CP-07/CP-08 (evidence honesty), CP-09/CP-10 (analogy records + Case 01 disposition). Net effect should be a *smaller* normative document than v0.2, not a larger one. If v0.2.1 is longer than v0.2, the phase has failed. + +**W2 — Thin instrument, not "the reference binding".** +Build only what makes Core observable: one persistent state file, one append-only event log, and the smallest set of operations that record status transitions, deferrals, challenges and commitments. Harness-specific, deliberately disposable, explicitly *not* the architecture. This is a probe, in the protocol's own vocabulary — it should carry a stated learning question and a discard condition. + +**W3 — Case 03 pre-registration, frozen before the discovery starts.** +Domain and object; declared Core conformance; at most five hypotheses with predicted observables and falsification criteria (my suggested five: criticism surface, enumerated human-first events per CP-03, external state, candidate-first terminology, adversarial gate); the trace plan; the cold-start handoff test design and who runs it. Frozen and committed before the first discovery turn — that freeze is what converts the pack's retrospective evidence into prospective evidence, and it is the single highest-value thing this project can do next. + +**Explicitly deferred through P1:** T-01 (name), T-02 (topology), T-05 (skill/hook architecture), T-06 (review UI), T-07 (evaluator architecture), T-08 (release workflow), and all public README/IA work. + +**Exit criteria for P1:** +1. v0.2.1 exists, is smaller, and its Core is stated as checkable rules. +2. A run can be shown to be Core-conformant or not, from its trace alone. +3. Case 03 pre-registration is frozen, with falsification criteria that could actually come out negative. +4. One independent adversarial pass on v0.2.1 (per Q6) has happened — this document does not count. + +**If Q1 comes back negative** (no real Case 03 domain available): substitute W3 with a re-analysis of Case 01 under v0.2.1 Core semantics — i.e. replay the Cookframe discovery as a conformance and expressiveness test of the protocol, which would at minimum test whether v0.2 can *describe* a real object-level discovery. That is weaker evidence, but it is not fake evidence. + +--- + +## 8. Evaluation of the handover itself + +The handover is unusually good at stating what is *not* proven. It is weak at stating what exists, what was rejected, and under what conditions the work was produced. Concretely, here is what I had to rediscover, infer, or guess. + +### Had to rediscover +1. **Case 01 contains a complete method (M1–M8, G1–G7, L1–L5, AP-01..AP-10, M-01..M-12), not just a retrospective.** The handover describes it in one line: *"A long product discovery produced an implementation handoff."* This concealed the single largest traceability gap in the pack (F-19) and is the one thing I would most want fixed in the next handover. +2. **Case 01 and all four `source-research` reports are in German**; `protocol/`, `research/` and Case 02 are in English. Stated nowhere. It has direct consequences for publication, external review, and citation verification effort. +3. **The strongest disconfirming study in the pack** (Shin et al.) exists only in the raw German research and never propagates upward (F-10). Nothing in the handover flags that curation dropped counter-evidence. +4. **`05` narrows the researched metric set** while keeping a decision rule that references a dropped signal (F-13). +5. **Five separate hypothesis ID spaces** that partly duplicate each other. I had to reconcile them by hand to answer the handover's own question 2. + +### Had to infer +6. **Production conditions:** ChatGPT-family models plus a Deep Research tool, inferred from Case 01's prose. Never stated as a fact about the baseline, and it is a controlled variable for any future comparison (F-17). +7. **Configuration:** one human, one AI, solo practitioner, no team, no client, no deadline. Load-bearing everywhere, declared nowhere (F-14). +8. **That the practitioner, protocol author, evidence analyst and evaluator are the same person.** This is the central threat to validity in the whole evidence apparatus and the handover never names it. +9. **Deliverable form for this task.** Handover §5 says "produce an orientation" and "perform a review" but not what artifact results, where it belongs in the layer model, or whether it is meant to be public. I placed this at project-input level next to the handover; that was a guess. +10. **Whether v0.2's omissions were decisions.** With no "considered and rejected" record, I could not distinguish a deliberate simplification from a loss (F-19). + +### Had to guess +11. **Effort, budget and cadence.** No constraints stated anywhere. +12. **Whether Case 01's product (Cookframe) is publishable at all** — ownership, third-party material, confidentiality (Q4). +13. **What "independent" means for the D-21 review** — different session, different model, or a human (Q6). +14. **The intended relationship between "protocol" and "binding" beyond the README paragraph.** *Binding* is itself an un-canonicalized model-bearing term, which the pack's own `04` policy would flag: it is used normatively across five documents with no entry, no definition of what makes something a binding rather than a usage, and no boundary test. + +### What the baseline should carry next time +A one-page **baseline fact sheet**: document languages; production tooling and model family; dates; who the practitioner is and their relation to the evidence; citation verification status (currently: none verified); one hypothesis register; a "considered and rejected" record; and a definition of the deliverable expected from the next agent. Eight of the fourteen items above would disappear. + +### One structural criticism of the handover +Tasks 1–5 are a waterfall: orient → design the public project → evidence hygiene → research implementation options → implementation-discovery proposal → then evidence. The protocol these tasks serve says *commitment late, probes early and reversible*, and that structure for exploration must not be confused with structure for convergence. The handover's own task ordering front-loads the most committing, least reversible decisions (project name, repo topology, public IA) before the first prospective evidence exists. That is worth noticing precisely because it is the failure mode the protocol was written to prevent — which is also mild evidence for the protocol's central claim. diff --git a/docs/review/disposition.md b/docs/review/disposition.md new file mode 100644 index 0000000..a0eb0f8 --- /dev/null +++ b/docs/review/disposition.md @@ -0,0 +1,84 @@ +# Status Ledger & Change-Proposal Disposition + +> **Published record.** The status ledger kept in the private working repository after the [adversarial review](2026-09-adversarial-review-v0.2.md), reproduced as it stood when this repository was derived (14 September 2026). It records what happened to each of the review's change proposals and to the findings they address, which is the part an outside reader cannot otherwise check. Six findings have no change proposal of their own and are not tracked in the ledger: F-04, F-06, F-12, F-15, F-20 and F-24. Of these, F-12 (no citation independently verified) has since been addressed by [`research/REFERENCES.md`](../../research/REFERENCES.md); the other five have no recorded disposition. +> Paths are those of the working repository: `phase0/observation-model/` is now [`protocol/observation-model/`](../../protocol/observation-model/), `phase0/capabilities/` is now [`capabilities/`](../../capabilities/), and `proposals/CP-09…` / `proposals/CP-11…` are now [`research/analogy-records.md`](../../research/analogy-records.md) and [`research/hypothesis-register.md`](../../research/hypothesis-register.md). Numbered documents (`09`–`15`) are working documents of that repository; [`docs/provenance.md`](../provenance.md) says what each is. This is the public index of change proposals `CP-01`–`CP-21`; the current state of each is kept here from now on. + +> **Status:** Live index for this branch. Update it when a document's standing changes. +> **Why it exists:** an adversarial review of this branch found that documents 10–15 dropped all fifteen change proposals from doc 09 with no record of whether each was accepted, deferred or rejected — which is precisely finding F-19's charge against Protocol v0.2 ("no file records the disposition"). The finding was correct. This file is the fix, and it is the file F-19 says v0.2 is missing. + +--- + +## 1. Document standing + +| Doc | Standing | Notes | +|---|---|---| +| `09` orientation & adversarial review | **live** | two in-place precisions added after review (F-02 scope, F-11 example) | +| `10` project setup & evolution loop | **live** | §0 now states plainly that it overturns doc 09's deferral of T-02 rather than reinterpreting it | +| `11` naming brief | **live, ready to run** | self-block removed; CWS added to the family context | +| `12` capability & eval model / CWS | **live with withdrawals** | §2 stands in full; §3 corrected for overstatement; §4's "directly violated" and "CWS is descriptive" claims withdrawn; §5's two "superseded" rows withdrawn; §6 Option C withdrawn | +| `13` portfolio model | **live** | §8's change table corrected | +| `14` methodplane-state & placement | **live with withdrawals** | §1 and §4: both proposed standards demoted from "earned" to "candidate" | +| `15` portfolio thread parked | **live** | its §6 "settled" list is scoped to design decisions, not to doc 09's findings; those are dispositioned here | +| `phase0/` | **draft, explicitly a probe** | observation model + capability catalogue + empty conformance matrix | + +## 2. Change-proposal disposition + +Doc 09 proposed CP-01..CP-15. None were applied to the protocol; that remains correct — the protocol is untouched. What follows is where each now stands. + +| CP | Proposal | Disposition | Where | +|---|---|---|---| +| CP-01 | adversarial challenge as a convergence gate | **accepted** | drafted as `CAP-10`; protocol edit pending v0.2.1 | +| CP-02 | remove problem/solution split from mode names | **deferred to v0.2.1** | phase 0 deliberately defines no mode capability, so nothing depends on the outcome | +| CP-03 | enumerated anchor-risk events instead of a self-judged trigger | **accepted** | drafted as `CAP-14`, whose L1 measurability depends on the list existing | +| CP-04 | collapse the four tentativeness devices into one ladder | **deferred to v0.2.1** | editorial; no capability depends on it | +| CP-05 | recorded-disagreement object | **accepted** | drafted as `CAP-06` | +| CP-06 | define the commitment act | **accepted** | drafted as `CAP-04`, including the required payload | +| CP-07 | reinstate disconfirming evidence in `research/` | **done** | `research/01` §2A + correction in §2, with an amendment note recording the change | +| CP-08 | one evidence-strength scale, two columns | **deferred** | evidence-layer edit, lower urgency than the publication decision it serves | +| CP-09 | analogy records for the protocol's own four imports | **done** | `proposals/CP-09-analogy-records.md` — four records, four findings, two of them not reconstructable from the pack | +| CP-10 | Case 01 method disposition table | **done** | `proposals/CP-10-case01-method-disposition.md` — 62 elements: 26 kept, 18 transformed, 18 dropped, **0 object-level**. Six new proposals arise (CP-16..CP-21) | +| CP-11 | Core/Extended split + one hypothesis register | **done** | Core/Extended in `phase0/capabilities/catalogue.md`; register in `proposals/CP-11-hypothesis-register.md`, `HYP-001..025` over five legacy ID spaces | +| CP-12 | conformance definition + minimal trace | **executed as a draft** | `phase0/observation-model/` and `phase0/capabilities/CONFORMANCE.md` | +| CP-13 | redesign evaluation for n=1 | **deferred to v0.2.1** | phase 0 supplies the trace it would need; the redesign itself is untouched | +| CP-14 | declare the configuration assumption | **deferred to v0.2.1** | one paragraph in `protocol/01` §1 | +| CP-15 | context-scope concept status; move turn economics to binding guidance | **deferred** | `CAP-11` inherits the unresolved context-scope question and says so | + +**All four outstanding accepted proposals are now done** (CP-07, CP-09, CP-10, CP-11). Five remain deferred to a v0.2.1 consolidation pass (CP-02, CP-04, CP-08, CP-13, CP-14), and executing CP-10 produced six new ones. *[Note added on publication: the table above defers six proposals, four of them to v0.2.1 (CP-02, CP-04, CP-13, CP-14) and two without a target version (CP-08, CP-15).]* + +### New proposals from CP-10 + +| CP | Proposal | Priority | +|---|---|---| +| CP-16 | close the review loop: repair leg, resolution log, scoped re-review | high | +| CP-17 | an in-session stop heuristic | high | +| CP-18 | information-loss gate, generalized beyond terminology | high | +| CP-19 | reuse gate before architecture | medium | +| CP-20 | review timing on cost-of-change; reinstate the dropped open question | medium | +| CP-21 | classify counterexamples local vs structural; define saturation | medium | + +None applied. CP-18 would add `CAP-16` to the catalogue. + +## 3. Priority findings from doc 09 §4 + +| Finding | Status | +|---|---| +| F-02 no adversarial gate | addressed by CP-01 → `CAP-10`; protocol text unchanged | +| F-22 commitment act undefined | addressed by CP-06 → `CAP-04` | +| F-25 no conformance definition | addressed by CP-12 → phase 0, as a draft | +| F-08 no declared core | addressed by CP-11 → Core/Extended proposal in the catalogue | +| F-09 evaluation undecidable at n=1 | **open** — CP-13 deferred | +| F-10 / F-11 curated layer drops counter-evidence | **closed** — CP-07 applied to `research/01` | + +## 4. Corrections applied after the branch review + +Accepted and fixed: the missing disposition record (this file); doc 12 §5's superseded rows and §4's violation claims surviving their own premise's withdrawal; "CWS is descriptive" (its output is prescriptive — what it lacks is an a priori method); doc 14's two standards counted as earned on same-author recurrence, which is the leniency doc 09 refused the protocol; doc 11 still blocking itself and omitting CWS from the family; doc 10 §0 reinterpreting rather than overturning the T-02 deferral; `aos` eval slots are 12, not 13; the `aos` monitor hooks require an instance directory and are not usable "as-is"; loomwise's `eval-baseline` tags the materialized tree, not the tool; the Transfer Release citation; doc 14 §6 describing an impulse stream that does not exist yet; the Q5/Q6 overlap; F-02's and F-11's precision. + +## 5. Two gaps the review named that are now answered + +**Where Phase 0 writes.** Doc 10 §3 gives paths in a repository that does not exist. Phase 0 is drafted in `phase0/` in this repository, mirroring that layout so it moves in one piece once the project has a name and a home. + +**The capability ID scheme.** Doc 10 §2's central mitigation — "the cut is enforced by a lint rule over stable IDs" — had no ID scheme anywhere. It is now `CAP-nn`, allocated in `phase0/capabilities/catalogue.md`, immutable once allocated, never reused after retirement. What the lint would check: every `CAP-nn` referenced under `adapters/**` resolves to a catalogue entry; no file under `protocol/**` names a vendor, product or harness; every Core capability appears in the conformance matrix. + +## 6. Still genuinely blocking + +Unchanged from doc 09 §6, and none of it blocks Phase 0 or the naming: a real, available, non-method, non-Cookframe domain for Evidence Case 03; who v1 is for; whether the model-family change is experiment or confound; what is publishable from `evidence/`. diff --git a/examples/naming-decision.md b/examples/naming-decision.md new file mode 100644 index 0000000..4362023 --- /dev/null +++ b/examples/naming-decision.md @@ -0,0 +1,73 @@ +# Example — a real commitment, recorded in the protocol's own shape + +> **What this is:** the decision to name the project, recorded on 14 September 2026 with the protocol's commitment act (`CAP-04`) and recorded challenge (`CAP-06`). It is a real use of both capabilities on the project's own material, done by hand at enforcement level L0/L1. +> **What this is not:** a discovery run. Nothing was instrumented, and one decision says nothing about whether the protocol improves discovery. It shows what the two records look like and why their fields exist. + +## Why it is worth looking at + +Naming is a framing-bearing commitment, which [`protocol/03`](../protocol/03_DECISION_RIGHTS_AND_AUTONOMY.md) §7 puts in human ownership. The protocol says such a commitment exists only as a recorded transition — with the alternatives it rejected, its rationale, and the condition under which it should be reopened. Anything else is a draft, however confident it sounds. + +The second record matters as much. The AI objected to the name; the owner overruled the objection. Under the protocol an overruled challenge **stays visible**, with its resolution, instead of disappearing from the conversation. If one of the accepted risks materialises later, the record shows it was seen, by whom, and why it was accepted — which is also live material for `HYP-025` in the [hypothesis register](../research/hypothesis-register.md): does recorded dissent predict later reopenings? + +## The commitment + +```yaml +commitment.recorded: + object: project name + value: earnedshape + owner: project owner + date: 2026-09-14 + rationale: > + Discovery exists to develop a form of understanding whose robustness has been + earned through inquiry. The name states a demand on the process rather than a + promise about the result, which is the one thing a name here must not get + wrong: the protocol's own discipline is that most of its mechanisms are + unproven. "Earned" maps directly onto progressive commitment — status is + earned through recorded transitions, never acquired through fluency. + rejected_alternatives: > + Descriptive constructions (discovery-protocol and its family) were rejected + before the brief ran. The full candidate set from the naming session is not + reproduced here; the brief that generated it is working document 11. + evidence_refs: + - naming brief (working document 11, unpublished) + - protocol/01 §3 (progressive commitment, the concept the name carries) + revisit_condition: > + Revisit if either accepted risk below materialises in practice: if the name is + repeatedly misspelled by people who have only heard it, or if readers in the + product community systematically read it as Shape Up adjacent. Before the + plugin/package name is published is the last cheap moment to change it; a + repository rename is cheap indefinitely because GitHub keeps redirects. +``` + +## The recorded challenge + +```yaml +challenge.raised: + id: CH-001 + by: ai + target: project name = earnedshape + summary: > + Two objections. (1) Spellability after hearing it once, which the naming brief + made an explicit criterion: the /d/+/ʃ/ junction collapses in speech, so + listeners plausibly write earnshape, earnt shape or earned shape. German + speakers additionally face the /ɜːr/ vowel and the silent -ed. (2) "Shape" is + occupied in this exact audience by Shape Up (Basecamp), where "shaping" is a + term of art for pre-commitment problem and solution outlining — adjacent to + discovery but meaning something else. Smaller: "shape" foregrounds the exit + artifact while the protocol is mostly about the middle, and "earned" carries a + mild moral register that can age into preachiness. +challenge.resolved: + resolution: overruled + by: owner + reason: the concept is what the name is for, and it fits + status_at_handoff: closed, risks accepted and carried in the revisit condition +``` + +## How to read the fields + +| Field | Why it is there | +|---|---| +| `rejected_alternatives` | A commitment without the alternative it beat cannot be re-examined later: nobody can tell what was traded away. | +| `rationale` | States the reason at the time, so a later reader can judge whether it still holds rather than guessing. | +| `revisit_condition` | Turns "we decided" into "we decided, unless X". It is what keeps a commitment revisable without making it permanently open. | +| `challenge.resolved` · `overruled` | Disagreement is recorded, not resolved by attrition. The objection is kept with the reason it was overruled. | diff --git a/research/01_EVIDENCE_AND_RESEARCH_SYNTHESIS.md b/research/01_EVIDENCE_AND_RESEARCH_SYNTHESIS.md index ab7c731..5908523 100644 --- a/research/01_EVIDENCE_AND_RESEARCH_SYNTHESIS.md +++ b/research/01_EVIDENCE_AND_RESEARCH_SYNTHESIS.md @@ -1,6 +1,6 @@ # Evidence & Research Synthesis — v0.2 -> **Amendment note.** Section 2A and the correction inside §2 were added under change proposal **CP-07** (`09_ORIENTATION_AND_ADVERSARIAL_REVIEW_V0.2.md`), after a review found that this curated layer had dropped the disconfirming evidence present in `source-research/`, and had promoted one contested finding to "strongly supported". No other claim in this file was altered. The underlying sources are unchanged and, as §9 states, still unverified against primaries. +> **Amendment note.** Section 2A and the correction inside §2 were added under change proposal **CP-07** of the [adversarial review](../docs/review/2026-09-adversarial-review-v0.2.md), after it found that this curated layer had dropped the disconfirming evidence present in the raw research reports (`source-research/`, unpublished; see [`docs/provenance.md`](../docs/provenance.md)), and had promoted one contested finding to "strongly supported". No other claim in this file was altered. In September 2026 the sources were checked against publisher and index records; see [`REFERENCES.md`](REFERENCES.md) and §9. ## 1. Evidence architecture @@ -20,7 +20,7 @@ The current baseline is informed by: - Shared Conceptual Language, terminology formation & AI anchoring. ### Prior adjacent design work -- Turn / Adaptive Cognitive Handoff for predominantly autonomous implementation agents. +- Turn / Adaptive Cognitive Handoff for predominantly autonomous implementation agents — the author's earlier, unpublished design work ([`docs/provenance.md`](../docs/provenance.md)). None of these alone validates the full protocol. @@ -29,33 +29,53 @@ None of these alone validates the full protocol. ## 2. Strongly supported directions ### Human–AI collaboration is not automatically synergistic +*Sources:* [Vaccaro2024](REFERENCES.md#vaccaro2024), [Bansal2021](REFERENCES.md#bansal2021) + Task allocation and interaction design matter. ### Human situated input remains epistemically special +*Sources:* [Ferrari2016](REFERENCES.md#ferrari2016), [Ferrari2022](REFERENCES.md#ferrari2022) — support is narrower than the heading; see [REFERENCES.md](REFERENCES.md#where-the-support-is-thinner-than-the-wording). + But human experience is evidence, not infallible truth. ### AI can be highly valuable for articulation, search, synthesis and critique +*Sources:* [Mircea2026](REFERENCES.md#mircea2026), [Noy2023](REFERENCES.md#noy2023) — articulation and drafting only; see [REFERENCES.md](REFERENCES.md#where-the-support-is-thinner-than-the-wording). + The correct response to AI anchoring risk is not "AI stays silent". ### Early AI output can influence human exploration/agency +*Sources:* [Qin2025](REFERENCES.md#qin2025), [Wadinambiarachchi2024](REFERENCES.md#wadinambiarachchi2024), [Doshi2024](REFERENCES.md#doshi2024) + Timing and presentation matter. ### Problem and solution can co-evolve +*Sources:* [DorstCross2001](REFERENCES.md#dorstcross2001) + The important protection is against premature commitment. ### Shared terminology reduces coordination cost +*Sources:* [BrennanClark1996](REFERENCES.md#brennanclark1996), [PickeringGarrod2004](REFERENCES.md#pickeringgarrod2004), [GlinzFricker2015](REFERENCES.md#glinzfricker2015) — laboratory tasks, not measured in discovery work. + But lexical agreement does not prove conceptual agreement. ### Language and model can co-evolve +*Sources:* [Evans2003](REFERENCES.md#evans2003) — practitioner literature only. + Important terminology can be part of the conceptual model, not just labels applied afterward. ### Human-computer lexical alignment is real +*Sources:* [Branigan2011](REFERENCES.md#branigan2011), [Ostrand2023](REFERENCES.md#ostrand2023) + This creates a plausible path for AI-originated terminology to become default without explicit conceptual agreement. ### Explanations alone do not guarantee appropriate reliance +*Sources:* [Bansal2021](REFERENCES.md#bansal2021), [Bucinca2021](REFERENCES.md#bucinca2021) + Selective cognitive engagement matters. ### Structured scaffolding can outperform generic chat — *contested* +*Sources:* [Heyman2024](REFERENCES.md#heyman2024), [Luan2025](REFERENCES.md#luan2025); against: [Wadinambiarachchi2024](REFERENCES.md#wadinambiarachchi2024) + One experiment (Supermind Ideator) found a structured GenAI scaffold produced solutions rated more innovative than both generic chat and human-only work. `source-research/01` rates this evidence **medium and explicitly contradictory**, because the fixation studies point the other way on adjacent tasks. It was previously listed here without that qualification, which overstated it. It remains the most direct support for the protocol's central bet, and it comes from ideation, not discovery. --- @@ -65,14 +85,20 @@ One experiment (Supermind Ideator) found a structured GenAI scaffold produced so A curated layer that only carries supporting evidence is not a synthesis. This section exists so that the strongest counter-evidence is visible at the same altitude as the support, and it is a standing section: **every future research pass must either add to it or state explicitly that it found nothing to add.** ### The closest study to the target task found no benefit +*Sources:* [Shin2025](REFERENCES.md#shin2025) + Shin et al. (CHI 2025, N=280) compared free LLM use, direct frame generation, a theory-guided structured LLM method, and a control condition. It found **no improvement in problem-frame quality** from LLM use; the competence gap between experienced and inexperienced designers **widened**; and inexperienced participants reported **lower perceived agency**. Of all the evidence in this pack, this is the study nearest to what this protocol is for — problem framing, by individuals, with an LLM — and it is negative. It appears in `source-research/04` and had not previously reached this layer. ### Human–AI combination is on average worse than the better single actor +*Sources:* [Vaccaro2024](REFERENCES.md#vaccaro2024) + Vaccaro et al. (2024) is cited in §2 as "not automatically synergistic". Stated fully: across 106 experiments, mean synergy relative to the better of human-alone and AI-alone was **negative** (Hedges' g ≈ −0.23), with losses concentrated in decision tasks. The optimistic reading — that design of the coupling is what matters — is an inference from the heterogeneity, not a finding. ### The mechanism this protocol relies on is disliked by the people it helps +*Sources:* [Bucinca2021](REFERENCES.md#bucinca2021) + Cognitive forcing reduced overreliance more than explainable-AI variants, and was rated **worse** by users (Buçinca et al. 2021). This protocol deliberately adds friction at high-consequence points. The evidence says that will work and be unpopular, which is a real adoption risk, not only a design trade-off. ### What follows honestly @@ -194,14 +220,8 @@ The next important evidence is a genuinely independent Case 03. ## 9. Public citation hygiene -The raw Deep Research reports in `source-research/` contain internal tool citation markers. - -Before public publication: +The raw research reports (unpublished, see [`docs/provenance.md`](../docs/provenance.md)) carry internal citation markers of the tool that produced them. They remain discovery input, not public literature reviews. -- re-open primary sources; -- verify study details; -- replace internal citations with stable DOI/URL references; -- mark preprints and practitioner sources; -- distinguish external evidence from protocol inference. +Done on 22 September 2026: every source this file and `02` rely on was looked up again and is listed with a stable DOI or URL, marked peer-reviewed, preprint or practitioner, in [`REFERENCES.md`](REFERENCES.md). The same check located the claims in §2–§5 whose support is thinner than their wording; they are listed there and in [`docs/known-findings.md`](../docs/known-findings.md) (KF-3), not silently reworded. -The raw reports are discovery evidence input, not public-ready literature reviews. +Still open: reading the key papers in full rather than at abstract level, and re-checking the few fields marked unconfirmed in `REFERENCES.md`. diff --git a/research/02_SHARED_LANGUAGE_RESEARCH_SYNTHESIS.md b/research/02_SHARED_LANGUAGE_RESEARCH_SYNTHESIS.md index 97bd5a9..5980294 100644 --- a/research/02_SHARED_LANGUAGE_RESEARCH_SYNTHESIS.md +++ b/research/02_SHARED_LANGUAGE_RESEARCH_SYNTHESIS.md @@ -1,5 +1,7 @@ # Shared Language Research — Protocol Implications +> **Sources** for the research claims below are listed in [`REFERENCES.md`](REFERENCES.md), with a table of which claim rests on which study. Two claims here are weaker than they read — the effect of labels on attention, and LLM reformulation revealing implicit content — and two are protocol inferences rather than findings (candidate terms as semantic probes; the failure-mode list); see [the notes there](REFERENCES.md#where-the-support-is-thinner-than-the-wording). + ## Decision Shared Conceptual Language should enter Protocol v0.2 as a **first-class cross-cutting concern**, but not as: diff --git a/research/REFERENCES.md b/research/REFERENCES.md new file mode 100644 index 0000000..ee8e85c --- /dev/null +++ b/research/REFERENCES.md @@ -0,0 +1,128 @@ +# References + +Checked sources for the claims in [`01_EVIDENCE_AND_RESEARCH_SYNTHESIS.md`](01_EVIDENCE_AND_RESEARCH_SYNTHESIS.md) and [`02_SHARED_LANGUAGE_RESEARCH_SYNTHESIS.md`](02_SHARED_LANGUAGE_RESEARCH_SYNTHESIS.md). + +The synthesis was first written from four unpublished research reports produced with a deep-research tool (see [`docs/provenance.md`](../docs/provenance.md)). On 22 September 2026 every source the public text relies on was looked up again at the publisher or in an index record. Every cited work exists, and title, year, venue and DOI match what is given here. Where a field could not be confirmed (some author lists, one year, some volume and page numbers, one study's sample size), the entry says "not confirmed" and why. Numbers stated in the synthesis were checked against the paper where the entry says so. The check found one wrong author list in the unpublished reports (Bansal et al. 2021, corrected here) and no mismatched numbers. + +**How to read the tags.** *peer-reviewed*, *preprint* or *practitioner* (books and method literature, not controlled evidence). The note after each entry says how it was checked — **C**: publisher metadata retrieved; **S**: found at the publisher or an index whose record matches, DOI confirmed where the publisher URL carries it — and names any field that could not be confirmed. + +**What this check did not do.** It did not re-read the papers in full, and it did not run a new literature search. It establishes that the cited work exists and says what the synthesis says it says at the level of its abstract. Where that is weaker than the synthesis's wording, the section [Where the support is thinner than the wording](#where-the-support-is-thinner-than-the-wording) says so. + +## References + +### Human–AI collaboration, reliance and creativity + +- **[Vaccaro2024]** Vaccaro, M., Almaatouq, A., & Malone, T. W. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. *Nature Human Behaviour, 8*, 2293–2303. https://doi.org/10.1038/s41562-024-02024-1 **[peer-reviewed]** (C; the abstract confirms 106 experiments, 370 effect sizes, Hedges' g = −0.23 against the best of human or AI alone, losses in decision tasks and gains in content creation.) +- **[Shin2025]** Shin, J., Polyanskaya, A., Lucero, A., & Oulasvirta, A. (2025). No evidence for LLMs being useful in problem reframing. In *Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25)*. ACM. https://doi.org/10.1145/3706598.3713273 **[peer-reviewed]** (C. Preprint: arXiv:2503.01631. The abstract gives N=280 and "three ways of using LLMs" (free-form, direct generation, a structured approach informed by a theory of reframing); the full text adds a manual no-LLM baseline, which makes four conditions.) +- **[Bucinca2021]** Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. *Proceedings of the ACM on Human-Computer Interaction, 5*(CSCW1), Article 188. https://doi.org/10.1145/3449287 **[peer-reviewed]** (S; ACM DL URL carries the DOI. Preprint: arXiv:2102.09692.) +- **[Bansal2021]** Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In *Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21)*. ACM. https://doi.org/10.1145/3411764.3445717 **[peer-reviewed]** (C. The author list in source report 03 is wrong; it is corrected here. See (c).) +- **[Raees2026]** Raees, M., Khan, V. J., Lykourentzou, I., & Papangelis, K. (2026). Do people appropriately rely on AI-advice? An analytical review of HCI research on human-AI decision-making. In *Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26)*, Article 805. ACM. https://doi.org/10.1145/3772318.3791467 **[peer-reviewed]** (S; ACM DL URL carries the DOI; title and venue confirmed. Authors and article number not confirmed: they come from the source report.) +- **[Heyman2024]** Heyman, J. L., Rick, S. R., Giacomelli, G., Wen, H., Laubacher, R. J., Taubenslag, N., Knicker, M. S., Jeddi, Y., Ragupathy, P., Curhan, J., & Malone, T. W. (2024). Supermind Ideator: How scaffolding human-AI collaboration can increase creativity. *Collective Intelligence, 3*(4). https://doi.org/10.1177/26339137241305117 **[peer-reviewed]** (C; the abstract confirms "significantly more innovative ideas than … ChatGPT or people working alone". An earlier conference version exists: ACM CI 2024, https://doi.org/10.1145/3643562.3672611.) +- **[Luan2025]** Luan, Y. L., Kim, Y. J., & Zhou, J. (2025). Augmented learning for joint creativity in human-GenAI co-creation. *Information Systems Research*. https://doi.org/10.1287/isre.2024.0984 **[peer-reviewed]** (S; INFORMS URL carries the DOI. Authors not confirmed: they come from the source report.) +- **[Wadinambiarachchi2024]** Wadinambiarachchi, S., Kelly, R. M., Pareek, S., Zhou, Q., & Velloso, E. (2024). The effects of generative AI on design fixation and divergent thinking. In *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24)*, Article 380. ACM. https://doi.org/10.1145/3613904.3642919 **[peer-reviewed]** (C. Preprint: arXiv:2403.11164.) +- **[Qin2025]** Qin, P., Yang, C.-L., Li, J., Wen, J., & Lee, Y.-C. (2025). Timing matters: How using LLMs at different timings influences writers' perceptions and ideation outcomes in AI-assisted ideation. In *Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25)*. ACM. https://doi.org/10.1145/3706598.3713146 **[peer-reviewed]** (C. Preprint: arXiv:2502.06197.) +- **[Doshi2024]** Doshi, A. R., & Hauser, O. P. (2024). Generative AI enhances individual creativity but reduces the collective diversity of novel content. *Science Advances, 10*, eadn5290. https://doi.org/10.1126/sciadv.adn5290 **[peer-reviewed]** (S; science.org URL carries the DOI.) +- **[Noy2023]** Noy, S., & Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. *Science, 381*(6654), 187–192. https://doi.org/10.1126/science.adh2586 **[peer-reviewed]** (S; science.org URL carries the DOI.) +- **[Dhillon2024]** Dhillon, P. S., Molaei, S., et al. (2024). Shaping human-AI collaboration: Varied scaffolding levels in co-writing with language models. In *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24)*. ACM. https://doi.org/10.1145/3613904.3642134 **[peer-reviewed]** (S; ACM DL URL carries the DOI. Authors after the first two not confirmed: the index record lists only those two.) +- **[Jakesch2023]** Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023). Co-writing with opinionated language models affects users' views. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23)*. ACM. https://doi.org/10.1145/3544548.3581196 **[peer-reviewed]** (S; ACM DL URL carries the DOI. Preprint: arXiv:2302.00560. N=1,506 not confirmed: it comes from the source report and was not re-checked against the paper.) +- **[Agarwal2025]** Agarwal, D., Naaman, M., & Vashistha, A. (2025). AI suggestions homogenize writing toward Western styles and diminish cultural nuances. In *Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25)*. ACM. https://doi.org/10.1145/3706598.3713564 **[peer-reviewed]** (S; ACM DL URL carries the DOI. Preprint: arXiv:2409.11360.) + +### Requirements, elicitation and shared understanding + +- **[Ferrari2016]** Ferrari, A., Spoletini, P., & Gnesi, S. (2016). Ambiguity and tacit knowledge in requirements elicitation interviews. *Requirements Engineering, 21*(3), 333–355. https://doi.org/10.1007/s00766-016-0249-3 **[peer-reviewed]** (S; ACM DL URL carries the DOI. Volume and pages not confirmed: they could not be checked against the publisher record. The conference version appeared at RE 2015.) +- **[Ferrari2022]** Ferrari, A., Spoletini, P., & Debnath, S. (2022). How do requirements evolve during elicitation? An empirical study combining interviews and app store analysis. *Requirements Engineering*. https://doi.org/10.1007/s00766-022-00383-7 **[peer-reviewed]** (S for the work and for arXiv:2208.00825; the DOI is confirmed on the published article. Volume and pages not confirmed: the publisher page could not be opened from here.) +- **[Mircea2026]** Mircea, M., Gevrek, E., Schmid, E., & Schneider, K. (2026). Supporting stakeholder requirements expression with LLM revisions: An empirical evaluation. In *Requirements Engineering: Foundation for Software Quality (REFSQ 2026)*, LNCS 16497, 303–319. Springer. https://doi.org/10.1007/978-3-032-21423-2_21 **[peer-reviewed]** (S; preprint arXiv:2601.16699. The abstract confirms 26 participants and 130 requirement statements, each paired with an LLM revision; the body gives 43 % of revisions surfacing additional aspects, 35 % improving understanding and 5 % introducing inaccuracies.) +- **[GlinzFricker2015]** Glinz, M., & Fricker, S. A. (2015). On shared understanding in software engineering: An essay. *Computer Science – Research and Development, 30*(3–4). https://doi.org/10.1007/s00450-014-0256-x **[peer-reviewed]** (S; Springer and ACM URLs carry the DOI. This is an essay, i.e. a conceptual paper, not an empirical study.) + +### Design research: co-evolution, fixation, prototyping + +- **[DorstCross2001]** Dorst, K., & Cross, N. (2001). Creativity in the design process: Co-evolution of problem–solution. *Design Studies, 22*(5), 425–437. https://doi.org/10.1016/S0142-694X(01)00009-6 **[peer-reviewed]** (S; the ScienceDirect PII S0142694X01000096 matches the DOI.) +- **[JanssonSmith1991]** Jansson, D. G., & Smith, S. M. (1991). Design fixation. *Design Studies, 12*(1), 3–11. https://doi.org/10.1016/0142-694X(91)90003-F **[peer-reviewed]** (S; the PII matches the DOI.) +- **[Dow2010]** Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010). Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. *ACM Transactions on Computer-Human Interaction, 17*(4), Article 18. https://doi.org/10.1145/1879831.1879836 **[peer-reviewed]** (S; ACM DL URL carries the DOI.) +- **[HoudeHill1997]** Houde, S., & Hill, C. (1997). What do prototypes prototype? In M. Helander, T. K. Landauer, & P. Prabhu (Eds.), *Handbook of Human-Computer Interaction* (2nd ed.). Elsevier. https://doi.org/10.1016/B978-044481862-1.50082-0 **[peer-reviewed book chapter]** (S; the ScienceDirect PII matches the DOI.) +- **[Lim2008]** Lim, Y.-K., Stolterman, E., & Tenenberg, J. (2008). The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. *ACM Transactions on Computer-Human Interaction, 15*(2), Article 7. https://doi.org/10.1145/1375761.1375762 **[peer-reviewed]** (S; ACM DL URL carries the DOI.) + +### Psycholinguistics: alignment, conceptual pacts, labels + +- **[BrennanClark1996]** Brennan, S. E., & Clark, H. H. (1996). Conceptual pacts and lexical choice in conversation. *Journal of Experimental Psychology: Learning, Memory, and Cognition, 22*(6), 1482–1493. https://doi.org/10.1037/0278-7393.22.6.1482 **[peer-reviewed]** (S; a search on the DOI string returns PubMed 8921603 and the APA record.) +- **[PickeringGarrod2004]** Pickering, M. J., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. *Behavioral and Brain Sciences, 27*(2), 169–190. https://doi.org/10.1017/S0140525X04000056 **[peer-reviewed]** (S; a search on the DOI string returns PubMed 15595235 and Cambridge Core.) +- **[Branigan2011]** Branigan, H. P., Pickering, M. J., Pearson, J., McLean, J. F., & Brown, A. (2011). The role of beliefs in lexical alignment: Evidence from dialogs with humans and computers. *Cognition, 121*(1), 41–57. https://doi.org/10.1016/j.cognition.2011.05.011 **[peer-reviewed]** (S; ScienceDirect and ERIC records match, including volume, issue and pages.) +- **[Ostrand2023]** Ostrand, R., Ferreira, V. S., & Piorkowski, D. (2023). Rapid lexical alignment to a conversational agent. In *Proceedings of Interspeech 2023* (pp. 2653–2657). ISCA. https://doi.org/10.21437/Interspeech.2023-2332 **[peer-reviewed]** (S; ISCA archive.) +- **[Lupyan2007]** Lupyan, G., Rakison, D. H., & McClelland, J. L. (2007). Language is not just for talking: Redundant labels facilitate learning of novel categories. *Psychological Science, 18*(12), 1077–1083. https://doi.org/10.1111/j.1467-9280.2007.02028.x **[peer-reviewed]** (S; the SAGE URL carries the DOI.) +- **[Gervits2023]** Gervits, F., Johanson, M., & Papafragou, A. (2023). Relevance and the role of labels in categorization. *Cognitive Science, 47*(12), e13395. https://doi.org/10.1111/cogs.13395 **[peer-reviewed]** (S; PubMed record 38148613.) +- **[BoutonnetLupyan2015]** Boutonnet, B., & Lupyan, G. (2015). Words jump-start vision: A label advantage in object recognition. *Journal of Neuroscience, 35*(25), 9329–9335. https://www.jneurosci.org/content/35/25/9329 **[peer-reviewed]** (S. Its early ERP effects did not replicate in [NonReplication2021](#nonreplication2021).) +- **[NonReplication2021]** Words may jump-start meaning more than vision: A non-replication of early ERP effects in Boutonnet and Lupyan (2015). (2021). *Collabra: Psychology, 7*(1), 29763. University of California Press. https://online.ucpress.edu/collabra/article/7/1/29763/119070 **[peer-reviewed]** (S; title, journal, volume and article number from the publisher URL. Authors and DOI not confirmed: the publisher page could not be read reliably from here.) + +### Method and practitioner literature + +- **[Evans2003]** Evans, E. (2003). *Domain-driven design: Tackling complexity in the heart of software*. Addison-Wesley. ISBN 978-0-321-12521-7. **[practitioner]** (S.) +- **[HoferSchwentner2021]** Hofer, S., & Schwentner, H. (2021). *Domain storytelling: A collaborative, visual, and agile way to build domain-driven software*. Addison-Wesley. ISBN 978-0-13-745891-2. **[practitioner]** (S. Year not confirmed: the e-book is listed as 2021 and some listings say 2022.) + +### Checked, not currently cited + +- **[deRooij2026]** de Rooij, A., & Biskjaer, M. M. (2026). Does generative AI make us think alike? A systematic review and meta-analysis of homogenisation effects in human–AI co-creation. *Behaviour & Information Technology*. https://doi.org/10.1080/0144929X.2026.2726451 **[peer-reviewed]** (S; authors confirmed in the Tilburg University research portal. The DOI comes from the Taylor & Francis URL; the article page itself could not be opened, and volume and issue are not confirmed. A preprint exists. This could add weight to §2 "Early AI output…".) +- **[SalgadoNeto2026]** Salgado Neto, M., Araujo, A., & de Souza Santos, R. (2026). Collaborative and AI-supported requirements elicitation: An empirical study. arXiv:2606.24060. https://arxiv.org/abs/2606.24060 **[preprint]** (S; exists and the title matches. It is the only source in the pack for "AI is valuable for synthesis".) + +--- + +## Which claim rests on which source + +| File | Section / claim | Keys | Notes | +|---|---|---|---| +| research/01 | §2 Human–AI collaboration is not automatically synergistic | Vaccaro2024; Bansal2021 | | +| research/01 | §2 Human situated input remains epistemically special / evidence, not infallible truth | Ferrari2016; Ferrari2022 | These support "co-constructed, not simply read off" and tacit knowledge. "Epistemically special" is an inference (see below). | +| research/01 | §2 AI valuable for articulation, search, synthesis, critique | Mircea2026; Noy2023 (optional: Dhillon2024) | Articulation and drafting are supported; synthesis rests on one preprint (SalgadoNeto2026, not cited in the synthesis); search and critique are not covered (see below). | +| research/01 | §2 Early AI output can influence exploration/agency; timing matters | Qin2025; Wadinambiarachchi2024 (optional: Doshi2024, deRooij2026) | | +| research/01 | §2 Problem and solution can co-evolve | DorstCross2001 | | +| research/01 | §2 Shared terminology reduces coordination cost; lexical ≠ conceptual agreement | BrennanClark1996; PickeringGarrod2004; GlinzFricker2015 | Lab referential tasks plus a conceptual essay; no measurement in discovery settings. | +| research/01 | §2 Language and model can co-evolve | Evans2003 | Practitioner source only (see below). | +| research/01 | §2 Human-computer lexical alignment is real | Branigan2011; Ostrand2023 | | +| research/01 | §2 Explanations alone do not guarantee appropriate reliance | Bansal2021; Bucinca2021 | | +| research/01 | §2 Structured scaffolding can outperform generic chat (contested) | Heyman2024; Luan2025; counter-evidence: Wadinambiarachchi2024 | | +| research/01 | §2A Closest study found no benefit (N=280, four conditions, frame quality, gap widened, lower agency) | Shin2025 | | +| research/01 | §2A Human–AI combination on average worse than best single actor (106 experiments, g ≈ −0.23, decision tasks) | Vaccaro2024 | Numbers match the abstract. | +| research/01 | §2A Cognitive forcing reduces overreliance but is rated worse | Bucinca2021 (optional: Raees2026) | | +| research/01 | §3 Contributions as epistemic/normative roles, not computational superiority | Vaccaro2024; Ferrari2022; Qin2025 (for Independent Position) | This is synthesis and inference (see below). | +| research/01 | §4 Commitment late; probes early and reversible; probe feeds back into frame | DorstCross2001; Dow2010; HoudeHill1997; Lim2008; JanssonSmith1991 | The Probe ≠ Candidate ≠ … chain is a protocol construct. | +| research/01 | §5 AI may name early | Mircea2026; Lupyan2007 | | +| research/01 | §5 Model-bearing terminology begins as proposal | Branigan2011; Jakesch2023 | Risk rationale only; a design choice, not a tested rule. | +| research/01 | §5 Important human source meaning may need preservation | Ferrari2022; Mircea2026 (semantic drift) | | +| research/01 | §5 Scenario/boundary use can test meaning | Evans2003; HoferSchwentner2021 | Practitioner sources only. | +| research/01 | §5 Canonical language contextual and revisable | BrennanClark1996; Evans2003 | | +| research/01 | §5 No full glossary is justified | none | Absence of evidence (see below). | +| research/02 | "Meaning before commitment" (core refinement) | BrennanClark1996; Mircea2026; Evans2003 | Synthesis. | +| research/02 | Humans naturally converge lexically with partners | BrennanClark1996; PickeringGarrod2004 | | +| research/02 | Humans can align strongly with computer/agent language | Branigan2011; Ostrand2023 | | +| research/02 | Labels can influence categorization and attention | Lupyan2007; Gervits2023; BoutonnetLupyan2015 | The attention/perception part needs a caveat (see below). | +| research/02 | LLM suggestions can affect later expression, ideation, reported attitudes | Jakesch2023; Agarwal2025; Qin2025 | | +| research/02 | Early LLM involvement can reduce independent ideation/ownership | Qin2025 | | +| research/02 | LLM reformulation can improve articulation and reveal implicit content | Mircea2026 | Single study (see below). | +| research/02 | Correct goal is calibrated co-construction, not AI silence | Heyman2024; Dhillon2024; Qin2025 | Synthesis. | +| research/02 | Candidate terms as semantic probes | none | Protocol inference. | +| research/02 | Failure modes: false shared understanding; lexical without conceptual alignment | GlinzFricker2015; PickeringGarrod2004 | | +| research/02 | Failure mode: semantic drift | Mircea2026 | | +| research/02 | Failure mode: context collision (bounded contexts) | Evans2003 | Practitioner source. | +| research/02 | Why this matters with AI: AI is not automatically a good reframing partner (implicit) | Shin2025 | Research/02 does not currently cite Shin; adding it would be sensible. | + +--- + +## Where the support is thinner than the wording + +The citation check found claims in `research/01` §2 — the section headed "strongly supported" — and in `research/02` whose support is narrower than their wording. They are listed here rather than silently reworded, and recorded as known finding [KF-3](../docs/known-findings.md). + +| Claim | What the sources actually support | +|---|---| +| 01 §2 *Language and model can co-evolve* | Practitioner method literature only (Evans 2003). No empirical study. | +| 01 §2 *AI can be highly valuable for articulation, search, synthesis and critique* | Articulation and drafting (one requirements study; Noy & Zhang 2023). Nothing in the evidence covers search or critique; synthesis rests on one preprint ([SalgadoNeto2026](#salgadoneto2026)), which the synthesis does not cite. | +| 01 §2 *Human situated input remains epistemically special* | Requirements are co-constructed and partly tacit (Ferrari et al.). "Epistemically special" is the project's normative reading, not a finding. | +| 01 §2 *Shared terminology reduces coordination cost* | Shown in laboratory referential tasks (Brennan & Clark 1996). Not measured in discovery or software work. | +| 01 §2 *Selective cognitive engagement matters* | Buçinca et al. tested cognitive forcing, not selective forcing. "Selective" is inference. | +| 01 §3 contributions as epistemic/normative roles | Interpretation, consistent with Vaccaro et al. 2024; no study tests it. | +| 01 §4 *Probe ≠ Candidate ≠ Recommendation ≠ Decision ≠ Commitment* | A protocol construct built on co-evolution, parallel prototyping and fixation research; the construct itself is untested. | +| 01 §5 *no full glossary is justified* | Absence of evidence for a glossary, not evidence against one. | +| 01 §5 *scenario/boundary use can test meaning* | Practitioner literature only. | +| 02 *labels can influence categorization and attention* | Categorization: supported (Lupyan et al. 2007; Gervits et al. 2023), with task-dependent effects. Attention/perception rests on work whose early effects did not replicate ([NonReplication2021](#nonreplication2021)). | +| 02 *LLM reformulation can reveal implicit content* | One study, in structured requirements writing rather than open discovery. | +| 02 *candidate terms as semantic probes*; the failure-mode list | Protocol inference, not findings. | + +Two smaller points: `01` §2A says Vaccaro et al. are cited in §2 — they now are, through this file. And the statement that the meta-analysis "predates most current models" has not been checked against its inclusion window. For Shin et al. 2025, the sample size (N=280) is confirmed in the abstract, and the four conditions in the full text (three LLM approaches plus a manual baseline). diff --git a/research/analogy-records.md b/research/analogy-records.md new file mode 100644 index 0000000..aeb2040 --- /dev/null +++ b/research/analogy-records.md @@ -0,0 +1,152 @@ +# CP-09 — Analogy Records for the Protocol's Own Imports + +> **Published record of change proposal CP-09** (see [`docs/review/disposition.md`](../docs/review/disposition.md)). The protocol requires an analogy boundary check for imported frameworks; this file applies it to the protocol's own four imports. `source-research/03` is one of the unpublished research reports described in [`docs/provenance.md`](../docs/provenance.md). The fixes it proposes are not yet applied to `protocol/`. + +> **Status:** Proposal artifact. Candidate appendix to `protocol/`, not applied there. +> **Executes:** change proposal CP-09, addressing finding F-18 — the protocol mandates an Analogy Boundary Check (`01_PROTOCOL_V0.2` §11, schema in `02_INTERACTION_AND_STATE_MODEL` §10) and has never performed one on its own four imports. +> **Also:** the cheapest available exercise of `CAP-15`, on material already at hand. + +A check that finds nothing is theatre. §5 states what these four found. + +--- + +## A-01 · Domain-Driven Design / Ubiquitous Language + +```yaml +source: Domain-Driven Design; Ubiquitous Language; Bounded Context +original_problem: > + Keeping a team of developers and domain experts aligned on one model, + over years, in a codebase, where the model IS the code. +similarities: + - language and model co-evolve; a rename signals a model change + - "canonical" is bounded by context, not global + - forced lexical uniformity across contexts is a mistake +important_differences: + - DDD aligns a multi-party team with shared incentives and persistent memory; + this protocol governs a two-party dyad where one party has neither + - in DDD the compiler is a witness: a term used wrongly breaks something. + In a conversation nothing witnesses misuse + - DDD's contexts are organisational and architectural; a discovery's + "context" is a conversational scope with no enforcement surface + - DDD assumes the model will be built; discovery may end in a decision not to +transferable_mechanisms: + - language/model co-evolution + - context boundaries instead of forced uniformity + - rename, split and merge as evidence of an improved model +non_transferable_assumptions: + - that repeated use demonstrates shared meaning — in DDD it is checked by code + - that a glossary is maintained by parties with a shared stake in its accuracy + - that "ubiquitous" is achievable, or desirable, inside an exploratory dialogue +``` + +**Defect surfaced:** DDD and Ubiquitous Language appear **nowhere** in `protocol/` or `research/`, although `04_SHARED_LANGUAGE_POLICY` §§7–9 rest on them. The import is invisible above `source-research/04`. An unnamed import cannot be boundary-checked by a reader, which is how it stayed unchecked. + +--- + +## A-02 · Common ground, lexical entrainment, conceptual pacts + +```yaml +source: psycholinguistics of dialogue — common ground, lexical entrainment, conceptual pacts +original_problem: > + Explaining how two humans establish reference in dialogue, largely in + referential communication tasks with concrete referents, over minutes. +similarities: + - a pact is partner-specific, local and revisable + - shared words do not entail shared concepts +important_differences: + - the studied process is symmetric between peers; a human and an LLM adapt + asymmetrically, and mostly in one direction — toward the model's register + - both human partners have persistent memory and a stake in being understood; + the model has neither and cannot hold a pact across sessions + - the referents are concrete and jointly visible; product concepts are neither + - the timescale is minutes, not weeks +transferable_mechanisms: + - lexical agreement is not conceptual agreement + - pacts are local and revisable rather than definitional +non_transferable_assumptions: + - that entrainment implies mutual adaptation + - that a pact, once established, persists in both parties + - that the effect sizes observed on concrete referents transfer to abstract + model-bearing terminology +``` + +**Consequence:** the asymmetry cuts both ways. It strengthens the candidate-first rule — the human drifts toward the model's language more than the reverse. It also weakens the transfer of the entrainment evidence itself, which `research/02` leans on more heavily than the difference in symmetry supports. + +--- + +## A-03 · Turn (prior work: adaptive cognitive handoff) + +```yaml +source: Turn — adaptive cognitive handoff for predominantly autonomous implementation agents +original_problem: > + Placing human intervention well in work that an agent otherwise executes + alone, where the goal is throughput at acceptable quality. +similarities: + - initiative as a property of the move, not a role + - handoff as a first-class object + - human attention treated as a scarce resource +important_differences: + - in implementation the human's involvement is overhead to be minimised; + in discovery the human's cognition IS the product + - implementation work is externally verifiable — tests, builds, runtime; + discovery has no such witness + - Turn's success measure is throughput with maintained quality; discovery's + is understanding, breadth before commitment, and a handoff that holds +transferable_mechanisms: + - dynamic initiative + - the handoff artifact + - explicit decision rights per move +non_transferable_assumptions: + - that human involvement is a cost to be reduced + - that the work object can be verified without the human + - that increasing autonomy is the direction of improvement +``` + +**Defect surfaced — the important one.** `D-04 "Human attention is scarce"` is a Turn-derived premise. In Turn's setting, minimising human involvement is the goal. In discovery, minimising human involvement is a **failure mode** — it is the thing the protocol's own governing principle exists to prevent. The protocol currently holds both: attention scarcity as a design driver, and "preserve the cognition that materially improves quality" as the objective. They are not identical, and where they conflict — an expensive elicitation that the human would rather skip — the protocol does not say which wins. Evidence Case 02 recorded that Turn was over-assimilated once (E-02-07); this is a second instance, and it survived into v0.2 unexamined. + +--- + +## A-04 · Cognitive forcing functions + +```yaml +source: cognitive forcing functions in AI-assisted decision-making +original_problem: > + Reducing overreliance on AI advice in single discrete decisions that have a + ground truth, in short sessions with non-expert users. +similarities: + - deliberate friction improves appropriate reliance + - explanations and confidence scores alone do not +important_differences: + - those tasks have an accuracy criterion; a discovery decision has none + available at the time it is made + - the decision is one moment; a discovery commitment is distributed over weeks + - the studies separate decider from evaluator; here they are the same person + - the effective designs were rated worse by users — and a solo practitioner + can simply switch them off +transferable_mechanisms: + - friction proportional to consequence + - explanation is not calibration +non_transferable_assumptions: + - that overreliance is detectable in the moment + - that a fixed forcing schedule is appropriate to open-ended work + - that an external accuracy criterion exists against which reliance is judged +``` + +**Consequence:** the missing ground truth is why `Beneficial AI Uptake / Defective AI Resistance` — recommended by `source-research/03` and dropped from `protocol/05` §3 — is not implementable as specified. Finding F-13 flagged that drop as undocumented. It now looks **justified but unrecorded**: the right fix is a note saying why, not reinstating the metric. Seeded defective cases would be the only way to measure it, and that is a controlled-study instrument, not a run-level signal. + +--- + +## 5. What this exercise found + +Four checks, four findings, none of them cosmetic: + +1. **An invisible import.** DDD/Ubiquitous Language carries `04_SHARED_LANGUAGE_POLICY` and is named nowhere above the raw research. *Proposed fix:* name it in `04` §1 and link the record. +2. **An unexamined inherited premise.** `D-04` imports attention scarcity from a setting whose goal is the inverse of this protocol's. *Proposed fix:* state the conflict in `03_DECISION_RIGHTS` §8 and say which side wins when they collide. +3. **An over-leveraged evidence family.** The entrainment work is asymmetric to the human–LLM case in a way `research/02` does not qualify. *Proposed fix:* one qualifying sentence; the practical rule it supports is unaffected. +4. **A justified-but-unrecorded omission.** The defective-AI-resistance metric was dropped for a good reason that nobody wrote down. *Proposed fix:* record the reason in `protocol/05` §3. + +Two of these — 2 and 4 — are things no reader could have reconstructed from the pack. That is the argument for the mechanism: the check is cheap, and it found an inherited premise pulling against the protocol's own governing principle. + +## 6. Where this should live + +As `protocol/appendix-analogy-records.md`, referenced from `01_PROTOCOL_V0.2` §11, so that the rule and the project's own compliance with it sit next to each other. A protocol that mandates a check and cannot show its own is not credible on the point. diff --git a/research/hypothesis-register.md b/research/hypothesis-register.md new file mode 100644 index 0000000..ed93270 --- /dev/null +++ b/research/hypothesis-register.md @@ -0,0 +1,56 @@ +# CP-11 — Unified Hypothesis Register + +> **Published record of change proposal CP-11** (see [`docs/review/disposition.md`](../docs/review/disposition.md)). Status unchanged: a proposal; nothing in `protocol/` has been renamed yet. The legacy IDs in the fifth column come from the two unpublished evidence cases and the protocol files; [`docs/provenance.md`](../docs/provenance.md) explains them. `phase0/capabilities/catalogue.md` is now [`capabilities/catalogue.md`](../capabilities/catalogue.md). + +> **Executes:** the second half of change proposal CP-11, addressing the ID sprawl recorded in `09` §2.2: the same claims currently carry different identifiers in five places — `H-01..H-12` (Case 02 §15), `H-SL-01` (Case 02A), `E1..E10` (`protocol/05` §10), `P-01..P-14` / `E-01..E-08` / `T-01..T-13` (`protocol/07`; *added on publication:* `T-11..T-13` were proposed in the adversarial review), and `M-01..M-12` (Case 01). A reader cannot tell whether two IDs are one claim. +> **Status:** proposal. Nothing renamed in `protocol/` yet. + +## Structure + +One ID space, `HYP-nnn`, immutable once allocated. Three kinds, because they were being mixed: + +- **hypothesis** — a testable claim about what improves discovery +- **question** — genuinely open, not yet a claim +- **decided** — adopted as a protocol decision; listed so its evidential basis stays visible + +The `evidence` column is the honest one: it says what the claim actually rests on, not how confident the prose sounds. + +## Register + +| ID | Claim | Kind | Evidence today | Legacy IDs | Capability | Testable now? | +|---|---|---|---|---|---|---| +| HYP-001 | An incomplete, explicitly provisional model elicits more correction than a polished one | hypothesis | both cases, dependent | H-01, E1, M-01, L2/L3 (Case 01) | — | yes, needs `turn.recorded` | +| HYP-002 | Capturing an independent human position before strong AI output preserves information that would otherwise be lost | hypothesis | timing studies, adjacent task | H-02, E2 | CAP-14 | only if the trigger is an enumerated list (CP-03) | +| HYP-003 | Explicit probe labelling plus a debrief beats natural reflection | hypothesis | not tested | H-03, E3, HYP-02/03 (Case 02) | CAP-13 | yes | +| HYP-004 | Research should produce an explicit model delta | **decided** | D-20; Case 01 M-09 kept | H-04, M-09 | CAP-12 | — | +| HYP-005 | Importing a prior framework requires an explicit non-transferability analysis | **decided** | D-19; one observed failure (Turn) | H-05, E8 | CAP-15 | exercised in CP-09 | +| HYP-006 | Initiative is a per-move property, not a role | **decided** | D-06; observed in both cases | H-06 | — | — | +| HYP-007 | A five-factor router allocates initiative well | hypothesis | never exercised | H-07 | none — deliberately | **no** — no event would show it fired | +| HYP-008 | Persistent external state beats strong implicit memory | hypothesis | need observed, effect untested | H-08, E4 | CAP-01 | yes | +| HYP-009 | Compact navigation turns beat repeated long synthesis | hypothesis | one case, contradicted in execution | H-09, E9 | — | **only if** `turn.recorded` is reachable | +| HYP-010 | Handoff with explicit deferrals beats more exhaustive discovery | hypothesis | one case | H-10, E10, M-12 | CAP-05, CAP-09 | yes | +| HYP-011 | A whole-baseline adversarial review before release catches what the pair normalized | **decided** | D-21; its absence was the defect | H-11 | CAP-10 | — | +| HYP-012 | Public context known early changes artifacts for the better | **decided** | D-13, `06` §6; M-11 | H-12, M-11 | — | — | +| HYP-013 | Model-bearing AI terms treated as candidates surface disagreement earlier | hypothesis | mechanism plausible, causality absent | H-SL-01, E5 | CAP-11 | yes | +| HYP-014 | Meaning-first selective terminology work pays for its interaction cost | hypothesis | untested | E6 | CAP-11 | yes | +| HYP-015 | Preserving the human source expression beside the AI reformulation reveals drift | hypothesis | untested | E7 | CAP-11 | yes | +| HYP-016 | The human's counterexamples are worth more than the human's solutions | **decided without test** | Case 01 hypothesis, promoted in v0.2 | M-07 | — | reopen? see note | +| HYP-017 | An explicit decision horizon reduces overdesign | **decided without test** | Case 01 hypothesis, promoted | M-08 | CAP-05 | reopen? see note | +| HYP-018 | Artifact version churn degrades discovery past a point | **decided without test** | Case 01 hypothesis, promoted | M-10 | — | reopen? see note | +| HYP-019 | Adversarial review is most valuable while the model is still cheap to change | hypothesis | **dropped from v0.2** | M-02, G6 | CAP-10 | yes — CP-20 | +| HYP-020 | Broad falsification then narrow verification converges better than one large review | hypothesis | **dropped from v0.2** | M-03, M7, EV-4 | CAP-10 | yes — CP-16 | +| HYP-021 | Real counterexamples produce better models than feature brainstorming | hypothesis | **dropped from v0.2** | M-04 | — | yes — CP-21 | +| HYP-022 | Source / canonical / derived separation is a generalizable AI-discovery pattern | hypothesis | **dropped from v0.2** | M-05, G3, L3, L7 | proposed CAP-16 | yes — CP-18 | +| HYP-023 | A future-readiness check prevents irreversible information loss | hypothesis | **dropped from v0.2** | M-06, AP-05 | proposed CAP-16 | yes — CP-18 | +| HYP-024 | An in-session halt heuristic prevents unproductive continuation | hypothesis | **dropped from v0.2** | SC-3, AP-09 | — | yes — CP-17 | +| HYP-025 | Recorded dissent predicts later reopenings | hypothesis | new, from CP-05 | — | CAP-06 | yes — the one predictive test available at n=1 | + +Open **questions** (`P-01..P-14`, `E-01..E-08`) and deferred **technical decisions** (`T-01..T-13`: `T-01..T-10` in `protocol/07`, `T-11..T-13` proposed in the adversarial review) keep their existing IDs and are not folded in: they are not claims, and merging them would recreate the confusion this register removes. `07` should link here rather than restate. + +## Three findings from building it + +**1. Three claims were promoted from hypothesis to decision without a test.** HYP-016, 017 and 018 were open questions in Case 01's learning backlog and are settled model in v0.2, on the evidence of one dependent case. They are not wrong — they are plausible and probably right — but they are marked `decided` on the same evidential basis that other claims are marked `experimental`, and nothing records why they were treated differently. + +**2. One hypothesis is untestable as specified, and that is worth saying out loud.** HYP-007, the five-factor router, has no event that would show it fired, because it describes a judgement rather than an act. `phase0/capabilities/catalogue.md` deliberately defines no capability for it. Either it becomes an act that leaves a trace, or it should be marked untestable in the protocol rather than sitting in the list looking like a mechanism awaiting evidence. + +**3. Two hypotheses depend on one unresolved technical question.** HYP-001 and HYP-009 both need `turn.recorded`. If Phase 1 finds conversational turns unreachable, both are unmeasurable in this project — and HYP-001 is the criticism-surface claim, which is the single most-cited mechanism in both evidence cases. That dependency deserves to be visible before the spike, not discovered after it. diff --git a/tools/boundary_lint.py b/tools/boundary_lint.py index 1e327b7..6da3e5f 100644 --- a/tools/boundary_lint.py +++ b/tools/boundary_lint.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 -"""Draft boundary lint for the layer cut described in 10_PROJECT_SETUP §2. +"""Draft boundary lint for the layer cut described in docs/overview.md. -Doc 10 argues the protocol/adapter boundary should be enforced by a check over +The project setup (working document 10, see docs/provenance.md) argues the protocol/adapter boundary should be enforced by a check over stable capability IDs rather than by a repository boundary. That argument was made without an implementation, which made it an assertion. This is the implementation, deliberately small and disposable.