Add capability reliability ledger (#186)

.github/workflows/web-ci.yml

@@ -269,10 +269,13 @@ jobs:
               EXECUTE format('GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE public.%I TO forge_app_test', table_name);
             END LOOP;
             REVOKE ALL ON TABLE public.execution_outcomes, public.operation_runs,
-              public.operation_run_events FROM forge_app_test;
+              public.operation_run_events, public.capability_attempts,
+              public.capability_attempt_adjudications FROM forge_app_test;
             GRANT SELECT, INSERT, UPDATE ON TABLE public.execution_outcomes,
               public.operation_runs TO forge_app_test;
             GRANT SELECT, INSERT ON TABLE public.operation_run_events TO forge_app_test;
+            GRANT SELECT, INSERT ON TABLE public.capability_attempts,
+              public.capability_attempt_adjudications TO forge_app_test;
             GRANT USAGE, SELECT ON SEQUENCE public.task_logs_sequence_seq TO forge_app_test;
           END;
           $grant_s4_application_acl$;
@@ -512,7 +515,8 @@ jobs:
               'app_settings', 'task_questions'
             ];
             operation_ledger_tables constant text[] := ARRAY[
-              'execution_outcomes', 'operation_runs', 'operation_run_events'
+              'execution_outcomes', 'operation_runs', 'operation_run_events',
+              'capability_attempts', 'capability_attempt_adjudications'
             ];
             protected_tables constant text[] := ARRAY[
               'forge_release_signer_keys', 'forge_release_signer_key_lifecycle_audits',
@@ -559,6 +563,8 @@ jobs:
             GRANT SELECT, INSERT, UPDATE ON TABLE public.execution_outcomes,
               public.operation_runs TO forge_app_test;
             GRANT SELECT, INSERT ON TABLE public.operation_run_events TO forge_app_test;
+            GRANT SELECT, INSERT ON TABLE public.capability_attempts,
+              public.capability_attempt_adjudications TO forge_app_test;
             FOREACH table_name IN ARRAY operation_ledger_tables LOOP
               FOREACH table_privilege IN ARRAY ARRAY[
                 'SELECT', 'INSERT', 'UPDATE', 'DELETE', 'TRUNCATE', 'REFERENCES', 'TRIGGER'
@@ -568,7 +574,7 @@ jobs:
                 ) IS DISTINCT FROM (
                   (table_name IN ('execution_outcomes', 'operation_runs')
                     AND table_privilege IN ('SELECT', 'INSERT', 'UPDATE'))
-                  OR (table_name = 'operation_run_events'
+                  OR (table_name IN ('operation_run_events', 'capability_attempts', 'capability_attempt_adjudications')
                     AND table_privilege IN ('SELECT', 'INSERT'))
                 ) THEN
                   RAISE EXCEPTION 'ordinary app has unexpected % on operation ledger table public.%',

docs/adr/0012-capability-reliability-ledger.md

@@ -0,0 +1,98 @@
+# ADR 0012: Capability reliability ledger
+
+## Status
+
+Accepted. Implemented in migration `0031_capability_reliability_ledger.sql`.
+
+Primary design document:
+`docs/architecture/issue-186-capability-reliability-ledger.md`.
+
+## Context
+
+ADR 0010 gave Forge a canonical record of *what happened* on one attempt. ADR
+0011 gave it deterministic typed operations. Neither answers whether a specific
+kind of work has repeatedly succeeded under comparable conditions, which is the
+evidence issue #189 will need before it may widen any permission.
+
+A single reliability score per agent or model would hide the differences that
+matter — scope, project, model, harness, and policy — and would encourage exactly
+the unsafe promotion this Epic exists to prevent.
+
+## Decision
+
+Forge records an append-only ledger of individual **capability attempts** and
+computes reliability metrics per **cohort**, on demand, from those attempts.
+
+A cohort is the domain-separated SHA-256 fingerprint of project, capability key,
+scope, runtime/model, and policy. Because a material change to any of those
+produces a different cohort, requalification is automatic: new conditions start a
+new sample count, and prior evidence is retained but no longer counted. The four
+component fingerprints are stored beside the cohort fingerprint so drift is
+attributable to a specific input rather than only detectable.
+
+Capability keys are namespaced — `workpackage:<role>/<capability>` from the
+existing `CAPABILITY_TAXONOMY`, or `operation:<id>@<version>` from the ADR 0011
+catalog — so model-executed work and deterministic operations can never share a
+cohort. One work package writes one attempt row per capability it exercised, all
+sharing an `attempt_group_id` and a multiplicity count, so per-capability and
+per-attempt views are both available without double-counting. When the Architect
+classification is missing or exceeds the fan-out bound, a single reserved
+`unclassified` row records the gap instead of guessing a capability.
+
+`capability_attempts` is immutable: identity and the ingest-time outcome
+snapshot cannot be updated or deleted, enforced by a database trigger. Evidence
+that arrives later — verification results, human decisions, rollbacks, overrides,
+and detected drift — is appended to `capability_attempt_adjudications` in gapless
+sequence order, never written back into the attempt.
+
+Whether an attempt counts as verified is decided by a closed `verification_mode`.
+`self_reported` and `human_review` never contribute to the independently verified
+pass rate; only `deterministic_adapter` (ADR 0011) and `independent_agent` do.
+`independent_agent` has no producer until issue #188, so v1 rejects it at ingest
+rather than allowing an unbacked value to be stored. Forge's current honest
+answer for most cohorts is reported explicitly as an unverified-completion rate
+instead of being folded into a pass rate.
+
+The ledger has no free-text column. Every `text` column is a closed enum, a
+64-hex fingerprint, or the bounded capability-key grammar, each enforced by a
+`CHECK` constraint. Model prose, file paths, repository-relative names, and
+credentials therefore cannot enter the ledger even by mistake, and no redaction
+helper is needed on this path. Scope is fingerprinted from the project's opaque
+`root_ref` and revisions, never from `local_path`.
+
+Metrics are a pure function of stored attempts, adjudications, a window, and an
+explicit `now`. No materialized summary is stored in v1: a cache that can
+disagree with its evidence is a class of bug this ledger exists to avoid, and the
+cohort index makes on-demand computation a bounded scan. Below a minimum sample
+size a cohort reports `insufficient_evidence` with null rates; if any in-window
+attempt's linked outcome has changed since ingest, the cohort reports
+`evidence_drift` and suppresses all rates. Critical failures are reported
+unconditionally in every state, so no aggregate can conceal one.
+
+Ingest hangs off the existing canonical-outcome boundaries — the three work
+package handoff sites and, after its transaction commits, the ADR 0011 operation
+finalize path. Writes are best-effort and idempotent on
+`(execution_outcome_id, capability_key)`: a ledger failure never fails a task,
+package, run, or operation, and a recovered worker re-running a boundary writes
+nothing new. Historical attempts are not backfilled; a missing attempt is
+unavailable evidence, never success. The ordinary application role receives
+`SELECT` and `INSERT` on the ledger tables and nothing else.
+
+## Consequences
+
+Issues #188, #189, #190, and #191 read this contract instead of deriving trust
+from statuses, free-text errors, or a worker's own account of its performance.
+Autonomy decisions in #189 can cite a cohort, a sample size, a verification mode,
+and the evidence rows behind each number.
+
+This ADR grants no autonomy and changes no permission. It adds no dashboard, HTTP
+route, scheduled job, or background recomputation. It does not produce
+independent verification, and it does not replace the task, work package, agent
+run, artifact, execution outcome, operation run, or approval gate records that
+remain authoritative for their own state.
+
+Two capabilities are defined but not yet producible: rollback and override
+adjudications have storage contracts and metrics but no writer until #189/#190,
+and independent-agent verification is refused until #188. Both are deliberate —
+the storage shape is stable, and the gaps are visible rather than filled with
+optimistic defaults.