Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions PAA.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ where the implementation deliberately stops.
| `AutonomyEvent`, `EventStore` | Autonomy event | Uses the contract's fourteen-field event representation and an implementation protocol with explicit ordering, transaction, uniqueness, and append-only semantics. |
| `SqliteEventStore` | Evidence/event log substrate | Supplies the default append-only SQLite store, including storage-level update/delete rejection. |
| `store_evidence`, `verify_evidence` | Evidence record binding | Content-addresses exact evidence bytes with SHA-256 and fails closed on missing or changed bytes. |
| `OperatingRecord`, `SqliteOperatingRecordStore` | Operating accounting | Stores usage, prices and provenance separately from evidence files and autonomy events, retrievable by subject; no transition rule reads it. |
| `import_events` | Archive replay | Imports already validated contract-shaped events without regenerating identifiers or timestamps. The legacy conformance capture proves field and projection continuity across extraction. |
| `paa-contracts` conformance suite | Published contract | Checks schema vocabulary, declarations, event histories, evidence addressing, invalid semantic cases, and the pre-cutover capture against the same packaged corpus. |

Expand Down Expand Up @@ -80,9 +81,26 @@ implementation.
### Worker identity

Events record an actor selected explicitly, from a configured environment
variable, or from the OS login. The current contract has no durable worker
identity or attestation field, so an evidence window cannot prove which worker
produced every verdict. That requires a future contract revision.
variable, or from the OS login. This actor is distinct from the optional
evidence-record `worker {id, version, configuration_ref}` and the evaluator-side
`producer`. Worker attribution is consumer-asserted identity, not attestation.
The exact evidence bytes, including worker attribution, are retained. Current
transition logic does not inspect worker identity or operating cost.

### Future declaration rules — draft only, not implemented

A later contract revision may declare a single-configuration evidence window:
every qualifying verdict would need explicit matching worker identity and a
configuration reference. Missing attribution would not establish compliance;
evidence from another configuration would not silently count toward the window.

A separate configuration-change rule may require restoring the declared safer
position before operating a changed configuration. It would need to define
which identity changes trigger it, the fallback when already at the safest
position, window reset behavior, and atomicity with the consumer's configuration
switch. Such a rule must be separately versioned and approved. This revision
adds no declaration fields or automatic demotion, and does not claim to enforce
"no inherited autonomy." Cost would remain outside transition eligibility.

### Governed-effect atomicity with the default store

Expand Down
65 changes: 61 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,14 @@ PAA is implementation-neutral by construction — it describes *what* a governed
- **Position resolution** — current autonomy position is never stored. It is folded fresh from the declaration's `initial_position` plus the latest exact-scope `position_changed` event.
- **Evidence binding** — every motion binds to the exact bytes of its evidence artifact by SHA-256, re-verified at approval. Tamper or loss is a fail-closed error.
- **An append-only event store** — append-only enforced by the storage layer, not by convention.
- **Operating-record storage** — a separate optional append-only store preserves subject-linked usage, prices, worker configurations, and source attribution. It has no role in resolving authority.

## What it does not do

- **Produce evaluator verdicts.** The runtime governs; consumers evaluate. Which evaluators exist and what code produces each verdict is consumer domain data, supplied as a registry.
- **Evaluate promotion rules.** Thresholds and windows are *declared*, not machine-evaluated. Approval is an operator judgment.
- **Carry worker identity.** The contract has no worker-identity field yet, so evidence windows cannot prove which worker produced them. Tracked for a later contract cycle.
- **Enforce worker-configuration policies.** Evidence records can carry optional worker attribution; evidence storage preserves those bytes. Single-configuration windows and configuration-change demotion remain future declaration rules, not implemented behavior.
- **Calculate or govern by cost.** Consumers produce prices, retain constituent usage, distinguish actuals from estimates, assess coverage, and reconcile overlapping summaries. Neither operating records nor worker attribution changes transition eligibility in this revision.

## Install

Expand Down Expand Up @@ -122,7 +124,7 @@ The conformance suite runs against the published contract artifacts rather
than fixtures of its own, so that "passes the published conformance suite" is
a claim about the contract and not about this repo's idea of it.

Those artifacts come from `paa-contracts` — the four normative schemas, the
Those artifacts come from `paa-contracts` — the five normative schemas, the
positive fixture corpus, and the invalid-case tables. It is the other package
this repo publishes, and a workspace member here, so the extra is all it takes:

Expand Down Expand Up @@ -186,6 +188,61 @@ release would carry the corpus.

## Status

`0.3.0`, tracking the `paa-task/0.2.1-draft` and `paa-autonomy-event/0.1.0-draft` schema families. Package and spec versions drift independently — the schema families a release targets are stated here and asserted by the conformance suite, not inferred from the package version.
`0.4.0` adds optional operating-record storage against `paa-operating-record/0.1.0-draft` and preserves optional worker attribution in `paa-evidence-record/0.2.0-draft`. Existing `paa-evidence-record/0.1.0-draft` records without worker attribution remain valid. No evidence files or event histories require migration. Task and event families remain `paa-task/0.2.1-draft` and `paa-autonomy-event/0.1.0-draft`. Package and schema-family versions are independent; this PR prepares release versions but does not publish packages.

`0.3.0` is a breaking change to the declaration access layer, and it is the change that made the sentence above true. `0.2.0` claimed the `paa-task/0.2.1-draft` family while implementing an older evaluator identity — a single `oracle` field where the contract has `evaluation_basis` and `epistemic_status` — and a `position_policy` requiring all four positions at fixed modes, where the contract admits any non-empty subset with per-evaluator placement overrides. It could not load a single published declaration. Building the conformance suite is what surfaced that; `PaaEvaluator`, `ProducerRegistration`, and `PaaPositionPolicy` changed shape to fix it. Nothing was published at `0.2.0`, so no consumer is stranded.
`0.3.0` introduced a breaking change to the declaration access layer to align it with the published task family. `0.2.0` claimed the `paa-task/0.2.1-draft` family while implementing an older evaluator identity — a single `oracle` field where the contract has `evaluation_basis` and `epistemic_status` — and a `position_policy` requiring all four positions at fixed modes, where the contract admits any non-empty subset with per-evaluator placement overrides. It could not load a single published declaration. Building the conformance suite is what surfaced that; `PaaEvaluator`, `ProducerRegistration`, and `PaaPositionPolicy` changed shape to fix it. Nothing was published at `0.2.0`, so no consumer is stranded.

## Optional operating records

Evidence remains content-addressed files. Autonomy events remain in `EventStore`.
`SqliteOperatingRecordStore` adds an independent `operating_records` table and
connection; it may use the same database path or a separate database. It does
not extend the `EventStore` protocol or participate in motion transactions.

```python
from pathlib import Path
from paa_runtime import SqliteOperatingRecordStore, decode_operating_record

record = decode_operating_record(Path("operating-record.json").read_bytes())
accounting = SqliteOperatingRecordStore(Path("paa_runtime.db"))
try:
accounting.append(record)
records = accounting.get_by_subject(record["subject"])
finally:
accounting.close()
```

An append validates structure and commits one record atomically. Reusing a
record ID is an error, including an identical retry of an already committed
write; read by subject to reconcile an uncertain write before retrying. Reads
return detached records in insertion order. Subject kind and ID both match
exactly; each result retains task, declaration version, scope, and worker
configuration. Multiple tasks, attempts, and summaries may share a subject.
No update/delete API exists; SQLite triggers reject updates, deletes, and
replacement inserts. As with the event store, this is not protection against
an administrator dropping triggers or rewriting the database.

`usage` keys are open, nonnegative quantities; recommended names include
`input_tokens`, `output_tokens`, `cached_tokens`, and `llm_calls`. Null usage,
individual quantities, or price explicitly means unavailable. Zero is a real
measurement. A price requires `currency`, `amount`, and opaque `basis` together.
Optional components carry additional costs **not already included** in the
base price. Omitted components make no completeness claim. Component kinds and
currencies are consumer-defined; the runtime neither sums nor converts them.

Source references must identify the attributed work and attempts, including
failures and superseded retries, and preserve constituent quantities,
model/rate identities, pricing bases, and measured/estimated coverage. Pipeline
summaries are allowed; a catalog reference and aggregate tokens alone cannot
reprice mixed-model work. Do not sum pipeline totals and their constituent
task records as independent charges. Multiple verdicts about one output do
not create more charges. The runtime validates references structurally, not
their external contents or accounting accuracy.

Readers compute effective cost for one configuration, population, and window:
attributed costs of **all** attempts divided by distinct accepted outcomes in
that same population. Acceptance rules belong to the task/consumer. Zero
accepted outcomes means undefined effective cost, not zero; missing prices
cannot support an unqualified total-cost claim. Effective cost is not a stored
field, and operating records cannot grant authority or offset a behavioral
failure.
1 change: 1 addition & 0 deletions conformance/_corpus.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@

#: Which schema governs each case table's documents.
SCHEMA_FOR_KIND: dict[str, str] = {
"operating": "paa-operating-record",
"task": "paa-task",
"evidence": "paa-evidence-record",
"decision": "paa-decision-artifact",
Expand Down
11 changes: 6 additions & 5 deletions conformance/test_corpus_integrity.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
]

PUBLISHED_FIXTURES = [
*(("operating", p) for p in contracts.operating_record_paths()),
*(("task", p) for p in contracts.task_declaration_paths()),
*(("event", p) for p in contracts.autonomy_event_paths()),
*(("evidence", p) for p in contracts.evidence_record_paths()),
Expand All @@ -70,17 +71,17 @@ class TestTheCorpusIsWhatItClaims:
guard against that.
"""

def test_the_case_tables_carry_ninety_five_cases(self) -> None:
assert len(ALL_CASES) == 95
def test_the_case_table_count_is_pinned(self) -> None:
assert len(ALL_CASES) == 147

def test_fifty_of_them_are_structural(self) -> None:
assert len(STRUCTURAL_CASES) == 50
def test_the_structural_case_count_is_pinned(self) -> None:
assert len(STRUCTURAL_CASES) == 102

def test_fifteen_of_them_are_pinned(self) -> None:
assert len(PINNED_CASES) == 15

def test_every_published_fixture_is_discoverable(self) -> None:
assert len(PUBLISHED_FIXTURES) == 17
assert len(PUBLISHED_FIXTURES) == 22


class TestFormatAssertionIsLive:
Expand Down
2 changes: 1 addition & 1 deletion conformance/test_evidence_integrity.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def _sha_from_path(path: Path) -> str:

class TestCorpusIsPresent:
def test_evidence_records_are_discoverable(self) -> None:
assert len(contracts.evidence_record_paths()) == 3
assert len(contracts.evidence_record_paths()) == 4

def test_decision_artifacts_are_discoverable(self) -> None:
assert len(contracts.decision_artifact_paths()) == 5
Expand Down
94 changes: 94 additions & 0 deletions conformance/test_operating_records.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
"""Published operating records round-trip without becoming policy inputs."""

from __future__ import annotations

import json
from pathlib import Path

import paa_contracts as contracts
import pytest
from jsonschema import Draft7Validator

from conformance._corpus import case_documents, violations
from paa_runtime import RuntimeConfig, SqliteEventStore, approve, propose, show
from paa_runtime.operating import (
CURRENT_OPERATING_SCHEMA,
OperatingRecordError,
decode_operating_record,
)
from paa_runtime.operating_store import SqliteOperatingRecordStore


def test_operating_schema_stamp_matches_contract() -> None:
assert contracts.schema_version("paa-operating-record") == CURRENT_OPERATING_SCHEMA


@pytest.mark.parametrize("schema_id", ["paa-operating-record", "paa-evidence-record"])
def test_revised_contract_is_a_valid_draft7_schema(schema_id: contracts.SchemaId) -> None:
Draft7Validator.check_schema(contracts.load_schema(schema_id))


def test_worker_definitions_agree() -> None:
operating = contracts.load_schema("paa-operating-record")["definitions"]["worker"]
evidence = dict(contracts.load_schema("paa-evidence-record")["definitions"]["worker"])
evidence.pop("description")
assert operating == evidence


@pytest.mark.parametrize("path", contracts.operating_record_paths(), ids=lambda path: path.name)
def test_operating_fixture_round_trips(path: Path, tmp_path: Path) -> None:
record = decode_operating_record(path.read_bytes())
store = SqliteOperatingRecordStore(tmp_path / "records.db")
try:
store.append(record)
assert store.get_by_subject(record["subject"]) == (json.loads(path.read_bytes()),)
finally:
store.close()


@pytest.mark.parametrize("case", contracts.invalid_cases("operating"), ids=lambda case: case["id"])
def test_decoder_rejects_published_invalid_case(case: contracts.InvalidCase) -> None:
_, mutated = case_documents("operating", case)
with pytest.raises(OperatingRecordError):
decode_operating_record(json.dumps(mutated).encode())


def test_current_evidence_without_worker_remains_valid() -> None:
record = json.loads(contracts.evidence_record_paths()[0].read_bytes())
record["record_schema"] = "paa-evidence-record/0.2.0-draft"
record.pop("worker", None)
assert violations("evidence", record) == ()


def test_operating_records_do_not_change_motion_outcomes(
runtime_config: RuntimeConfig, tmp_path: Path,
) -> None:
"""Same database, deliberately changed cost/worker; approval still independent."""
events = SqliteEventStore(runtime_config.db_path)
operating = SqliteOperatingRecordStore(runtime_config.db_path)
evidence = tmp_path / "report.json"
evidence.write_bytes(b'{"operator_report": "illustrative"}')
try:
motion = propose(
events, runtime_config, task="outbound_content_publish", scope="publish:farcaster",
to_position="hotl", evidence_path=evidence, actor="test:operator",
)
before = show(
events, runtime_config, task="outbound_content_publish", scope="publish:farcaster",
)
for index, path in enumerate(contracts.operating_record_paths()):
record = decode_operating_record(path.read_bytes())
record.update(task="outbound_content_publish", scope="publish:farcaster")
record["worker"]["configuration_ref"] = f"cfg:changed-{index}"
operating.append(record)
assert show(
events, runtime_config, task="outbound_content_publish", scope="publish:farcaster",
) == before
approve(
events, runtime_config, motion_id=motion.motion_id,
actor="test:operator", reason="independent operator approval",
)
assert events.get_autonomy_events()[-1].to_position == "hotl"
finally:
operating.close()
events.close()
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"boundary":{"input_ref":"app://inbound_candidate/1001","output_ref":"app://surfaced_event/1001"},"declaration_version":1,"evaluator":{"authority":"advisory","epistemic_status":"proxy","evaluation_basis":{"kind":"rubric","ref":"response_quality_rubric"},"property":"response_quality","target":"output","technique":"llm_judge","version":"1"},"payload":{"model":"critic-v1","rubric_id":"response_quality_rubric_v1","score":0.86},"payload_schema":"https://paa.dev/payload-schemas/response-quality-llm.schema.json","producer":{"id":"llm-critic","version":"1"},"record_id":"evrec-inbound-case-1001-worker","record_schema":"paa-evidence-record/0.2.0-draft","scope":null,"source_references":["critiques:5501"],"subject":{"id":"case-1001","kind":"case"},"task":"inbound_reply_surfacing","timestamps":{"completed_at":"2026-01-05T12:00:02.000Z","recorded_at":"2026-01-05T12:00:02.500Z","started_at":"2026-01-05T12:00:00.000Z"},"verdict":{"reason_codes":["tone_within_bounds","no_hallucinated_claim"],"value":"approve"},"worker":{"id":"reply-drafter","version":"response-v7","configuration_ref":"cfg:reply-drafter-v7"}}
Loading
Loading