Research: bounded AgentOps approval workflow and synthetic control evaluation

research/iaai27_agentops/CLAIMS.md

@@ -0,0 +1,22 @@
+# Evidence-bounded claims
+
+| Claim | Evidence now | Permissible scope |
+|---|---|---|
+| Approval can resume in the reference app | Persistent approve/execute test, CLI smoke transcript | Local trusted-operator workflow; not main Mizan chat/API |
+| Single finalized request does not execute twice | Replay and two-worker race tests | SQLite synthetic action transaction only |
+| Policy blocks tested disallowed proposals | 0 executions in 15 constructed cases under policy/full configurations | Tested cases; not a universal safety theorem or field rate |
+| Outcome observation detects silent failure | Full detects 3/3 injected failures vs. 2/3 without separate observation | Synthetic state and fault flags; not real provider timeouts |
+| Normal tasks remain resolvable | 4/4 deliberately simple clean cases across configurations | Three-bit planner domain; not broad incident-resolution quality |
+| Semantic results repeat | 96/96 matching outcomes across two runs | Identical deterministic fixtures; no stochastic-model reproducibility claim |
+| Proposed deployment path exists | Owner selected Hostlelo isolated staging; deployment plan names gates/resources | Proposed; actual host/access/pilot results unverified |
+| Arabic/English safety parity | None | Do not claim |
+| Live LLM behavior or prompt-injection robustness | None | Do not claim |
+| Production utility, uptime or commercial return | None | Do not claim |
+| Forgekit/Hikmah runtime integration | None in this application | Design lineage only; Mizan policy classifier is actual code reuse |
+| Human-written manuscript | Recovered writing offers style/direction but insufficient verified technical prose | Generated draft must remain labeled; author-written substance still required |
+
+The small acceptance suite intentionally includes faults matched to the controls.
+It is useful engineering evidence and weak evidence of generalization. Reviewer
+objections about realism, novelty and deployment distance remain valid. The paper
+must discuss those limitations rather than imply that synthetic acceptability
+guarantees acceptance.

research/iaai27_agentops/DEPLOYMENT.md

@@ -0,0 +1,71 @@
+# Proposed Hostlelo pilot
+
+Status: **proposed**, confirmed as the intended deployment path by Juber Shaikh
+in this conversation. No Hostlelo server was contacted or changed in this task.
+No pilot has started, no customer/user outcome was measured, and no production
+environment access is implied.
+
+## Scope and owner
+
+- Proposed owner/operator: Juber Shaikh.
+- Intended setting: isolated Hostlelo staging environment with generated data.
+- First users: the owner and subsequently an explicitly authorized staging
+  operator. No claim that additional operators are recruited or committed.
+- Initial artifact: this repository's local symbolic controller and SQLite
+  simulator. Real infrastructure actions and model proposals require new adapters.
+- Data: generated service states, diagnostic observations and audit events.
+  Exclude customer, employer, airline, ticket, billing and production data.
+
+## Entry requirements
+
+Before calling the pilot “installed,” record the staging host's ownership and
+authorization, capacity, OS/runtime versions, isolation controls, artifact commit,
+operator access, and recovery procedure. A specific staging host and its resources
+were not verified in this task. The initial installation needs Python 3.11+, the
+Mizan dependencies and a writable private data directory; no GPU/model key is
+required for the symbolic reference application.
+
+Use a dedicated unprivileged OS account and private directory. Do not share its
+database credentials or local account with an untrusted proposal engine. The
+reference application has no remote identity boundary. Do not expose the CLI
+through an unauthenticated web terminal or model tool.
+
+## Proposed milestones (relative to confirmed staging access)
+
+| Milestone | Target window | Deliverable | Acceptance gate |
+|---|---|---|---|
+| Reproduce local artifact | Day 0-1 | Exact commit, manifest, tests and raw results | Application tests pass; finite-suite results reproduced; known repository gate failure recorded |
+| Install isolated staging simulator | Day 1-2 | Installation log and permissions evidence | No production credentials/data; only synthetic targets reachable; restart preserves pending/approved requests |
+| Operator walkthrough | Day 2-4 | Recorded propose/reject/approve/replay/expiry cases | Operator can inspect exact scope, reject and observe outcome; approval UI/CLI misunderstandings logged |
+| Read-only real-service adapter in staging | Day 4-7 | Separate probe adapter and seeded faults on disposable test services | Independent probe agreement, fault restoration and explicit environment scoping; no state-changing tools |
+| Bounded staging action adapter | After prior gate | Allowlisted action + idempotency + recovery handling | Fresh authorization, denial/replay/concurrency/crash tests; known-outcome verification and stop mechanism |
+| Live model comparison | After adapter gate and approved credentials | Versioned prompts, model outputs and held-out evaluation | Model never receives operator credentials/ground truth; control and task-quality metrics reported separately |
+
+These are proposed planning targets, not promised calendar commitments or
+completed activities. Failures move the schedule; they are not waived to fit it.
+
+## Stop and recovery rules
+
+Stop on any action outside the target allowlist, unauthorized execution, approval
+parameter mismatch, uncertain side effect, corrupted audit replay, or accidental
+production/customer-data access. Disable the executor before diagnosis. Preserve
+logs. Reconcile real external state before retrying an uncertain action. The
+simulator's SQLite transaction does not solve distributed action atomicity.
+
+For the current simulator, archive the private database and initialize a new
+file for a new trial. Never overwrite history while claiming uninterrupted audit
+integrity. For later real-service trials, snapshots and compensating actions must
+be specified and authorized before inducing faults.
+
+## Pilot evaluation to collect later
+
+Record operator task success, approval comprehension, veto effectiveness, time to
+verified recovery, unnecessary interventions, escalation workload, every failure,
+and system overhead. Compare against an operator-run documented runbook and an
+approval-only condition. Define representative scenarios and independent labels
+before collection. Do not reuse the deliberately adversarial control-suite mix
+as an estimate of actual incident prevalence.
+
+Juber intends to present at IAAI-27 in Montréal if accepted, subject to visa
+arrangements. Registration, visa issuance, travel booking and funding have not
+been completed or guaranteed by this task.

research/iaai27_agentops/README.md

@@ -0,0 +1,125 @@
+# Mizan-AgentOps: synthetic assurance reference application
+
+This research addition implements a persistent local incident-control workflow:
+observe, plan, inspect, approve/reject, execute, verify, and preserve an audit trail.
+It supports an IAAI-27 Emerging Applications investigation. **It is not a submitted
+paper, production deployment, or live LLM benchmark.**
+
+The planner performs uniform-cost symbolic search over three Boolean observations
+(worker, route, cache). Remediation operates on a local SQLite synthetic service.
+The verifier separately reads realized target state rather than trusting a tool's
+success response. This is an intentionally small, completely inspectable model.
+
+## Existing research reused
+
+- **Mizan**: imports `IznPermission` and `PermissionLevel` from the existing
+  permission module for tool policy classification. Its old `approve_pending`
+  method deletes a request without creating an executable approval grant. The
+  new application replaces that lifecycle with durable, plan-bound requests;
+  it does **not** claim that the main Mizan chat/API approval flow is repaired.
+- **Hikmah Stack**: informs separation of proposals from trusted controls, explicit
+  unknown outcomes, and hash-linked history. Its Rust runtime is not integrated
+  or benchmarked in this artifact.
+- **Forgekit**: informs evidence-producing verification, reproducible evaluation,
+  and conservative claims. Its MCP and model runtimes are not integrated or
+  benchmarked here.
+
+No unrelated mathematical theorem, new cognitive capability, novel planning
+algorithm, human oversight benefit, or general prompt-injection defense is claimed.
+
+## Run the local application
+
+From the Mizan repository root, with Python 3.11 or newer:
+
+```bash
+python3 -m pip install -e '.[dev]'
+python3 -m research.iaai27_agentops --db /tmp/agentops-demo.sqlite init
+python3 -m research.iaai27_agentops --db /tmp/agentops-demo.sqlite propose
+```
+
+Inspect the returned plan, evidence, environment, and policy binding. Copy its
+request ID into these commands. Each command is a separate process; approval
+survives process restart.
+
+```bash
+python3 -m research.iaai27_agentops --db /tmp/agentops-demo.sqlite request REQUEST_ID
+python3 -m research.iaai27_agentops --db /tmp/agentops-demo.sqlite approve REQUEST_ID
+python3 -m research.iaai27_agentops --db /tmp/agentops-demo.sqlite execute REQUEST_ID
+python3 -m research.iaai27_agentops --db /tmp/agentops-demo.sqlite inspect
+```
+
+Use `reject REQUEST_ID` instead of `approve` to veto. Evidence expires after 60
+seconds; a delayed execution needs a fresh proposal and approval. Reusing the
+same finalized request does not execute it twice. `init` requires a fresh database;
+it deliberately does not reset existing history.
+
+## Trust and isolation boundaries
+
+- CLI-only, trusted local operator; no web interface or network listener.
+- Local file access is the authority boundary. The operator label is descriptive,
+  not a remote identity assertion. Do not expose these calls to an untrusted LLM,
+  user, or network without an authenticated separation of operator and executor.
+- All action variants, including ungated research baselines, mutate only the
+  synthetic `targets` table. No arbitrary shell command or cloud API exists.
+- Approval is bound to stored action/evidence/environment/policy payloads, expires,
+  and is finalized transactionally. Current policy and target state are rechecked.
+- An entire synthetic execution and its audit update use one SQLite transaction.
+  This atomicity does not transfer to external cloud actions. A real adapter
+  requires durable outbox/idempotency, uncertain-outcome reconciliation, and
+  separately authorized compensation.
+- Audit chaining detects some changes under a trusted database boundary. It does
+  not prevent an administrator from replacing or recomputing the whole history,
+  detect all truncation without an external anchor, encrypt records, or implement
+  regulatory immutability.
+- Rollback restores simulated state and may restore the original fault. It does
+  not necessarily restore service availability.
+
+## Verification and reproduction
+
+```bash
+python3 -m pytest research/iaai27_agentops/test_agentops.py -q
+python3 -m ruff check research/iaai27_agentops
+python3 -m research.iaai27_agentops.evaluate --out /tmp/agentops-evaluation
+```
+
+The recorded run has 24 designed scenarios x 4 configurations = 96 episodes.
+Results, complete per-episode audit traces, fixture definitions, source hashes,
+dependency versions and repeatability findings are in `results/`.
+
+| Configuration | Prohibited executions / 15 cases | False completions / 24 episodes | Clean resolutions / 4 | Failed remediations detected / 3 |
+|---|---:|---:|---:|---:|
+| Ungated sandbox baseline | 14 | 3 | 4 | 2 |
+| Approval only | 11 | 3 | 4 | 2 |
+| Approval + policy | 0 | 1 | 4 | 2 |
+| Full + independent outcome observation | 0 | 0 | 4 | 3 |
+
+All variants share a restricted synthetic executor, request-payload integrity and
+finalization behavior. Ablations isolate approval, policy and outcome observation;
+“ungated” does not mean unconstrained operating-system access. Approval decisions
+are simulated. A second run matched 96/96 semantic outcomes, excluding timings
+and audit UUID/hash differences. This is deterministic replay, not 96 independent
+statistical samples. These are finite-suite counts with no population confidence
+interval, significance test, or production failure-rate interpretation.
+
+Some scenarios force malformed/prohibited proposals directly into the control
+boundary. They measure downstream enforcement, not a model's tendency to produce
+those proposals. The planner does not ingest natural-language logs. Consequently
+there is no measured natural-language prompt-injection resistance, no Arabic vs.
+English comparison, no live LLM call, and no general incident-resolution result.
+
+The initial test run passed 23 application tests. The existing repository suite
+passed 557 tests with 8 skips. `make check` passes lint but stops on the existing
+mypy duplicate-module error for `_version` / `backend._version`; this research-only
+addition does not modify the existing backend. See `results/validation.md`.
+
+## Research limitations and deployment
+
+See [the proposed pilot](DEPLOYMENT.md) and [the claim ledger](CLAIMS.md). Future
+external adapters, live models, operator trials and held-out incident evaluation
+remain separate work. The current simulator is not a miniature production cloud
+and must not be advertised as one.
+
+New reference code and evaluation assembly were generated with AI assistance in
+this task. Existing repository authorship and licenses remain intact. Manuscript
+authorship must be disclosed independently; a style-matched draft does not become
+human-written text merely because it is reviewed or approved.

research/iaai27_agentops/__init__.py

@@ -0,0 +1 @@
+"""Bounded synthetic incident application; no production infrastructure adapter."""

research/iaai27_agentops/__main__.py

@@ -0,0 +1,53 @@
+"""Local, persistent propose / approve / reject / execute workflow."""
+
+import argparse
+import getpass
+import json
+import time
+
+from .core import Policy, Store, execute, symbolic_plan
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--db", default="agentops-demo.sqlite")
+    sub = parser.add_subparsers(dest="command", required=True)
+    sub.add_parser("init")
+    sub.add_parser("inspect")
+    sub.add_parser("propose")
+    for command in ("approve", "reject", "execute", "request"):
+        sub.add_parser(command).add_argument("request")
+    args = parser.parse_args()
+    store, policy = Store(args.db), Policy()
+    now = time.time()
+    try:
+        if args.command == "init":
+            store.initialize((False, True, True))
+            value = {"environment": "synthetic-local", "state": store.state()}
+        elif args.command == "inspect":
+            value = {"state": store.state(), "audit_valid": store.audit_valid()}
+        elif args.command == "propose":
+            evidence = store.observe(now)
+            actions = symbolic_plan(evidence, policy.target)
+            if not actions:
+                value = {"status": "no_plan", "reason": "healthy_or_insufficient_evidence"}
+            else:
+                identifier = store.propose(actions, evidence, policy, now)
+                value = store.request(identifier)
+        elif args.command == "request":
+            value = store.request(args.request)
+        elif args.command in ("approve", "reject"):
+            value = {
+                "accepted": store.decide(
+                    args.request, args.command == "approve", getpass.getuser(), now
+                )
+            }
+        else:
+            value = execute(store, args.request, policy, now)
+        print(json.dumps(value, ensure_ascii=False, indent=2))
+    finally:
+        store.close()
+
+
+if __name__ == "__main__":
+    main()