Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/memory/architecture/api-endpoints.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,7 @@ Bulk ops scope writes to the caller's accessible agents (tri-state: admin = no f

WebSocket events: `operator_queue_new`, `operator_queue_responded` / `operator_queue_cancelled` (thin `{id, agent_name}` since trinity-enterprise#611 — listeners refetch), `operator_queue_acknowledged`, `operator_queue_cleared` (one per bulk op, #1017; count only — `notifications_cleared` for the notifications variant), `operator_queue_sync` (#2915, one per poll cycle; announces expiry too). Backed by the 5s Operator Queue Sync background service. Every ending — answer, cancel, bulk cancel, expiry — goes through one sink, `services/ask_service.py` (trinity-enterprise#611: CAS + endings ledger → one audit row per transition → one thin trigger → the registered ending observers); the machine keys never see the new person fields (`disposed_by_email`, `resolved_to`) on get/list. See [operating-room.md → Endings](../feature-flows/operating-room.md#endings-trinity-enterprise611).

**Addressed asks (ent#364).** `operator_queue.addressed_to_email` names the HUMAN an item is for; `agent_name` only ever said which agent. NULL — every pre-ent#364 row — means "the operator answers", so the column is additive by construction. It is a **column and not a key in `context`** because it is an authorization decision (who may answer, whose Workspace sidebar it appears in) and `context` is agent-authored JSON whose clamp bounds size and type only. Validated at the one agent-authored boundary (`operator_queue_service._validated_addressee`, inside the #1632 clamp) against `client_portal.service.agent_on_roster(agent, email, include_owned=False)` — an agent may address only someone already on its roster — and **fails closed**: malformed, off-roster, or an unreadable roster all drop to NULL. The read/answer surface is **OSS core** (ent#428): `client_portal/asks/`, mounted unconditionally in `main.py` at `/api/enterprise/client-portal/asks` — a prefix that is retained history, like the `enterprise_`-prefixed portal tables, not a licensing claim. It re-checks the roster at read time so a revoked share stops showing an already-raised ask, projects an explicit client-facing shape (never `context`), and answers through the same ask sink as the operator respond route (`services/ask_service.py`, trinity-enterprise#611), so the ledger, write-back, audit and the WS trigger are unforked, and it is person-only like respond (a non-person platform principal gets 403 `person_required`). `responded_by_id` stays NULL for a client answerer — there is no `users` row, and inventing one would be a lie in the audit trail.
**Addressed asks (ent#364).** `operator_queue.addressed_to_email` names the HUMAN an item is for; `agent_name` only ever said which agent. NULL — every pre-ent#364 row — means "the operator answers", so the column is additive by construction. It is a **column and not a key in `context`** because it is an authorization decision (who may answer, whose Workspace sidebar it appears in) and `context` is agent-authored JSON whose clamp bounds size and type only. Validated at the one agent-authored boundary (`operator_queue_service._validated_addressee`, inside the #1632 clamp) against `client_portal.service.agent_on_roster(agent, email, include_owned=False)` — an agent may address only someone already on its roster — and **fails closed**: malformed, off-roster, or an unreadable roster all drop to NULL. The read/answer surface is **OSS core** (ent#428): `client_portal/asks/`, mounted unconditionally in `main.py` at `/api/enterprise/client-portal/asks` — a prefix that is retained history, like the `enterprise_`-prefixed portal tables, not a licensing claim. It re-checks the roster at read time so a revoked share stops showing an already-raised ask, projects an explicit client-facing shape (never `context`), and answers through the same ask sink as the operator respond route (`services/ask_service.py`, trinity-enterprise#611), so the ledger, write-back, audit and the WS trigger are unforked, and it is person-only like respond (a non-person platform principal gets 403 `person_required`). `responded_by_id` stays NULL for a client answerer — there is no `users` row, and inventing one would be a lie in the audit trail. **The list fails loud (trinity-enterprise#610 PR A0):** `GET /api/enterprise/client-portal/asks` returns **503 `{code: "asks_unavailable", message}`** with `Retry-After: 20` (the Workspace poll interval) when any of its queue reads raises (the agent set, the count or the page, #3059) or the read-time roster re-check cannot read the roster (`asks/service.AsksUnavailable`, strict `_on_roster`); it never answers an outage with `[]`. `answer_ask`'s roster failure stays the uniform 404.

**Ingestion caps (#1632).** There is **no HTTP create** — every operator-queue item is created through `db.create_operator_queue_item`, and the only untrusted producer is the sync service reading the agent-authored `~/.trinity/operator-queue.json` (`operator_queue_service._sync_agent`). #1402 makes this queue the approval channel for irreversible actions, so a compromised / prompt-injected agent that floods plausible "approve this" items causes operator fatigue → reflexive approval (XSS is already handled by DOMPurify; the exposure is volume + social engineering). The one agent-authored seam is bounded by two independent limits plus field hygiene, all env-tunable (see requirements §26.7): (1) a per-agent **pending-DEPTH cap** (`db.count_operator_queue_pending_for_agent`, default 25) — the **primary** bound, DB-measured ⇒ Redis-independent; at the cap the sync **stops** ingesting (never drips a growing file) and holds the surplus behind one aggregated alert; (2) a per-agent + fleet **create RATE limit** (`rate_limiter.check`, 60/60s + 300/60s, fail-open); (3) a total field-hygiene clamp (`_clamp_ingested_item`, inside the #1525 create try/except) — truncate-with-marker title/question, context/options over cap → marker (validated `execution_id` only), non-dict context → `{}`, `created_at` normalized to ingest time, priority validate-only; (4) a **reserved-id guard** rejecting agent ids that start with a platform-reserved prefix (`queue-flood-`/`poison-`/`cb-dormant-`/`sync-failing-`/`git-bloat-`/`skill-not-found-`/`val_`/`system-seed-`/`base-image-stale-`/`alert-budget-`) so an agent can't pre-create — and via `on_conflict_do_nothing` silence — its own flood alarm or the #1402 poison alert, plus a malformed/oversize-id reject; (5) a per-cycle scan bound (skip an oversized file wholesale; cap at 500 requests/cycle). The 5s sync loop is **leader-locked** (`opqueue:leader`, mirror monitoring #1464) so `--workers 2` doesn't double-charge / double-broadcast / double-scan. Platform creates bypass `_sync_agent`, and the exemption is **scoped by influence, not by caller location (#1677)**: a *platform-only* emitter (volume bound by platform cadence — edge-triggered, idempotent/bucketed id, leader-locked, or operator-driven: lease-reaper poison-park #1402, `validation_service._notify_operator_on_failure` — converted to a **direct DB create** in #1632 — and the internal breaker/skill/git alerts) stays direct and unthrottled, while an *agent-influenceable* one — `task_execution_service._alert_skill_not_found` (#1410), whose per-command dedup distinct unknown slash-commands defeat at $0/turn — routes through `operator_queue_service.create_bounded_alert`: a per-(agent, registered-type) pending-DEPTH budget (`OPERATOR_ALERT_MAX_PENDING_PER_TYPE`, default 5; type derived from `item["type"]` against the `_BUDGETED_ALERT_TYPES` frozenset; **fail-closed on every arm** — an unreadable count / unregistered type / failed create suppresses the ALERT only, the FAILED execution rows stay the primary surface, and the paired notification is gated on the same bool), with ONE cooldown-gated `alert-budget-{agent}-{type}-b{bucket}` episode alert per window (deterministic bucket ⇒ the `(agent_name, request_id)` on-conflict target dedups cross-worker; no `held` count, no agent-controlled text). The classification is CI-forced by the AST caller-parity guard `tests/unit/test_1677_operator_alert_emitters.py` (every `create_operator_queue_item` call site must be allowlisted platform-only or routed; OSS tree only — the private submodule owns its twin); a sink-level default bound was rejected because it fails QUIETLY at the load-bearing poison-park create where the parity test fails LOUDLY at CI (#1890's lesson deliberately inverted). A generous DB-sink belt in `create_item` (reject title>4 KiB / question>16 KiB / context>64 KiB / id>512; the derived `execution_id` COLUMN belted to None on non-str/>512, #1677) is a second layer so the exemption is never solely load-bearing (#1525 validate-at-boundary-AND-at-sink). A depth-held / rate-skipped / oversize-file episode emits **one** `queue-flood-{agent}-{utc_now_iso()}` alert (un-guessable id, priority `high`, in-memory cooldown, emit-failure-safe). The platform-emitter residual named on the pull-mode default-ON gate list (#1081 / `TARGET_ARCHITECTURE.md`) is closed by #1677; the remaining gate items are unchanged.

Expand Down
32 changes: 32 additions & 0 deletions docs/memory/feature-flows/workspace-agent-page.md
Original file line number Diff line number Diff line change
Expand Up @@ -364,6 +364,23 @@ was already live: `_asks` excluded `alert` items while `/asks` includes them, so
the page already showed them through `<PortalAsks>`; only the duplicate hid
them.

**An outage is not "nothing waiting" (trinity-enterprise#610 PR A0).**
"Unreadable-roster fail-closed" above used to mean the ask silently vanished:
`list_asks` caught every error and returned `[]`, and `fetchAsks` set
`asks = []` and `asksAvailable = false` on any failure, so a 5xx cleared this
section and the sidebar badge alike. Now the list raises `AsksUnavailable` (a
queue-read fault, or a roster that cannot be READ — an agent genuinely off the
roster is still dropped) and the route answers **503 `asks_unavailable`**. The
store keeps the last good list on any non-404/403 failure, sets `asksFailed`,
leaves `asksAvailable` as it was, and latches `asksLoaded` on the first success
— so `<PortalAsks>` keeps rendering what it last knew instead of blanking.
A **401** is not an outage but the end of the session: `fetchAsks` calls
`endSession({expired: true, resumePath})` exactly as the roster fetch does
(ent#375), because the 20s poll never re-reads the roster and `portalHttp`'s 401
interceptor acts only for a platform session. A read that resolves after its
session ended (sign-out, expiry, another client) is dropped, never written.
The 503 carries `Retry-After: 20`.

### The Overview row is unconditional, and asks moved below it (#2169)

#2161 put asks and recent work in a grid whose column count was **bound to
Expand Down Expand Up @@ -558,6 +575,21 @@ destination" is finally true.

## Tests

`tests/unit/test_ent610_asks_unavailable.py` (trinity-enterprise#610 PR A0) — the
asks list fails loud: a queue-read fault and an unreadable roster each raise
`AsksUnavailable` and answer **503 `asks_unavailable`**; a clean off-roster agent
is still dropped with a 200; `answer_ask` stays a uniform 404 on a roster outage;
the suggestions build still degrades to no ask ids (a guard for the new raise
path, not a regression test: it also passes on the pre-fix code). The raise and
roster halves are mutation-tested (revert the raise to `return []`; make
`_on_roster` swallow again → red).
`src/frontend/tests/unit/workspaceAsks.spec.js` pins the store half: a 5xx keeps
the last good list with `asksFailed` set and `asksAvailable` untouched, a 404
is absence (and resets `asksLoaded`), a success clears the failure, a
sign-out drops the kept list so the next client never inherits it, a 401 ends a
portal-token session and drops the list, and a read resolving after sign-out
writes nothing (each mutation-tested).

`tests/unit/test_ent360_workspace_agent_page.py` — the projections (no
message/cost/model; alerts excluded; `context` never present, asserted against
the rendered repr too), cross-agent report isolation and its 404 uniformity,
Expand Down
5 changes: 3 additions & 2 deletions docs/memory/requirements/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -564,8 +564,9 @@
- **Escalation = a visible count:** the list response carries `undelivered_count` and `closed_by_filer_count`; the Operations header renders both and offers one-click cancel of `closed_by_filer` rows through the existing bulk-cancel. No new queue item is minted about queue items.
- **Audit** (`platform_audit_service`, `event_type=operator_queue`, `source=system`, target = the row uuid, details = ids and enums only, never agent text): `ingested` · `diverged` (→ changed / closed_by_filer / missing / stale_id) · `reconciled` (diverged → confirmed) · `written_back` · `undeliverable`. The `confirmed ↔ unconfirmed` flap is **never** audited (a fleet restart writes zero rows); the edge is the UPDATE's rowcount under `col IS NULL OR col != :v`, so two overlapping leaders cannot double-write.
- **Surfaces:** `QueueCard` / `ResolvedCard` (Operations), `/m` (MobileAdmin), and `PortalAsks` render ONE shared rule (`utils/operatorQueue.js::queueSyncBadge`); the portal projection exposes a coarse `sync ∈ {confirmed, changed, closed, unconfirmed}` + `aging` only — no reason, no timestamp. MCP `list_operator_queue` passes the fields through.
- **Files**: `src/backend/services/operator_queue_service.py`, `src/backend/db/operator_queue.py`, `src/backend/routers/operator_queue.py`, `src/backend/client_portal/asks/{models,service}.py`, `src/backend/services/agent_client/client.py`, `docker/base-image/agent_server/routers/files.py`, `src/backend/config.py`, `src/frontend/src/utils/operatorQueue.js` + the four callers, migrations `operator_queue_sync_state` (SQLite) / `0073_operator_queue_sync_state` (Alembic)
- **Tests**: `tests/unit/test_2915_operator_queue_sync_honesty.py`, `tests/unit/test_2915_agent_server_files_if_match.py`, `src/frontend/tests/unit/operatorQueueSyncBadge.spec.js`, `src/frontend/tests/unit/queueCardSyncBadge.spec.js`
- **The Workspace asks read fails loud, never empty (trinity-enterprise#610 PR A0):** `GET /api/enterprise/client-portal/asks` answers **503 `{code: "asks_unavailable"}`** when the queue read fails **or** the roster cannot be read (in list mode an unreadable roster is an outage, not "off the roster"; a genuinely off-roster agent is still dropped silently, and `answer_ask` keeps its uniform 404). The 503 carries `Retry-After: 20`. The store (`stores/clientPortal.js::fetchAsks`) treats only 404/403 as absence and a 401 on a portal-token session as expiry (`endSession({expired: true})`, the roster fetch's own handling, which clears the list); any other failure sets `asksFailed`, **keeps the last good list** and leaves `asksAvailable` untouched, and `asksLoaded` latches on the first success (a 404/403 resets it); `signOut` clears the kept list and all three flags so the next client never inherits it, and a read that resolves after its session ended is dropped — so a 5xx can no longer read as "nothing needs you". This deliberately reverses the pre-#610 behaviour of clearing the list on a real failure (the contract's "never overwrite the data with a synthetic empty payload in a catch", ent#253). The suggestions build keeps degrading through its own `attempt(..., [])`.
- **Files**: `src/backend/services/operator_queue_service.py`, `src/backend/db/operator_queue.py`, `src/backend/routers/operator_queue.py`, `src/backend/client_portal/asks/{models,service,router}.py`, `src/backend/services/agent_client/client.py`, `docker/base-image/agent_server/routers/files.py`, `src/backend/config.py`, `src/frontend/src/utils/operatorQueue.js` + the four callers, `src/frontend/src/stores/clientPortal.js` (`fetchAsks`, `signOut`), migrations `operator_queue_sync_state` (SQLite) / `0073_operator_queue_sync_state` (Alembic)
- **Tests**: `tests/unit/test_2915_operator_queue_sync_honesty.py`, `tests/unit/test_2915_agent_server_files_if_match.py`, `src/frontend/tests/unit/operatorQueueSyncBadge.spec.js`, `src/frontend/tests/unit/queueCardSyncBadge.spec.js`, `tests/unit/test_ent610_asks_unavailable.py`, `src/frontend/tests/unit/workspaceAsks.spec.js`
- **Residuals (registered)**: `closed_by_filer` rows still count toward the depth cap until cancelled (trinity-enterprise#619); a response undelivered because the entry changed *after* the answer has no re-answer path (ent#619/#611 — narrowed by OPS-001-ENDINGS: the agent reads the answer back by its own `request_id` over MCP, so a refused file write-back no longer leaves it without the decision); a re-ask with a reused id after `acknowledged` is swallowed by #1631's design (ent#619).
- **Env knobs** (all env-tunable, generous by design — "cap a hostile/runaway agent, not throttle normal use"): `OPERATOR_QUEUE_CREATE_RATE_LIMIT` (60), `OPERATOR_QUEUE_CREATE_RATE_WINDOW` (60), `OPERATOR_QUEUE_FLEET_CREATE_RATE_LIMIT` (300), `OPERATOR_QUEUE_MAX_PENDING_PER_AGENT` (25), `OPERATOR_QUEUE_MAX_SCAN_PER_CYCLE` (500), `OPERATOR_QUEUE_MAX_FILE_BYTES` (2 MiB), `OPERATOR_QUEUE_TITLE_MAX` (300), `OPERATOR_QUEUE_QUESTION_MAX` (4000), `OPERATOR_QUEUE_CONTEXT_MAX_BYTES` (8192), `OPERATOR_QUEUE_OPTIONS_MAX_BYTES` (4096), `OPERATOR_QUEUE_ID_MAX` (256), `OPERATOR_QUEUE_EXECUTION_ID_MAX` (128), `OPERATOR_QUEUE_FLOOD_ALERT_COOLDOWN_SECONDS` (300), `OPERATOR_ALERT_MAX_PENDING_PER_TYPE` (5, #1677).
- **Files**: `src/backend/services/operator_queue_service.py`, `src/backend/db/operator_queue.py`, `src/backend/services/validation_service.py`, `src/backend/services/task_execution_service.py` (#1677), `src/backend/database.py`
Expand Down
16 changes: 15 additions & 1 deletion src/backend/client_portal/asks/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,12 @@

from . import service
from .models import WorkspaceAsk, WorkspaceAskAnswer
from .service import AskError
from .service import AskError, AsksUnavailable

# The 503 `asks_unavailable` Retry-After, in seconds (trinity-enterprise#610,
# PR A0). It is the Workspace's asks poll interval (`Portal.vue::ASKS_POLL_MS`):
# its next read is the retry anyway.
ASKS_RETRY_AFTER_SECONDS = "20"

router = APIRouter(
prefix="/api/enterprise/client-portal/asks",
Expand Down Expand Up @@ -55,10 +60,19 @@ def list_asks(
stops being true. `include_ended` (trinity-enterprise#611) adds the asks that
ended in the last 7 days, so a person sees how an ask ended instead of
watching it vanish; the sidebar count stays pending-only on the client.

An unreadable queue or roster is **503 `asks_unavailable`**, never `[]`
(trinity-enterprise#610, PR A0): an empty list is a claim that nothing is
waiting, and the client keeps its last good list on this answer.
"""
try:
page = service.list_asks_page(principal.email, principal.is_platform, agent_name,
include_ended=include_ended, limit=limit, cursor=cursor)
except AsksUnavailable:
raise HTTPException(status_code=503, detail={
"code": "asks_unavailable",
"message": "Couldn't load your asks — try again.",
}, headers={"Retry-After": ASKS_RETRY_AFTER_SECONDS})
except AskError as e:
_raise(e)
response.headers["X-Total-Count"] = str(page.total)
Expand Down
Loading
Loading