Skip to content

fix(dispatch): carry inter-agent chain depth through loops, schedule triggers, session turns and events (#2973) - #3120

Merged
vybe merged 3 commits into
devfrom
feature/2973-chain-depth-laundering
Sep 30, 2026
Merged

vybe merged 3 commits into
devfrom
feature/2973-chain-depth-laundering

Conversation

@obasilakis

Copy link
Copy Markdown
Contributor

Summary

  • Follow-up to bug: agent-to-agent chat chains have no depth guard — a runaway A→B→A bounce is bounded only by capacity parks and hop timeouts #2806. An agent near inter_agent_max_chain_depth could reset its depth to 0 by starting a loop, manually triggering a schedule, driving a chat-session turn, or emitting an event its subscription answers. All four now inherit the caller's depth and are refused at the max with the same named 403 inter_agent_depth_exceeded.
  • Event depth travels as a signed chain_depth claim in the EVT-001 loopback JWT, so an emit on another agent's behalf and system agent.task.* completions (finished row's depth + 1) carry it too. _chain_caller falls back to vouched_source_agent.
  • New per source→subscriber hourly event-dispatch budget (CSO 2026-09-29 Finding 3, from the issue thread): ops setting event_dispatch_max_fires_per_hour, default 120, one high-priority notification per window, fail-open on Redis errors.

Changes

  • routers/{loops,schedules,sessions,event_subscriptions}.py — call enforce_inter_agent_depth; refusal propagates to the new app-level handler (error_handlers.inter_agent_depth_exceeded, registered in main.py).
  • services/loop_service.py, db/loops.py + agent_loops.chain_depth (SQLite migration loop_chain_depth, Alembic 0084, schema.py, tables.py).
  • src/scheduler/* — ExecutionOrigin.chain_depth (validated 1–1000), stamped on insert, kept on retry.
  • services/event_dispatch_service.py — loopback claim, terminal-event depth, dispatch budget; dependencies.py / models.User.loopback_chain_depth; services/dispatch_admission_service.py — claim read before the root early-return.
  • services/task_execution_service.execute_task(chain_depth=).
  • services/monitoring_alerts.py — budget alert; config.py / settings_service.py — new ops setting.
  • MCP: client.ts::depthRefusalFromError; run_agent_loop, trigger_agent_schedule, emit_event return the refusal as a non-retryable result.
  • Docs: requirements/core-agent.md §9.1.1, requirements/mcp.md, architecture/{execution,database,api-endpoints}.md, feature flows, IA-04 gaps, learnings fragment, CSO diff report.

Test Plan

Deferred: webhook tokens, agent-created cron schedules and self-reminders still start a depth-0 root — #3116 (needs a design call on whether a stored trigger carries its creator's depth forever).

Fixes #2973

🤖 Generated with Claude Code

…triggers, session turns and events (#2973)

Follow-up to #2806. Loop starts, manual schedule triggers, chat-session turns
and agent event emits by an agent principal now inherit the caller's chain
depth and are refused past inter_agent_max_chain_depth with the named 403
inter_agent_depth_exceeded, instead of starting a fresh depth-0 root.

- Loops: depth stored on agent_loops.chain_depth (SQLite migration +
  Alembic 0084) and stamped on every iteration row.
- Schedule trigger: depth forwarded to the scheduler; retries keep it.
- Session turns: depth passed through run_resumable_turn to execute_task.
- Events: depth signed into the EVT-001 loopback JWT (chain_depth claim),
  minted whether or not the source is vouched; task-completion events carry
  the finished row's depth + 1 (the max when the row is unreadable);
  _chain_caller falls back to vouched_source_agent.
- Per source->subscriber hourly event-dispatch budget in
  trigger_subscription (ops setting event_dispatch_max_fires_per_hour,
  default 120, one alert per window, fail-open on Redis errors).
- App-level handler maps InterAgentDepthExceeded to the #2806 403 body.
- MCP run_agent_loop, trigger_agent_schedule and emit_event render the
  refusal as a non-retryable result.

Mutation-checked: reverting each fix turns its test red in
tests/unit/test_2973_depth_new_roots.py, test_2973_event_dispatch_budget.py
and src/mcp-server/src/tools/depth-refusal.test.ts.

Webhook tokens, agent-created cron schedules and self-reminders remain
roots; tracked in #3116.

Fixes #2973

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@obasilakis
obasilakis force-pushed the feature/2973-chain-depth-laundering branch from 3d4278c to 6b02181 Compare September 30, 2026 13:25
…ons, budget alert write (#2973)

Review follow-ups on #3120: the cold retry in run_resumable_turn keeps
chain_depth (mutation-checked), a missing session answers 404 before the
depth 403, and the budget alert writes a high-priority notification on the
subscriber. Doc counts updated for the four covered paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@obasilakis obasilakis added the ui PR touches the frontend UI — triggers Playwright e2e tests label Sep 30, 2026
 single-flight scan

The dispatch-budget alert flag is a once-guard (no release, no token, TTL
only), the same non-lock nx shape as the heartbeat and voip entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vybe

vybe commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

merge-train: this PR is on today's train with #3122, #3119 and #3085. The one Tier 1 blocker was the test_1920 allowlist, and your 5c0bc209d already fixes it.

Two gaps to consider. Neither blocks this train. Each was found by a mutation that stayed green:

  • main.py handler registration isn't pinned. Deleting app.add_exception_handler(InterAgentDepthExceeded, …) (around main.py:1217) leaves all 48 new tests green, because they build their own app. In production, every depth refusal would then be a 500 instead of the named 403. One assertion covers it: InterAgentDepthExceeded in main.app.exception_handlers.
  • The scheduler's manual-trigger hand-off isn't pinned. Changing chain_depth=origin.chain_depth in src/scheduler/main.py to None leaves 2973, 1968 and 1970 green. Extending the handler-driven test in test_1968 to send a chain_depth and assert it on the stored row would cover it.

Also worth a decision, from /cso --diff at low severity: anyone with shared access to agent X can use up X→S's hourly fire budget through POST /api/agents/X/emit-event. That silently skips X's real events for up to an hour. You could key that route's budget on the emitting principal, or record it in #3116.

This PR goes in ahead of #3085, and #3085 will re-parent onto your 0084_agent_loops_chain_depth.

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge-train: batch validated on train/20260930-1429 (#3126, all gates green)

@vybe
vybe merged commit f93a124 into dev Sep 30, 2026
28 checks passed
vybe pushed a commit that referenced this pull request Sep 30, 2026
…5_ent720_email_identity (#3085)

#3120 landed 0084_agent_loops_chain_depth off the same parent as this PR's
0084_ent720_email_identity. Disjoint tables, so the revision is re-parented:
0085_ent720_email_identity <- 0084_agent_loops_chain_depth. SQLite list keeps
both entries, dev's first — the same resolution train #3126 was gated on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trinity-ability pushed a commit that referenced this pull request Oct 9, 2026
…#3432)

The hand-written fixture DDL lacked the chain_depth column that #3120 added
to SchedulerDatabase.create_execution, so every agent-path INSERT in the
scheduler tier raised OperationalError (70 failed / 173 passed). With the
column: 3 failed / 240 passed; the remaining three are a separate drift
(claim_token), left out of this fixture-only commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trinity-ability pushed a commit that referenced this pull request Oct 9, 2026
…#3432)

The hand-written fixture DDL lacked the chain_depth column that #3120 added
to SchedulerDatabase.create_execution, so every agent-path INSERT in the
scheduler tier raised OperationalError (70 failed / 173 passed). With the
column: 3 failed / 240 passed; the remaining three are a separate drift
(claim_token), left out of this fixture-only commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vybe added a commit that referenced this pull request Oct 11, 2026
* test(scheduler): conftest schedule_executions DDL carries chain_depth (#3432)

The hand-written fixture DDL lacked the chain_depth column that #3120 added
to SchedulerDatabase.create_execution, so every agent-path INSERT in the
scheduler tier raised OperationalError (70 failed / 173 passed). With the
column: 3 failed / 240 passed; the remaining three are a separate drift
(claim_token), left out of this fixture-only commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(scheduler): remove the process-schedule remnant from the scheduler source (#3432)

The scheduler still created process_schedules / process_schedule_executions
at boot, re-read them every sync tick, and executed rows by POSTing to
/api/processes/{id}/execute, a route the backend lost in #430. Nothing can
create a row. Removed: the boot DDL, the snapshot + sync, the job
add/remove/execute path, the skipped-execution recorder and its
max-instances branch, the database section, the two models, the stranded
_INTEGRITY_ERRORS import. The two tables are left in place on existing
installs (no migration).

The agent-schedule path, its lock-denial audit (#1969/#1808) and reminder
machinery (#1296) are unchanged; the _on_job_max_instances unknown-prefix
warning stays (reminder jobs share the listener).

Red-first tests: test_legacy_process_schedule_tables_are_ignored (an enabled
legacy row registered process_schedule_ps-legacy at HEAD) and
test_on_job_max_instances_legacy_process_prefix_is_unknown (the legacy id
reached _record_skipped_process_schedule at HEAD). Tests that exercised the
removed identifiers are deleted or re-pinned in this commit: test_1994
deleted, process cases cut from test_skipped_executions / test_1474 /
test_sync_loop, test_1296's ensure_process_schedules_table call dropped,
test_1969's slice end re-anchored on `async def _execute_schedule(`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(scheduler): test docstrings stop naming the removed process-schedule writer (#3432)

test_1832 narrated three duration writers (one was the deleted
process_schedule_executions finalizer); test_1823 pointed at the deleted
test_1994 as its structural model, now test_1969. Docstrings only; the
scanner logic is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(scheduler): docs, run-core.sh and nightly workflows drop the process-schedule remnant (#3432)

- tests/run-core.sh and both nightly workflows drop --ignore=process_engine
  (the directory no longer exists); the workflow comments stop citing a
  run-core.sh line number.
- test_1896's literal --ignore=process_engine assertion becomes a parity
  check: the workflow's --ignore set must equal run-core.sh's (red with
  run-core.sh edited alone, green once both sides match).
- scheduler-service.md: Flow 3 (process execution) removed and Flows 4-10
  renumbered 3-9; process sync, events, max-instances branch, DB operations
  and method rows removed; revision history entry added.
- requirements/scheduling.md, architecture/execution.md: the scheduler has
  one duration finalizer, not two.
- architecture/database.md: the two tables are recorded as orphans left on
  upgraded installs (no migration).
- architecture.md Invariant #16: "+ process-schedule variants" removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(scheduler): conftest schedule_executions DDL carries claim_token (#3433)

The hand-written fixture DDL lacked the claim_token column (TEXT, as in
db/schema.py) that SchedulerDatabase's require_unclaimed guard (#2514)
reads, leaving three scheduler tests red:
test_1472::test_a2_dispatch_exception_advances_next_run and two in
test_async_dispatch. With the column: 243 passed, 0 failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(testing): strategy doc stops listing the removed process_engine exclusion (#3432)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Trinity Agent (trinity) <trinity-agent@ability.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: trinity-ability <309458136+trinity-ability@users.noreply.github.com>
vybe added a commit that referenced this pull request Oct 11, 2026
…references (#3433) (#3472)

* test(scheduler): conftest schedule_executions DDL carries chain_depth (#3432)

The hand-written fixture DDL lacked the chain_depth column that #3120 added
to SchedulerDatabase.create_execution, so every agent-path INSERT in the
scheduler tier raised OperationalError (70 failed / 173 passed). With the
column: 3 failed / 240 passed; the remaining three are a separate drift
(claim_token), left out of this fixture-only commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(scheduler): conftest schedule_executions DDL carries claim_token (#3433)

The hand-written fixture DDL lacked the claim_token column (TEXT, as in
db/schema.py) that SchedulerDatabase's require_unclaimed guard (#2514)
reads, leaving three scheduler tests red:
test_1472::test_a2_dispatch_exception_advances_next_run and two in
test_async_dispatch. With the column: 243 passed, 0 failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(services): remove dead service symbols, checkpoint A1 (#3433)

Removes symbols with no production caller on the live tree (re-verified
over src, docker, scripts, tests, docs/memory and the enterprise
submodule immediately before deletion):

- ssh_service: cleanup_expired_keys, list_active_keys, cleanup_agent_keys
  (cleanup_agent_credentials itself is kept; see #3433 adjacent finding)
- telegram_media: process_photo, process_document (+ os/tempfile imports)
- github_service: get_github_service, GitHubAuthError, GitHubPermissionError
- monitoring_alerts: alert_stuck_execution
- subscription_auto_switch: handle_rate_limit_error
- template_service: get_name_from_template
- skill_service: _persisted_commit
- settings_service: clear_template_registry_lkg
- system_agent_service: is_registered
- slot_service: is_at_capacity + get_active_count (its only caller)
- slack_service: verify_slack_signature (+ get_slack_signing_secret import)
- heartbeat_service: heartbeat_status (T1: 2 unique tests re-pointed to
  heartbeat_status_bulk, 4 duplicates deleted)

Tests that only exercised a removed symbol are deleted; memory docs that
described them are corrected. Targeted unit files: 164 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(services): remove dead service symbols, checkpoint A2 (#3433)

Removes symbols with no production caller on the live tree (re-verified
over src, docker, scripts, tests, docs/memory and the enterprise
submodule immediately before deletion):

- dispatch_breaker: get_all_dispatch_states (live reader is
  get_dispatch_states_for; its Redis-None fail-open is already pinned by
  test_states_for_no_redis_is_all_closed)
- idempotency_service: derive_schedule_key (scheduler builds sched:{id})
- agent_mcp_key_service: build_mcp_key_env_overrides (test re-pointed to
  _mint_and_build_env)
- mcp_tool_names: compute_tool_name (3 slug tests re-pointed to
  compute_tool_names)
- agent_client.http_pool: _get_http_client (tests use _acquire_client)
- agent_call_limiter: _set_cancel_sync
- a2a_payment_gate: AllowlistRefused
- a2a_gate: get_skills_provider (T6)
- compatibility.static_checks: _is_skill_md
- git_service.trinity_files: _persistent_state_for (4 fallback tests
  re-pointed to _read_trinity_yaml_list over the persistent-state file)
- skill_sets: split_set_ref
- event_bus: _serialize
- backlog_service: get_backlog_service + the _backlog_service global

redelivery_governor.distinct_failing_count is kept (T2: declared
test/observability accessor). Targeted unit files: 374 passed, 32 skipped
(lupa-less fakeredis + a pre-existing #1609 skip).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(adapters): remove dead adapter, db and utils symbols, checkpoint B (#3433)

Removes symbols with no production caller on the live tree (re-verified
over src, docker, scripts, tests, docs/memory and the enterprise
submodule; DatabaseManager delegates by name, no __getattr__ passthrough):

- message_router: _get_channel_timeout + _DEFAULT_CHANNEL_TIMEOUT (caller
  removed on purpose in #221; the channel_timeout_seconds setting has no
  other reader, and the docs now say so), _format_file_size,
  _sanitize_filename (+ _FILENAME_MAX_LENGTH, _FILENAME_SAFE_CHARS_RE and
  the os/unicodedata/format_file_size imports only it used). The copy was
  code-identical to upload_service.sanitize_filename, the live path; its
  8 tests are re-pointed there (T4), as is the file-size test.
- voip_audio: pop_frames
- db.schedules.webhooks: set_webhook_enabled, _decrypt_webhook_secret
- db.slack_channels: unbind_channel
- db.telegram_channels: delete_groups_for_binding
- db.role_readiness: delete_role_readiness (+ the delete import)
- db.agent_cleanup: find_orphan_agent_names
- utils.password_validation: is_password_weak
- scheduler database: should_freeze_schedules (T3: asserts re-pointed to
  sync_freeze_reason)

agent_cleanup.registered_columns is kept (T2: parity-guard accessor).
Targeted unit files: 146 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(scheduler): remove dead scheduler symbols, checkpoint C (#3433)

Removes scheduler symbols with no production caller (re-verified over
src, docker, scripts, compose, tests, docs/memory and the enterprise
submodule; the backend's update_schedule and asyncio's loop.run_forever
are unrelated same-name symbols):

- service: add_schedule, remove_schedule, update_schedule,
  reload_schedules (the dead push-API section; the scheduler syncs by
  polling) and run_forever. The container runs
  `python -m scheduler.main` -> SchedulerApp.start() ->
  _run_until_shutdown(), which carries its own heartbeat + sync loop; no
  compose command override, no script calls it.
- database: get_recent_executions
- locking: is_schedule_locked
- models: TriggerSource, AgentTaskResponse + AgentTaskMetrics

Tests: add/remove job tests re-pointed to _add_job/_remove_job; the
disabled-schedule test is deleted with add_schedule (its enabled gate
existed only there); update/reload tests and the reload reminder-reclaim
test deleted (reclaim stays covered via _sync_schedules/_reconcile_reminders);
the skipped-execution probe now reads the row with a direct SELECT.
No process-schedule symbol is touched (#3432 owns those);
reload_schedules is deleted whole, so a textual conflict with #3432's
cut inside it resolves by taking the deletion (T5).

scheduler_tests tier: 235 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(heartbeat): bulk test pins the payload fields the single-agent test carried (#3433)

A1 deleted test_status_alive with heartbeat_status. Its unique asserts
(heartbeat_memory_mb, heartbeat_active_executions, the last-heartbeat age
bound) now live on test_bulk_mixed_states_single_round_trip, the live
path. Mutation-checked: nulling the memory_mb / active_executions
pass-through in _compute_status turns this test red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(scheduler): sync loop refuses and drops disabled schedules (#3433)

The deleted test_add_disabled_schedule only exercised the dead
add_schedule push API. Pin the live sync path instead: a disabled row
(present at boot or inserted between ticks) never becomes a job, and a
schedule disabled between ticks is removed on the next sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: ssh cleanup and scheduler client snippets describe the code as it is (#3433)

- ssh-access / async-docker-operations: cleanup_agent_credentials() exists
  but no stop/delete path calls it; say so instead of claiming it runs at
  agent stop/delete (follow-up bug).
- scheduling / execution-queue / model-selection / scheduler-service:
  src/scheduler/agent_client.py was removed in v0.7.0. The scheduler
  dispatches through _call_backend_execute_task -> POST
  /api/internal/execute-task; the backend's TaskExecutionService POSTs the
  agent's /api/task via agent_post_with_retry. Replace the snippets and
  drop the file-table rows that named the missing module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(services): drop the empty section headers the dead-symbol removal left behind (#3433)

slack_service.py kept a 'Request Verification' banner and a module-docstring
bullet for the removed verify_slack_signature (the live check is
adapters/transports/slack_webhook.py::_verify_signature); github_service.py
kept a trailing 'Factory Function' banner for the removed get_github_service.
Comment-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Trinity Agent (trinity) <trinity-agent@ability.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: trinity-ability <309458136+trinity-ability@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ui PR touches the frontend UI — triggers Playwright e2e tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants