Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion constraints.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ httpx==0.28.1
pydantic==2.13.4
pydantic-settings==2.14.2
tenacity==9.1.4
anyio==4.13.0
anyio==4.14.2
certifi==2026.10.19
prometheus-client==0.25.0
redis==7.4.0
ruff==0.15.12
Expand Down
9 changes: 9 additions & 0 deletions docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,15 @@ mandatory even when workspace counters are disabled. Gateway deployments require
`ORCH_BASE_URL` and `ORCH_SERVICE_TOKEN` using the existing control-plane service
credential, with the existing internal TLS settings where configured.

Every queued attempt, including work recovered from Redis, checks the
control-plane acquire endpoint before starting. Closing control-plane
`WORKSPACE_DISPATCH_ENABLED` keeps that work queued even when capacity limits are
disabled; already executing work may drain. Disabled mode requires a compatible
`disabled` acquire response and uses only the local worker semaphore, without
ownership leases, renewal, or release. An incompatible acquire response starts no
work. Apply the usual bounded queue-expiry and cancellation checks before reopening
dispatch after a maintenance window.

Engine `MAX_CONCURRENT_RUNS` remains an independent local worker limit. Queued work
does not hold local/distributed execution gates; a waiting coordinator or approval
releases both gates after checkpointing. Initial queue expiry is 600 seconds and
Expand Down
1 change: 1 addition & 0 deletions docs/exec-plans/active/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ Reviewed 2026-10-18. Implementation-only plans that have landed are in
[Completed plans](../completed/README.md). Historical validation results are not fresh release acceptance.

- [Production Tool Results](production-tool-results.md) — Implementation complete; coordinated staging soak and production release acceptance still pending.
- [Disabled-capacity dispatch gate](disabled-capacity-dispatch-gate.md) — Honor dispatch pauses for accepted and recovered work even when capacity limits are disabled.
139 changes: 139 additions & 0 deletions docs/exec-plans/active/disabled-capacity-dispatch-gate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
# Disabled-capacity dispatch gate

Status: capacity implementation, local validation, and independent review complete;
runtime dependency correction, local validation, and parent review complete.
The follow-up request on 2026-10-26 authorizes final review, publication, and an
in-place demo upgrade. Production acceptance remains separate.

## Goal and scope

Honor the control-plane dispatch pause before starting accepted or recovered work
when `WORKSPACE_CAPACITY_ENABLED=false`. Reuse the existing acquire contract;
do not add an API, dependency, lease, or cross-repository implementation change.
The separately approved dependency correction below updates two existing packages.

Work in the explicitly requested current checkout on `fix/production-readiness`.
The task instructions authorize inline execution without subagents or commits.
This record is the scoped plan and progress ledger; the parent agent owns final
independent review and cross-repository validation.

## Task 1: Gate disabled-mode execution

- [x] Add failing tests in `tests/test_capacity.py` for disabled-mode wait,
queue expiry, recovered durable work, successful disabled acquisition without
ownership/renewal/release, and malformed/mismatched acquisition responses.
- [x] Run `.venv/bin/python -m pytest -q tests/test_capacity.py`; expect the new
disabled-mode tests to fail because execution bypasses acquisition.
- [x] In `execution_engine/capacity.py`, always acquire before starting work.
Preserve existing wait/retry/cancellation bounds and the local semaphore;
accept `disabled` only with local capacity disabled, and reject incompatible
response shapes before execution. Leave enabled lease handling unchanged.
- [x] Update the integration harness's existing control-plane acquire contract
and add local, socket-free coverage using the actual orchestrator client.
- [x] Run targeted tests, `task validate`, and safe local integration checks.
Expected: all invoked checks pass; report environmental limits separately.
- [x] Update `docs/OPERATIONS.md` and record validation below. Leave changes
uncommitted for parent review.

## Review focus

Recovered queues must not bypass the pause. Disabled mode must not acquire
ownership or start lease cleanup. Enabled mode must reject `disabled`, and
disabled mode must reject a grant. Missing/invalid contract versions and
malformed grants must fail closed. Existing bounded queue expiry and local
concurrency must remain unchanged.

## Evidence and decisions

- Root cause reproduced before implementation: disabled mode executed a worker
body without consulting an acquire response of `wait`.
- Scope ruling: use the requested current checkout, repo plan location, and
no-commit/no-subagent workflow instead of generic skill defaults.
- Pre-flight: one task; no shared task interfaces. Existing control-plane acquire
response is authoritative: `wait` or `disabled` with `contractVersion: 1` in
disabled mode, and a positive-integer-generation `granted` response when enabled.
- RED: `.venv/bin/python -m pytest -q tests/test_capacity.py` produced 12 expected
assertion failures and 26 passes before the implementation change. Disabled
work incorrectly started, acquisition was never sent, and malformed grants
could be used. No setup/collection failure obscured the regressions.
- GREEN: `.venv/bin/python -m pytest -q tests/test_capacity.py
tests/test_unit.py::test_worker_keeps_internal_tool_identity_and_adds_readable_model_name`
passed 39 tests, including a socket-free integration of the real orchestrator
client, harness HTTP route, and recovered durable queue through pause/reopening.
- `task validate` passed Ruff, 297 unit tests, contract and harness checks, and
all 29 keyless scenarios. Pytest emitted the existing Starlette/httpx
deprecation warning; there were no failing tests or skipped scenarios.
- `git diff --check` passed. The change introduces no secrets, additional
privileges, API shape, dependency, or database mutation. Enabled-mode ownership
and cleanup logic remain unchanged after acquisition validation.
- Docker integration was not run: read-only `docker ps --format '{{.Names}}'`
could not access the Docker socket under the sandbox (`operation not permitted`).
No containers were started, stopped, modified, or used for tests. Real
multi-service/Redis integration and production acceptance remain unverified.
- Independent review passed with no additional actionable findings; the reviewer
reran all 38 capacity tests. No additional fixes or deferred minor changes were added.

## Task 2: Runtime dependency security correction

Scope: update only AnyIO 4.13.0 to 4.14.2 and Certifi 2026.4.22 to 2026.10.19
in `constraints.txt` and the generated `requirements.lock`. Preserve other locked
versions and all existing capacity changes. No application code, Dockerfile,
container, live provider, or deployment changes are authorized for this task.

- [x] Reproduce the runtime lock audit with `.venv/bin/python -m pip_audit
-r requirements.lock --no-deps --disable-pip --strict --progress-spinner off`.
RED: three advisories in two packages across 21 runtime dependencies.
- [x] Update the two pins and regenerate with the README's `pip-compile
--constraint=constraints.txt --generate-hashes --output-file=requirements.lock
--strip-extras requirements.txt` workflow. Compare every locked version;
expected: only the two approved version changes.
- [x] Install the reviewed lock with `.venv/bin/python -m pip install
--require-hashes -r requirements.lock`; run `.venv/bin/python -m pip check`,
the runtime audit above, and `task validate`. Expected: no broken requirements,
no runtime advisories, and all canonical checks pass.
- [x] Record results and hand off the uncommitted diff for parent review.

Security evidence: upstream marks the [AnyIO TLS hostname advisory](https://github.com/agronholm/anyio/security/advisories/GHSA-82r6-8w77-94w6)
and [Certifi compromised CA advisory](https://github.com/certifi/python-certifi/security/advisories/GHSA-6q48-6c7w-9gw2)
critical. The [AnyIO process-pool advisory](https://github.com/agronholm/anyio/security/advisories/GHSA-5p39-cfhj-2xmp)
is moderate. HTTPX already IDNA-encodes hosts before AnyIO; no application
process-pool use was found. Certifi remains reachable for outbound HTTPS with
normal HTTPX trust; explicit internal TLS uses the configured private CA bundles.

Ruling: retain the approved current-checkout, no-commit workflow and this existing
plan as the ledger. Use the failing dependency audit as the security regression
check and canonical tests for compatibility; no application behavior is changed.
Parent review confirmed exactly the two intended runtime version/hash changes.
Image/base-OS scanning and live
acceptance remain outside this dependency-only correction.

Validation, 2026-10-26:

- Regenerated with `.venv/bin/python -m piptools compile` and the documented
flags above, using a temporary pip-tools cache. Only AnyIO and Certifi versions
and their hashes changed; the other 19 runtime pins are unchanged.
- Hash-locked installation passed. It also synchronized the local virtualenv's
existing Pydantic Settings drift from 2.14.1 to the unchanged locked 2.14.2.
- `.venv/bin/python -m pip check` passed; the runtime lock audit passed with
zero known advisories across 21 dependencies, without ignores or suppression.
- `task validate` passed Ruff, 297 unit tests, contract/harness checks, and all
29 keyless scenarios. Only the existing Starlette/httpx deprecation warning
remained. `git diff --check` passed.
- Separate `.venv/bin/python -m pip_audit --local --strict` still reports 23 raw
entries (15 unique package/advisory pairs) in Filelock 3.29.4, Msgpack 1.2.0,
Pip 25.0.1, Setuptools 82.0.1, and urllib3 2.7.0. These are absent from the
production application lock and were not upgraded in this two-package scope.
The Docker image retains its base installer; image and vendored dependencies
still require an image-level audit before release.
- Local JSON audit and install reports plus the full validation log are in
`/tmp/acornops-production-audit-krjHWi/`, named
`execution-engine-runtime-audit-final.json`,
`execution-engine-runtime-install.json`,
`execution-engine-local-environment-audit-final.json`, and
`execution-engine-security-validate.log`. No containers, application files,
other repositories, commits, releases, or deployments were changed by Task 2.

Release preflight, 2026-10-26: independent final review found no new blocker.
Fresh `task validate` passed all 297 unit tests and 29 keyless scenarios, Ruff,
contracts, and harness. Hosted CI must additionally pass the multi-container
integration and actionable-image-vulnerability gates before tagging `.16`.
18 changes: 12 additions & 6 deletions execution_engine/capacity.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,15 +55,13 @@ async def execute_with_capacity(worker, state):
"""A waiting grant returns to the bounded queue without holding a local slot."""
from execution_engine.run_registry import RunStatus

if not settings.WORKSPACE_CAPACITY_ENABLED:
async with worker._semaphore:
await worker._do_execute_run(state)
return
capacity_enabled = settings.WORKSPACE_CAPACITY_ENABLED
owner = state.capacity_owner_id
queued_since = state.resume_requested_at or state.created_at
try:
grant = await worker.orchestrator_client.capacity(state.run_id, "acquire", {"ownerId": owner})
if grant.get("contractVersion") != 1:
if (not isinstance(grant, dict) or type(grant.get("contractVersion")) is not int
or grant["contractVersion"] != 1):
raise RuntimeError("Incompatible capacity authority")
if grant.get("status") == "wait":
if (datetime.now(UTC) - queued_since).total_seconds() > 600:
Expand All @@ -84,7 +82,11 @@ async def execute_with_capacity(worker, state):
# Reload durable truth if a future checkpoint resume lands on this replica.
worker.registry.forget_local_run(state.run_id)
return
if grant.get("status") != "granted" or not isinstance(grant.get("generation"), int):
if not capacity_enabled:
if grant.get("status") != "disabled":
raise RuntimeError("Dispatch capacity mode differs from engine")
elif (grant.get("status") != "granted" or type(grant.get("generation")) is not int
or grant["generation"] < 1):
raise RuntimeError("Execution capacity denied")
except httpx.HTTPError as error:
transient = not isinstance(error, httpx.HTTPStatusError) or error.response.status_code >= 500
Expand All @@ -102,6 +104,10 @@ async def execute_with_capacity(worker, state):
state.ended_at = datetime.now(UTC)
worker.registry.persist_state(state)
return
if not capacity_enabled:
async with worker._semaphore:
await worker._do_execute_run(state)
return
authority = Authority(state.run_id, owner, grant["generation"])
worker.registry.save_authority(authority)
token = current_authority.set(authority)
Expand Down
13 changes: 7 additions & 6 deletions requirements.lock
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,19 @@ annotated-types==0.7.0 \
--hash=sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53 \
--hash=sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89
# via pydantic
anyio==4.13.0 \
--hash=sha256:08b310f9e24a9594186fd75b4f73f4a4152069e3853f1ed8bfbf58369f4ad708 \
--hash=sha256:334b70e641fd2221c1505b3890c69882fe4a2df910cba14d97019b90b24439dc
anyio==4.14.2 \
--hash=sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494 \
--hash=sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f
# via
# -c constraints.txt
# -r requirements.txt
# httpx
# starlette
certifi==2026.4.22 \
--hash=sha256:3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a \
--hash=sha256:8d455352a37b71bf76a79caa83a3d6c25afee4a385d632127b6afb3963f1c580
certifi==2026.10.19 \
--hash=sha256:035c1e85572b141bf1c3a3eff0e33a8eb74917522f2559075ff8d2fea780a1ae \
--hash=sha256:42d35340cc5b9ba78e818ddbff0c5b1c2b779de448a340fd1b989f5fa8670e1b
# via
# -c constraints.txt
# httpcore
# httpx
click==8.4.0 \
Expand Down
6 changes: 6 additions & 0 deletions tests/harness_orchestrator.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ async def health():
received_events = {}
received_commits = {}

@app.post("/internal/v1/runs/{run_id}/capacity/acquire")
async def capacity_acquire(run_id: str):
"""Mirror the existing disabled-capacity control-plane dispatch gate."""
status = "disabled" if os.getenv("WORKSPACE_DISPATCH_ENABLED", "true") == "true" else "wait"
return {"status": status, "contractVersion": 1}

@app.post("/internal/v1/runs/{run_id}/bootstrap")
async def bootstrap(run_id: str):
return {
Expand Down
Loading
Loading