Skip to content

fix(lifecycle): establish process-global shutdown authority - #123

Merged
khanhkit merged 35 commits into
mainfrom
kit/KitDev2/session-1
Oct 17, 2026
Merged

khanhkit merged 35 commits into
mainfrom
kit/KitDev2/session-1

Conversation

@khanhkit

@khanhkit khanhkit commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

AP-ISS-0149 — one process-global lifecycle and hook authority

Consolidate main-server shutdown ownership into one bounded lifecycle coordinator while keeping true standalone MCP/CLI/server processes independently owned.

Current exact source head: 96db0e0691e4611f50e789be8b851d32dca61275
Current exact source tree: c2e87c29181872103e1c8d12d883c073080c318a
Current hosted base / merge-base: 2b9fafdea56bcb558bc3b72f0bdf113f04d2c586

68b8ffde… and every earlier published head are superseded and MUST NOT be used as final merge evidence.

Lifecycle contract

Main runtime tears down under one absolute deadline in this ordered composition:

STOP_ADMISSION → STOP_SCHEDULERS → DRAIN_ACTIVE → DRAIN_BACKGROUND → FLUSH_DURABLE → CLOSE_RESOURCES → STOP_CHILDREN → EXIT

DRAIN_BACKGROUND is an internal refinement of the approved drain layer: request/job/service producers finish in DRAIN_ACTIVE, then process-owned durable background tasks drain before flush. It does not introduce a second deadline or weaken fail-closed timeout behavior.

  • scripts/dev/run-next.mjs owns main custom-server SIGINT/SIGTERM/SIGHUP and final exit.
  • src/lib/gracefulShutdown.ts is fallback signal/final-exit owner only when a custom owner is absent.
  • leaf modules expose idempotent stop/drain/flush/close contracts rather than exiting or re-signalling the process.
  • JobRegistry and other schedulers distinguish stopping new scheduling from draining already-running work.
  • if any hook exhausts the single global deadline, later phases are skipped; durable resources are never deliberately closed underneath work that is still running.
  • rate-limit watchdog admission stops in STOP_SCHEDULERS; limiter persistence/teardown occurs after active producers quiesce.
  • API Bridge stops admission immediately, retains closing ownership, and exposes an awaited active-relay drain before resource close.
  • webhook and routing OTLP work have explicit drain/flush/drop accounting; stopped OTLP sinks are unregistered from dispatch.
  • DB adapters do not own SIGINT/SIGTERM or permanently replace process.emit.
  • packaged standalone establishes one explicit signal owner before Next starts, awaits AgentProxy lifecycle before transport/final exit, and now has a direct-run hard shutdown bound while keeping duplicate process-group/supervisor signals idempotent.

Review findings dispositioned

Real OCR findings were reproduced RED→GREEN before publication, including routing OTLP unregister, strict shutdown-timeout normalization, supervisor stop-failure propagation, rate-limit phase separation, producer-before-background drain ordering, JobRegistry cron stop fencing, API Bridge stop/drain serialization, provider-limit stale-setup fencing, webhook lifecycle reset safety, custom-runner HTTP close error handling, pre-listen signal handling, and direct packaged-standalone timeout bounding.

Findings whose premise conflicts with current runtime semantics were dispositioned with source/executable evidence instead of redundant machinery: JavaScript run-to-completion closes alleged same-stack registration/listen interleavings, and the coordinator skips all later phases after a timed-out hook. A second local drain budget would violate the approved single-deadline contract.

Latest lifecycle OCR remediation on e91bce7f…:

  1. run-next.mjs attaches a rejection handler to serverClosePromise immediately after creation so a close callback failure cannot surface as an unhandled rejection while lifecycle drain is awaited; the original promise remains rejected and the later await still marks shutdown non-clean.
  2. run-next.mjs checks server.listening before server.close(), so a signal received after handler registration but before bind is a clean admission-close no-op instead of ERR_SERVER_NOT_RUNNING.
  3. standalone-server-ws.mjs resolves SHUTDOWN_TIMEOUT_MS, arms an unref'd hard force-exit timer, and clears it only after terminal cleanup. Duplicate SIGTERM remains idempotent because normal agentproxy serve process-group + parent forwarding can legitimately deliver the same signal twice; the hard timer supplies the bound without cutting durable flush mid-flight.

All current inline review threads are resolved on the current source line.

Exact-head CI blocker follow-up on 777b3e0970: hosted CI run 38197848596 showed all 2785 UI tests passing but failed after suite teardown with the known baseline CursorAgentNudge.test.tsx jsdom race (ReferenceError: window is not defined). The file was unchanged from origin/main; the already-proven AP-ISS-0136 root-unmount cleanup was ported to the current tree as test-only code. Fresh local exact UI command: 415/415 files, 2785/2785 tests PASS, with no unhandled teardown error; focused file 8/8 PASS.

No heap limit, assertion, package check, lifecycle gate, docs gate, or CI guard was weakened to manufacture green.

Exact-head local verification

Current branch is 29 ahead / 0 behind the exact merge-base above. Provenance audit is 29/29 author=KitDev2 and committer=KitDev2. Worktree is clean and local/remote branch SHA exactly match 96db0e0691….

Fresh verification after the latest OCR remediation:

  • TDD RED: 2 new regression tests failed before production changes (custom runner early-close/rejection behavior; standalone direct-run shutdown bound).
  • TDD GREEN: targeted entrypoint tests 9/9 PASS.
  • focused lifecycle/package regression bundle 20/20 PASS, RC=0.
  • npm run typecheck:core: PASS, RC=0.
  • npm run check:open-sse-typecheck: PASS, RC=0, openSseTypecheckErrors=0.
  • npm run check:build-scope: PASS, 9731 < 12000.
  • npm run check:tracked-artifacts: PASS.
  • scoped ESLint on the four latest modified JS/TS files: 0 errors (2 repository-policy ignored-file warnings for the MJS entrypoints).
  • git diff --check: PASS.

Earlier current-line verification that remains applicable includes repo-native AP-ISS-0149 lifecycle/ownership regressions, standalone ownership checks, package closure checks, full hosted Build/Vitest/unit/integration/coverage/package/E2E success on the immediately preceding rebased head, and clean CodeQL/semgrep/DAST/API typecheck results.

File-size ratchet follow-up on 777b3e0970: exact-head a460750705 exposed only open-sse/services/rateLimitManager.ts at 1206/1200 lines. The shutdown cleanup block was simplified without behavior change; local exact ratchet is now 1199/1200 PASS, focused lifecycle wiring 13/13 PASS, and diff-check clean.

Second exact-head OCR remediation on 777b3e0970: fixed rate-limit shutdown idempotence, routing observability re-init, parent signal-delivery escalation margin, strict Windows timeout parsing, LiveWS bind-failure ownership cleanup, and awaitable LiveWS listener drain. Review-only findings for pre-existing scheduler overlap, intentional background unref, JavaScript run-to-completion, and already-handled promises were dispositioned with source evidence. Focused RED→GREEN 35/35 PASS; LiveWS EADDRINUSE 1/1 PASS; core/Open-SSE typechecks PASS; targeted ESLint PASS; file-size 1196/1200 PASS; all PR review threads resolved.

Third exact-head OCR remediation on de8339fff450a285be11a47e781f3b97798c3c9c: rate-limit shutdown single-flight state now clears after settlement so later lifecycle/retry calls get a fresh shutdown operation, and test reset awaits any in-flight shutdown before touching shared limiter state. Two additional OCR premises were closed with executable evidence: routing sink re-registration is Set-idempotent for stable memory/quality sink objects, and strict Windows timeout parsing still maps malformed 30000junk to the documented 31s fallback while preserving 1e3 as 2s including escalation margin. Rebased onto latest origin/main defda037e7; focused exact-head lifecycle/routing/windows bundle 32/32 PASS, file-size ratchet PASS, core/Open-SSE typechecks PASS, targeted ESLint PASS, diff-check clean.

Fourth exact-head OCR remediation on de8339fff450a285be11a47e781f3b97798c3c9c: final learned-rate-limit persistence now propagates DB/write failure into the lifecycle report while debounced background persistence remains best-effort/logged; packaged standalone now refuses creation of new HTTP servers once shutdown begins, freezing listener admission across the pre-drain snapshot. RED→GREEN focused remediation gate 10/10 PASS; rateLimitManager.ts 1197/1200; core/Open-SSE typechecks, file-size ratchet, ESLint and diff-check PASS. Eleven final OCR threads were dispositioned/resolved with exact-head evidence.

Fifth exact-head base-drift reconciliation on cb94c927f0f203ef61b9d01b21c848cd04be821c: main advanced by PR #112 to c0c91edd8b2b49cfbb5653451ab06bc0b0313d49 after the prior exact-head oracle completed. The intervening main commit touches only parked-replay SSE code/tests. A pre-rebase merge-tree was conflict-free; the 27 AP-ISS-0149 commits rebased cleanly, retained 27/27 KitDev2 author+committer provenance, and the rebased tree cbf9ae2f57df4f3d41e1bbf5372297d08bc8b953 exactly equals the precomputed merge-tree. The branch is 27 ahead / 0 behind, git diff --check passes, and the force-push used an exact lease pinned to superseded head de8339fff450a285be11a47e781f3b97798c3c9c. Fresh hosted gates on this rebased exact head supersede all prior hosted merge evidence.

Sixth exact-head review/base reconciliation on 4ffba64f942ab92b2964aaab64a974dc4d15e813: prior exact-head hosted gates completed successfully but OpenCodeReview #195 emitted six new threads. Three findings were reproduced RED→GREEN: API Bridge now owns/destroys upgraded relay sockets during stop, direct API drain idempotently initiates stop, and webhook drain closes notification admission before reporting empty. Three review premises were dispositioned with source/executable evidence: free-proxy cycle failures are already absorbed/logged by runCycle(), a timed-out lifecycle hook skips every later phase rather than closing resources beneath it, and provider-limit startup timers are synchronously cancelled in STOP_SCHEDULERS before DRAIN_ACTIVE. Focused lifecycle bundle 39/39 PASS; core/Open-SSE typechecks and git diff --check PASS.

main then advanced to 08bbb79b5dcbf3823e8826220c771cd7856ae054. The drift had zero changed-file overlap with AP-ISS-0149 and pre-rebase merge-tree was conflict-free at tree b2fa269052e9056fe7d77dc87b0f06f1fc33f4ae. The 28-commit branch rebased cleanly; exact head 4ffba64f942ab92b2964aaab64a974dc4d15e813 has the same tree byte-for-byte and retains 28/28 KitDev2 author+committer provenance. The force-push used an exact lease pinned to superseded remote head cb94c927f0f203ef61b9d01b21c848cd04be821c.

Final review/base reconciliation: main advanced to 2b9fafdea56bcb558bc3b72f0bdf113f04d2c586; the branch rebased cleanly and retained KitDev2 provenance. OpenCodeReview #247 emitted 12 findings. Eleven were stale/duplicate/contract/maintainability dispositions; one real shutdown race was reproduced deterministically and fixed: runtime config hot reload now closes admission before stop/drain can complete. Exact head 96db0e0691e4611f50e789be8b851d32dca61275 has focused lifecycle/config verification 45/45 PASS, core typecheck PASS, Open-SSE typecheck PASS (openSseTypecheckErrors=0), and git diff --check PASS. All review threads were replied to, resolved, and read back resolved before the final hosted oracle.

Acceptance oracle

Protected hosted CI and OpenCodeReview on exact head 96db0e0691e4611f50e789be8b851d32dca61275 are authoritative for final merge. Required Build, Package Boot Smoke, package/E2E/security/quality/docs gates, CodeQL, semgrep, DAST, API typecheck, and OpenCodeReview must be terminal-successful on this exact SHA; all review threads must remain resolved.

Do not delete the source branch until exact-head hosted CI is terminal, source-head → accepted-main provenance is durable, and AP-ISS-0149 KitControl closure evidence is recorded.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

Reviewed bcb4dce..41bb3fa only; earlier commits in this PR were reviewed in a previous run.

Comment thread scripts/dev/run-next.mjs Outdated
Comment thread src/lib/config/hotReload.ts Outdated
Comment thread src/mitm/detection/index.ts Outdated
Comment thread src/mitm/manager.ts
Comment thread scripts/dev/run-next.mjs
Comment thread src/lib/apiBridgeServer.ts
Comment thread src/lib/gracefulShutdown.ts
Comment thread src/lib/jobRegistry/registry.ts
Comment thread src/lib/lifecycle/coordinator.ts
Comment thread src/server/ws/liveServer.ts
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

CI Coverage Report

  • Coverage job: success
  • PR test policy: success

Coverage artifact was not available for this run.

Comment thread scripts/dev/run-next.mjs Outdated
Comment thread scripts/dev/run-next.mjs
Comment thread src/lib/apiBridgeServer.ts
Comment thread src/lib/gracefulShutdown.ts Outdated
Comment thread src/lib/lifecycle/backgroundTasks.ts
Comment thread src/lib/lifecycle/coordinator.ts
Comment thread src/shared/services/modelSyncScheduler.ts
Comment thread src/shared/services/providerLimitsSyncScheduler.ts Outdated
Comment thread src/shared/services/providerLimitsSyncScheduler.ts Outdated
@khanhkit
khanhkit force-pushed the kit/KitDev2/session-1 branch from b91b680 to dd4909b Compare October 11, 2026 11:26
Comment thread open-sse/services/rateLimitManager.ts Outdated
Comment thread open-sse/services/routing/index.ts
Comment thread scripts/dev/run-next.mjs Outdated
Comment thread scripts/dev/run-next.mjs Outdated
Comment thread src/app/api/tools/traffic-inspector/internal/ingest/route.ts
Comment thread src/lib/pricingSync.ts Outdated
Comment thread src/lib/quota/connectionRecovery.ts
Comment thread src/lib/services/registry.ts Outdated
Comment thread src/lib/webhookDispatcher.ts
Comment thread src/server/ws/liveServer.ts
@khanhkit
khanhkit force-pushed the kit/KitDev2/session-1 branch from 692dc95 to 8b0fb71 Compare October 11, 2026 14:43
Comment thread open-sse/services/routing/index.ts
Comment thread scripts/dev/run-next.mjs Outdated
Comment thread src/lib/apiBridgeServer.ts Outdated
Comment thread src/lib/gracefulShutdown.ts
Comment thread src/lib/lifecycle/coordinator.ts
Comment thread src/lib/webhookDispatcher.ts
Comment thread src/server/ws/liveServer.ts
Comment thread src/server/ws/liveServer.ts
Comment thread src/shared/services/providerLimitsSyncScheduler.ts Outdated
@khanhkit
khanhkit force-pushed the kit/KitDev2/session-1 branch 5 times, most recently from 9f6a0f5 to 68b8ffd Compare October 12, 2026 01:54
Comment thread scripts/dev/run-next.mjs Outdated
Comment thread scripts/dev/run-next.mjs Outdated
Comment thread scripts/dev/standalone-server-ws.mjs
@khanhkit
khanhkit force-pushed the kit/KitDev2/session-1 branch from d26b80e to fbb5f74 Compare October 12, 2026 05:37
Comment thread open-sse/services/rateLimitManager.ts Outdated
Comment thread scripts/dev/run-next.mjs
Comment thread scripts/dev/run-standalone.mjs Outdated
Comment thread scripts/dev/standalone-server-ws.mjs
Comment thread scripts/dev/standalone-server-ws.mjs Outdated
Comment thread src/app/api/tools/traffic-inspector/internal/ingest/route.ts Outdated
Comment thread src/lib/credentialHealth/scheduler.ts
Comment thread src/lib/freeProxyProviders/scheduler.ts Outdated
Comment thread src/shared/services/modelSyncScheduler.ts
Comment thread src/shared/services/providerLimitsSyncScheduler.ts
Comment thread open-sse/services/rateLimitManager.ts
Comment thread open-sse/services/rateLimitManager.ts Outdated
Comment thread open-sse/services/routing/index.ts
Comment thread scripts/dev/run-standalone.mjs
Comment thread src/lib/catalog/openrouterProviderStats.ts Outdated
Comment thread src/server/ws/liveServer.ts
Comment thread src/server/ws/liveServer.ts
Comment thread src/server/ws/liveServer.ts
Comment thread src/shared/platform/windowsProcess.ts Outdated
Comment thread src/shared/services/modelSyncScheduler.ts
Comment thread open-sse/services/rateLimitManager.ts
Comment thread open-sse/services/rateLimitManager.ts
Comment thread open-sse/services/rateLimitManager.ts
Comment thread open-sse/services/routing/index.ts Outdated
Comment thread src/shared/platform/windowsProcess.ts
@khanhkit
khanhkit force-pushed the kit/KitDev2/session-1 branch 2 times, most recently from e9e8bdd to 5a2dbe5 Compare October 12, 2026 15:29
Comment thread open-sse/services/rateLimitManager.ts
Comment thread scripts/dev/run-next.mjs
Comment thread scripts/dev/standalone-server-ws.mjs
Comment thread src/lib/catalog/openrouterProviderStats.ts
Comment thread src/lib/contextWindowResolver.ts
Comment thread src/lib/lifecycle/coordinator.ts
Comment thread src/lib/memory/typedDecay.ts
Comment thread src/server/ws/liveServer.ts
Comment thread src/shared/services/providerLimitsSyncScheduler.ts
Comment thread tests/unit/build/run-standalone-shutdown-budget-ap0149.test.mjs
Comment thread open-sse/services/rateLimitManager.ts Outdated
Comment thread open-sse/services/routing/otel.ts
Comment thread scripts/dev/run-standalone.mjs
Comment thread scripts/dev/standalone-server-ws.mjs
Comment thread src/lib/apiBridgeServer.ts
Comment thread src/lib/lifecycle/coordinator.ts
Comment thread src/lib/services/quotaAutoPing.ts Outdated
Comment thread src/lib/webhookDispatcher.ts
Comment thread src/shared/services/providerLimitsSyncScheduler.ts
Comment thread src/shared/services/providerLimitsSyncScheduler.ts
@khanhkit
khanhkit force-pushed the kit/KitDev2/session-1 branch from 4ffba64 to 703685a Compare October 14, 2026 01:24
Comment thread open-sse/services/rateLimitManager.ts Outdated
Comment thread scripts/dev/standalone-server-ws.mjs
Comment thread src/domain/quotaCache.ts Outdated
Comment thread src/lib/arenaEloSync.ts Outdated
Comment thread src/lib/catalog/openrouterProviderStats.ts Outdated
Comment thread src/mitm/manager.ts
Comment thread src/server/ws/liveServer.ts
Comment thread src/shared/services/modelSyncScheduler.ts
Comment thread src/shared/services/providerLimitsSyncScheduler.ts
Comment thread tests/unit/build/run-standalone-shutdown-budget-ap0149.test.mjs
Comment thread bin/cli/commands/serve.mjs Outdated
Comment thread bin/cli/runtime/processSupervisor.mjs
Comment thread open-sse/services/rateLimitManager.ts
Comment thread open-sse/services/routing/index.ts
Comment thread scripts/dev/run-next.mjs
Comment thread src/lib/proxyHealth/scheduler.ts Outdated
Comment thread src/lib/proxyHealth/scheduler.ts
Comment thread src/lib/radar/scheduler.ts Outdated
Comment thread src/lib/webhookDispatcher.ts
Comment thread src/server/ws/liveServer.ts
Comment thread bin/cli/runtime/processSupervisor.mjs Outdated
@khanhkit
khanhkit merged commit cf9a1a0 into main Oct 17, 2026

ghost commented Oct 20, 2026

Copy link
Copy Markdown
Owner Author

AP-ISS-0149 terminal lifecycle evidence (2026-10-20 ICT; KitDev2 Gen106, same durable task run tr_4e0c4d82b2e34664aeeaab794337c2e8):

  • Accepted source PR head: 41bb3fa3ac1dec1eafb48b9ac93fd584ab5d17b5; accepted merge commit on main: cf9a1a04e1d57c32d9b0d74d4a8b68ecfc6f921a (parent 166b4cbb52595005522d2aed5d47282a0942918e). Both source HEAD and accepted-main merge have exact tree 279d48ccfa5d96f6c899883a137606262bc1ca6b.
  • GitHub exact-source-head workflow readback: CI, CodeQL, OpenCodeReview, semgrep, DAST smoke, API Route Typecheck = 6/6 completed/success. Review threads: 156/156 resolved. Source worktree clean; git diff --check RC=0. No new code mutation or repeated full npm test.
  • Machine-private KitControl_Dev_kitcontrol_lifecycle recover with validated=true recovered expired Gen105 to live Gen106 at 19:10:39 ICT (no live attributable worktree processes). The same privileged lifecycle close --outcome DONE succeeded at 19:13 ICT, resulting in durable Episode episode://AgentProxy/AP-ISS-0149/tr_4e0c4d82b2e34664aeeaab794337c2e8, FINAL_HANDOFF state/task DONE, Distillation PENDING, claim removed, lease IDLE, pool AVAILABLE. Files verified on host; no raw runtime edits.
  • Legacy worktree parking is separate and NOT complete. Official park --dry-run returns eligible=false, blockers idle_grace (configured 1800 s) and observability (watcher PID liveness cannot be verified in current execution namespace). Preserve /home/coder/MCP/.worktrees/KitDev2/AP-ISS-0179-local-g93, HEAD 41bb3fa3..., and all Data-is-Gold until normal idle grace and healthy watcher allow official archive → verify → retire. Do not force-park, hand-edit KitControl runtime, manually delete/unlock/reset worktree, or claim a new task from this legacy workspace.

Status: PRODUCT MERGED / KITCONTROL TASK DONE / LEGACY PARK PENDING GUARDS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant