From 01dc41de46a0f55fb803a615b474100089e94983 Mon Sep 17 00:00:00 2001 From: Quentin Date: Sat, 10 Oct 2026 12:33:08 +0200 Subject: [PATCH 1/2] Skip waits as a weft concept, let instance nodes act on the program's own copies, have the install register a built version itself, and let an idle worker go quiet - Skipping a question ends the waiting step skipped (outputs closed, normal propagation); frozen examples record how every wait ended; a skip on a trigger is refused. - StopTagged queues its stop with the plain enqueue again. - StartInfra, StopInfra, TerminateInfra, ActivateTriggers and DeactivateTriggers take an optional instance; inputs can be requiredWhenWired. - Google connections always ask who the account is, so a Gmail-only connection works; GmailSend gains a `from` input. - A build answers at once; the install registers the version itself when its image builds end, in ask order per project; the CLI follows it and tells an unreachable cloud install apart. - An idle worker stops ticking the broker and wakes on work without making a run wait; a metered answer silent for a minute, or on a cancelled run, is cut and its cost booked. - The editor names the triggers that are off; CHANGELOG and the releasing skill keep the changelog per release. --- .agents/skills/releasing/SKILL.md | 10 + .claude/skills/releasing/SKILL.md | 10 + .kilo/skills/releasing/SKILL.md | 10 + CHANGELOG.md | 75 +- Cargo.lock | 343 +---- catalog/google/access/metadata.json | 22 +- catalog/google/gmail.rs | 7 +- catalog/google/gmail_send/metadata.json | 10 +- catalog/google/gmail_send/mod.rs | 82 +- catalog/google/gmail_send/tests.rs | 165 ++- catalog/human/query/metadata.json | 2 +- catalog/human/query/tests.rs | 22 + .../activate_instance_triggers/metadata.json | 46 - .../activate_instance_triggers/mod.rs | 32 - .../activate_instance_triggers/tests.rs | 37 - .../instances/activate_triggers/metadata.json | 97 ++ catalog/instances/activate_triggers/mod.rs | 37 + catalog/instances/activate_triggers/tests.rs | 51 + .../metadata.json | 136 -- .../deactivate_instance_triggers/mod.rs | 34 - .../deactivate_instance_triggers/tests.rs | 30 - .../deactivate_triggers/metadata.json | 178 +++ catalog/instances/deactivate_triggers/mod.rs | 38 + .../instances/deactivate_triggers/tests.rs | 47 + catalog/instances/lifecycle.rs | 33 +- catalog/instances/package.toml | 6 +- catalog/instances/start_infra/metadata.json | 105 ++ .../mod.rs | 18 +- .../tests.rs | 55 +- .../start_instance_infra/metadata.json | 54 - catalog/instances/stop_infra/metadata.json | 184 +++ catalog/instances/stop_infra/mod.rs | 34 + .../tests.rs | 30 +- .../stop_instance_infra/metadata.json | 142 -- catalog/instances/stop_instance_infra/mod.rs | 30 - .../instances/terminate_infra/metadata.json | 185 +++ catalog/instances/terminate_infra/mod.rs | 34 + .../tests.rs | 26 +- .../terminate_instance_infra/metadata.json | 143 -- .../instances/terminate_instance_infra/mod.rs | 30 - crates/weft-access-store/src/flows.rs | 10 +- crates/weft-access-store/tests/store_flows.rs | 21 + crates/weft-broker-client/src/protocol.rs | 14 +- crates/weft-broker/src/handlers.rs | 21 +- crates/weft-broker/src/records.rs | 2 +- crates/weft-broker/tests/door_let_go.rs | 2 +- crates/weft-cli/Cargo.toml | 3 +- crates/weft-cli/src/client.rs | 443 +++++- crates/weft-cli/src/commands/activate.rs | 2 +- crates/weft-cli/src/commands/assets.rs | 8 +- crates/weft-cli/src/commands/cancel_build.rs | 12 +- crates/weft-cli/src/commands/connect.rs | 9 +- crates/weft-cli/src/commands/ensure.rs | 485 +++++-- crates/weft-cli/src/commands/follow.rs | 68 +- crates/weft-cli/src/commands/mod.rs | 4 +- crates/weft-cli/src/commands/versions.rs | 37 +- crates/weft-cli/src/progress.rs | 118 +- crates/weft-compiler/src/enrich.rs | 19 + crates/weft-compiler/src/validate.rs | 2 +- crates/weft-compiler/tests/enrich_tests.rs | 21 + crates/weft-compiler/tests/validate_tests.rs | 48 +- crates/weft-core/src/access/spec.rs | 101 +- crates/weft-core/src/activation.rs | 10 +- crates/weft-core/src/builds.rs | 85 +- crates/weft-core/src/context.rs | 1 + crates/weft-core/src/error.rs | 8 + crates/weft-core/src/exec/ready.rs | 153 +- crates/weft-core/src/exec/skip.rs | 8 +- crates/weft-core/src/live_event.rs | 14 +- crates/weft-core/src/node.rs | 52 + crates/weft-core/src/node_test.rs | 10 + crates/weft-core/src/primitive.rs | 41 +- crates/weft-core/src/project.rs | 15 + crates/weft-core/src/project/selection.rs | 7 +- crates/weft-core/src/projects.rs | 22 +- crates/weft-core/src/run_spec.rs | 132 +- crates/weft-dispatcher/src/api/infra.rs | 10 +- crates/weft-dispatcher/src/api/mod.rs | 44 +- crates/weft-dispatcher/src/api/project.rs | 178 +-- crates/weft-dispatcher/src/api/signal.rs | 38 +- crates/weft-dispatcher/src/app.rs | 3 +- crates/weft-dispatcher/src/authenticator.rs | 33 +- crates/weft-dispatcher/src/build/follow.rs | 115 +- crates/weft-dispatcher/src/build/ledger.rs | 272 ++-- crates/weft-dispatcher/src/build/mod.rs | 340 +++-- crates/weft-dispatcher/src/build/prune.rs | 52 +- crates/weft-dispatcher/src/build/waiting.rs | 759 ++++++++++ crates/weft-dispatcher/src/journal/fake.rs | 16 +- crates/weft-dispatcher/src/journal/mod.rs | 4 +- .../weft-dispatcher/src/journal/postgres.rs | 17 +- crates/weft-dispatcher/src/parked_drain.rs | 2 +- crates/weft-dispatcher/src/project_store.rs | 318 +++-- crates/weft-dispatcher/src/projection.rs | 63 +- crates/weft-dispatcher/src/reaper.rs | 20 +- .../src/task_kinds/fire_signal.rs | 2 +- crates/weft-dispatcher/src/transition.rs | 158 ++- crates/weft-dispatcher/tests/db_build.rs | 1248 ++++++++++++++--- .../weft-dispatcher/tests/db_install_state.rs | 2 +- crates/weft-dispatcher/tests/db_lifecycle.rs | 42 +- .../weft-dispatcher/tests/db_project_ports.rs | 2 +- crates/weft-dispatcher/tests/db_tags.rs | 2 +- crates/weft-dispatcher/tests/db_versions.rs | 4 +- .../weft-dispatcher/tests/tenant_isolation.rs | 10 +- crates/weft-e2e/src/human.rs | 11 +- crates/weft-e2e/src/signal.rs | 6 + crates/weft-e2e/tests/human_form.rs | 28 + crates/weft-e2e/tests/instances.rs | 4 +- crates/weft-engine/src/context.rs | 21 +- crates/weft-engine/src/door/limits.rs | 157 ++- crates/weft-engine/src/door/mod.rs | 132 +- crates/weft-engine/src/door/ticks.rs | 282 ++++ crates/weft-engine/src/execution_driver.rs | 64 +- .../src/execution_driver_tests/rig.rs | 5 +- crates/weft-engine/src/held_waits.rs | 37 +- crates/weft-engine/src/journal_writer.rs | 21 +- crates/weft-engine/src/metering.rs | 869 ++++++++++-- crates/weft-engine/src/socket.rs | 8 +- crates/weft-engine/src/test_rig.rs | 5 +- crates/weft-engine/src/worker.rs | 16 +- crates/weft-infra-supervisor/src/lifecycle.rs | 8 +- crates/weft-journal/src/events.rs | 34 +- crates/weft-journal/src/fold.rs | 103 +- crates/weft-journal/src/lib.rs | 2 +- crates/weft-journal/src/record.rs | 9 +- crates/weft-platform-gcp/src/images.rs | 3 + crates/weft-platform-local/src/images.rs | 127 +- crates/weft-platform-traits/src/images.rs | 50 +- crates/weft-platform-traits/src/lib.rs | 2 +- ...61010T093458_image_build_version_build.sql | 41 + ...010T100248_image_build_build_ask_order.sql | 4 + ...009T114508_parked_fire_skipped_answers.sql | 3 + ...0261010T100248_project_build_ask_order.sql | 7 + crates/weft-task-store/src/db.rs | 17 + crates/weft-task-store/src/parked_fires.rs | 67 +- crates/weft-task-store/src/worker_door.rs | 10 +- docs/src/build/a-person-in-the-loop.md | 8 +- docs/src/connections/declaring-a-service.md | 13 +- docs/src/connections/using-one-in-a-node.md | 17 + docs/src/language/how-a-program-runs.md | 2 + docs/src/nodes/metadata.md | 11 +- docs/src/running/browser-extension.md | 2 +- docs/src/running/cli.md | 2 +- docs/src/running/cloud.md | 12 +- docs/src/running/instances.md | 21 +- extension-browser/package.json | 2 +- .../src/entrypoints/popup/App.svelte | 16 +- .../src/entrypoints/tasks/App.svelte | 17 +- extension-browser/src/lib/api.ts | 9 +- extension-vscode/package.json | 2 +- extension-vscode/src/actionBarState.test.ts | 1 + extension-vscode/src/cli.ts | 18 +- extension-vscode/src/execFollower.ts | 15 +- extension-vscode/src/extension.ts | 26 +- packages/weft-connect/src/core/recipe.test.ts | 15 +- packages/weft-connect/src/core/recipe.ts | 27 +- packages/weft-connect/src/core/wire.ts | 9 + packages/weft-graph/src/protocol.ts | 16 +- packages/weft-graph/src/run-spec.test.ts | 17 + packages/weft-graph/src/run-spec.ts | 29 +- packages/weft-graph/src/status.test.ts | 14 + packages/weft-graph/src/status.ts | 15 + .../lib/components/project/ActionBar.svelte | 26 +- .../lib/components/project/ProjectNode.svelte | 6 +- .../lib/utils/port-context-menu.test.ts | 8 + .../webview/lib/utils/port-context-menu.ts | 13 +- .../src/webview/lib/utils/port-marker.test.ts | 19 +- .../src/webview/lib/utils/port-marker.ts | 11 +- .../src/webview/lib/utils/status.test.ts | 1 + .../src/webview/lib/utils/status.ts | 1 + .../src/webview/lib/verb-gates.test.ts | 13 + .../weft-graph/src/webview/lib/verb-gates.ts | 8 +- .../.claude/skills/weft-catalog/SKILL.md | 2 +- .../.claude/skills/weft-consumers/SKILL.md | 2 +- .../.claude/skills/weft-debugging/SKILL.md | 4 +- .../.claude/skills/weft-editor/SKILL.md | 6 +- .../.claude/skills/weft-instances/SKILL.md | 26 +- .../.claude/skills/weft-members/SKILL.md | 4 +- .../skills/weft-node-authoring/SKILL.md | 27 +- .../.claude/skills/weft-sdp/SKILL.md | 31 +- .../cline/.cline/skills/weft-catalog/SKILL.md | 2 +- .../.cline/skills/weft-consumers/SKILL.md | 2 +- .../.cline/skills/weft-debugging/SKILL.md | 4 +- .../cline/.cline/skills/weft-editor/SKILL.md | 6 +- .../.cline/skills/weft-instances/SKILL.md | 26 +- .../cline/.cline/skills/weft-members/SKILL.md | 4 +- .../skills/weft-node-authoring/SKILL.md | 27 +- tangle/cline/.cline/skills/weft-sdp/SKILL.md | 31 +- .../.agents/skills/weft-catalog/SKILL.md | 2 +- .../.agents/skills/weft-consumers/SKILL.md | 2 +- .../.agents/skills/weft-debugging/SKILL.md | 4 +- .../codex/.agents/skills/weft-editor/SKILL.md | 6 +- .../.agents/skills/weft-instances/SKILL.md | 26 +- .../.agents/skills/weft-members/SKILL.md | 4 +- .../skills/weft-node-authoring/SKILL.md | 27 +- tangle/codex/.agents/skills/weft-sdp/SKILL.md | 31 +- .../.cursor/skills/weft-catalog/SKILL.md | 2 +- .../.cursor/skills/weft-consumers/SKILL.md | 2 +- .../.cursor/skills/weft-debugging/SKILL.md | 4 +- .../.cursor/skills/weft-editor/SKILL.md | 6 +- .../.cursor/skills/weft-instances/SKILL.md | 26 +- .../.cursor/skills/weft-members/SKILL.md | 4 +- .../skills/weft-node-authoring/SKILL.md | 27 +- .../cursor/.cursor/skills/weft-sdp/SKILL.md | 31 +- .../.windsurf/skills/weft-catalog/SKILL.md | 2 +- .../.windsurf/skills/weft-consumers/SKILL.md | 2 +- .../.windsurf/skills/weft-debugging/SKILL.md | 4 +- .../.windsurf/skills/weft-editor/SKILL.md | 6 +- .../.windsurf/skills/weft-instances/SKILL.md | 26 +- .../.windsurf/skills/weft-members/SKILL.md | 4 +- .../skills/weft-node-authoring/SKILL.md | 27 +- .../.windsurf/skills/weft-sdp/SKILL.md | 31 +- .../.agents/skills/weft-catalog/SKILL.md | 2 +- .../.agents/skills/weft-consumers/SKILL.md | 2 +- .../.agents/skills/weft-debugging/SKILL.md | 4 +- .../.agents/skills/weft-editor/SKILL.md | 6 +- .../.agents/skills/weft-instances/SKILL.md | 26 +- .../.agents/skills/weft-members/SKILL.md | 4 +- .../skills/weft-node-authoring/SKILL.md | 27 +- .../fallback/.agents/skills/weft-sdp/SKILL.md | 31 +- .../.gemini/skills/weft-catalog/SKILL.md | 2 +- .../.gemini/skills/weft-consumers/SKILL.md | 2 +- .../.gemini/skills/weft-debugging/SKILL.md | 4 +- .../.gemini/skills/weft-editor/SKILL.md | 6 +- .../.gemini/skills/weft-instances/SKILL.md | 26 +- .../.gemini/skills/weft-members/SKILL.md | 4 +- .../skills/weft-node-authoring/SKILL.md | 27 +- .../.gemini/skills/weft-sdp/SKILL.md | 31 +- .../.github/skills/weft-catalog/SKILL.md | 2 +- .../.github/skills/weft-consumers/SKILL.md | 2 +- .../.github/skills/weft-debugging/SKILL.md | 4 +- .../.github/skills/weft-editor/SKILL.md | 6 +- .../.github/skills/weft-instances/SKILL.md | 26 +- .../.github/skills/weft-members/SKILL.md | 4 +- .../skills/weft-node-authoring/SKILL.md | 27 +- .../.github/skills/weft-sdp/SKILL.md | 31 +- .../junie/.junie/skills/weft-catalog/SKILL.md | 2 +- .../.junie/skills/weft-consumers/SKILL.md | 2 +- .../.junie/skills/weft-debugging/SKILL.md | 4 +- .../junie/.junie/skills/weft-editor/SKILL.md | 6 +- .../.junie/skills/weft-instances/SKILL.md | 26 +- .../junie/.junie/skills/weft-members/SKILL.md | 4 +- .../skills/weft-node-authoring/SKILL.md | 27 +- tangle/junie/.junie/skills/weft-sdp/SKILL.md | 31 +- .../.kilo/skills/weft-catalog/SKILL.md | 2 +- .../.kilo/skills/weft-consumers/SKILL.md | 2 +- .../.kilo/skills/weft-debugging/SKILL.md | 4 +- .../.kilo/skills/weft-editor/SKILL.md | 6 +- .../.kilo/skills/weft-instances/SKILL.md | 26 +- .../.kilo/skills/weft-members/SKILL.md | 4 +- .../.kilo/skills/weft-node-authoring/SKILL.md | 27 +- .../kilo-code/.kilo/skills/weft-sdp/SKILL.md | 31 +- .../.opencode/skills/weft-catalog/SKILL.md | 2 +- .../.opencode/skills/weft-consumers/SKILL.md | 2 +- .../.opencode/skills/weft-debugging/SKILL.md | 4 +- .../.opencode/skills/weft-editor/SKILL.md | 6 +- .../.opencode/skills/weft-instances/SKILL.md | 26 +- .../.opencode/skills/weft-members/SKILL.md | 4 +- .../skills/weft-node-authoring/SKILL.md | 27 +- .../.opencode/skills/weft-sdp/SKILL.md | 31 +- 259 files changed, 9083 insertions(+), 3026 deletions(-) delete mode 100644 catalog/instances/activate_instance_triggers/metadata.json delete mode 100644 catalog/instances/activate_instance_triggers/mod.rs delete mode 100644 catalog/instances/activate_instance_triggers/tests.rs create mode 100644 catalog/instances/activate_triggers/metadata.json create mode 100644 catalog/instances/activate_triggers/mod.rs create mode 100644 catalog/instances/activate_triggers/tests.rs delete mode 100644 catalog/instances/deactivate_instance_triggers/metadata.json delete mode 100644 catalog/instances/deactivate_instance_triggers/mod.rs delete mode 100644 catalog/instances/deactivate_instance_triggers/tests.rs create mode 100644 catalog/instances/deactivate_triggers/metadata.json create mode 100644 catalog/instances/deactivate_triggers/mod.rs create mode 100644 catalog/instances/deactivate_triggers/tests.rs create mode 100644 catalog/instances/start_infra/metadata.json rename catalog/instances/{start_instance_infra => start_infra}/mod.rs (59%) rename catalog/instances/{start_instance_infra => start_infra}/tests.rs (67%) delete mode 100644 catalog/instances/start_instance_infra/metadata.json create mode 100644 catalog/instances/stop_infra/metadata.json create mode 100644 catalog/instances/stop_infra/mod.rs rename catalog/instances/{stop_instance_infra => stop_infra}/tests.rs (67%) delete mode 100644 catalog/instances/stop_instance_infra/metadata.json delete mode 100644 catalog/instances/stop_instance_infra/mod.rs create mode 100644 catalog/instances/terminate_infra/metadata.json create mode 100644 catalog/instances/terminate_infra/mod.rs rename catalog/instances/{terminate_instance_infra => terminate_infra}/tests.rs (51%) delete mode 100644 catalog/instances/terminate_instance_infra/metadata.json delete mode 100644 catalog/instances/terminate_instance_infra/mod.rs create mode 100644 crates/weft-dispatcher/src/build/waiting.rs create mode 100644 crates/weft-engine/src/door/ticks.rs create mode 100644 crates/weft-task-store/migrations/image_build/20261010T093458_image_build_version_build.sql create mode 100644 crates/weft-task-store/migrations/image_build/20261010T100248_image_build_build_ask_order.sql create mode 100644 crates/weft-task-store/migrations/parked_fire/20261009T114508_parked_fire_skipped_answers.sql create mode 100644 crates/weft-task-store/migrations/project/20261010T100248_project_build_ask_order.sql diff --git a/.agents/skills/releasing/SKILL.md b/.agents/skills/releasing/SKILL.md index 0b8b8762..de7c917d 100644 --- a/.agents/skills/releasing/SKILL.md +++ b/.agents/skills/releasing/SKILL.md @@ -22,6 +22,16 @@ into `main` through a PR, and the merge is what publishes. to the stores. The `extension versions bumped` CI check refuses a PR that forgot; `scripts/check-extension-bump.sh origin/main` is the same check locally. + In the same PR, write what changed into `CHANGELOG.md`, for the + people who update their weft: what they will notice, what they have + to do (a `./setup.sh` that asks, a command to run again, a setting + that moved), and anything breaking first. Each release gets its own + section, newest first, headed with its date and its version, the VS + Code extension's version the PR bumped to (`## 2026-10-09 (0.2.267)`); + a release that bumps no extension names its merge's short commit + instead. Work not merged yet stays under `## Unreleased`, and the PR + that ships it renames that heading. Write it plainly, for a person + reading it before they run `./setup.sh`, never as a list of commits. 2. `gh pr create --base main --head `. 3. Never `--auto`. Wait for every check to finish (`gh pr checks --watch`), then read what the PR collected besides the checks before diff --git a/.claude/skills/releasing/SKILL.md b/.claude/skills/releasing/SKILL.md index 0b8b8762..de7c917d 100644 --- a/.claude/skills/releasing/SKILL.md +++ b/.claude/skills/releasing/SKILL.md @@ -22,6 +22,16 @@ into `main` through a PR, and the merge is what publishes. to the stores. The `extension versions bumped` CI check refuses a PR that forgot; `scripts/check-extension-bump.sh origin/main` is the same check locally. + In the same PR, write what changed into `CHANGELOG.md`, for the + people who update their weft: what they will notice, what they have + to do (a `./setup.sh` that asks, a command to run again, a setting + that moved), and anything breaking first. Each release gets its own + section, newest first, headed with its date and its version, the VS + Code extension's version the PR bumped to (`## 2026-10-09 (0.2.267)`); + a release that bumps no extension names its merge's short commit + instead. Work not merged yet stays under `## Unreleased`, and the PR + that ships it renames that heading. Write it plainly, for a person + reading it before they run `./setup.sh`, never as a list of commits. 2. `gh pr create --base main --head `. 3. Never `--auto`. Wait for every check to finish (`gh pr checks --watch`), then read what the PR collected besides the checks before diff --git a/.kilo/skills/releasing/SKILL.md b/.kilo/skills/releasing/SKILL.md index 0b8b8762..de7c917d 100644 --- a/.kilo/skills/releasing/SKILL.md +++ b/.kilo/skills/releasing/SKILL.md @@ -22,6 +22,16 @@ into `main` through a PR, and the merge is what publishes. to the stores. The `extension versions bumped` CI check refuses a PR that forgot; `scripts/check-extension-bump.sh origin/main` is the same check locally. + In the same PR, write what changed into `CHANGELOG.md`, for the + people who update their weft: what they will notice, what they have + to do (a `./setup.sh` that asks, a command to run again, a setting + that moved), and anything breaking first. Each release gets its own + section, newest first, headed with its date and its version, the VS + Code extension's version the PR bumped to (`## 2026-10-09 (0.2.267)`); + a release that bumps no extension names its merge's short commit + instead. Work not merged yet stays under `## Unreleased`, and the PR + that ships it renames that heading. Write it plainly, for a person + reading it before they run `./setup.sh`, never as a list of commits. 2. `gh pr create --base main --head `. 3. Never `--auto`. Wait for every check to finish (`gh pr checks --watch`), then read what the PR collected besides the checks before diff --git a/CHANGELOG.md b/CHANGELOG.md index 6681fbe2..13adc862 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,80 @@ What changed between versions of weft that you need to know about when you update. To update, pull and run `./setup.sh` again. -## Unreleased: weft no longer needs Kubernetes +## Unreleased + +**Five instance nodes lost "Instance" from their names.** + +`StartInstanceInfra`, `StopInstanceInfra`, `TerminateInstanceInfra`, +`ActivateInstanceTriggers` and `DeactivateInstanceTriggers` are now +`StartInfra`, `StopInfra`, `TerminateInfra`, `ActivateTriggers` and +`DeactivateTriggers`. Rename them in your `.weft` files. Their `instance` +input is now optional: if you leave it unwired, the node acts on the +program's own copies, the ones every instance shares. + +**GmailSend can send from another address.** + +The new optional `from` input sends from one of the account's "Send mail as" +addresses set up in Gmail; any other address fails naming the ones that work. +To read that list, the Google connection needs "Read mail" or "Organize mail" +ticked. Replies are unchanged: they need one of those two or "Read mail +headers". + +**A program built before this version needs one rebuild.** + +How a waiting step receives its answer changed, so a worker built before this +version fails when a waiting step gets its answer. Resync each program once: +`weft resync` if its triggers are on, `weft build` otherwise. Either one +builds the worker again on this version of weft and moves the program to it. + +**Skipping a question ends the step that asked it.** + +When a person skips a waiting question (Skip in the browser extension, or +`POST /signal/{token}/skip`), the step that asked now ends skipped: its +outputs close and the steps reading them skip too. It used to answer the +question with null, so if your program reads a null answer as "skipped", it +now sees the step skipped instead. Skipping a trigger is refused with a `409`; +it used to fire the trigger. + +**A Google connection can hold Gmail alone.** + +The connect check now asks Google who the account is, through two permissions +every connection asks for ("Know who you are" and "See your email address"), +instead of reading Drive. So a connection with only Gmail, or only Calendar, +ticked now passes. If you run your own shared Google app, its `covers` does +not need to list those two. + +**Node authors: inputs can be required only when wired.** + +`requiredWhenWired: true` on an input means: left unwired, the node runs +without it; wired, a wire that delivers nothing skips the node. It is for an +input whose absence means something, like the instance nodes' `instance`. + +**Builds no longer hold one request open.** + +`weft build`, `run`, `activate`, `resync` and `infra start` used to keep one +request open to the install for the whole build. The install now answers as +soon as the builds start, and the CLI follows each one to its end. If a cloud +install does not answer, the error names that install instead of telling you +to start the local daemon. Update the VS Code extension too: it needs this +version. + +**A route's limits can overshoot briefly after a quiet spell.** + +When a worker copy wakes after a quiet spell, its first calls are decided on +what it heard before it went quiet, until weft's broker answers it: under a +second normally, a few seconds when the broker was asleep too. During that +window, copies waking at the same moment can each let in up to the full +`callsAtOnce`. We chose this so the first call after a pause never waits for +the broker to start. + +**The editor's activate button names the triggers that are off.** + +When some triggers are off while the others run, the second button now says +which ones ("Reactivate a, b" or "Reactivate a, b and 3 more") instead of +"Reactivate the rest". + +## 2026-10-09 (extension 0.2.266): weft no longer needs Kubernetes **Breaking: your run history does not come along.** diff --git a/Cargo.lock b/Cargo.lock index 5d427a06..bbfaca55 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -430,17 +430,17 @@ dependencies = [ "aws-smithy-async", "aws-smithy-runtime-api", "aws-smithy-types", - "h2 0.4.15", + "h2", "http 1.4.0", - "hyper 1.9.0", - "hyper-rustls 0.27.9", + "hyper", + "hyper-rustls", "hyper-util", "pin-project-lite", - "rustls 0.23.40", - "rustls-native-certs 0.8.3", + "rustls", + "rustls-native-certs", "rustls-pki-types", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tower", "tracing", ] @@ -598,14 +598,14 @@ checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" dependencies = [ "axum-core", "axum-macros", - "base64 0.22.1", + "base64", "bytes", "form_urlencoded", "futures-util", "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.9.0", + "hyper", "hyper-util", "itoa", "matchit", @@ -678,12 +678,6 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4c7f02d4ea65f2c1853089ffd8d2787bdbc63de2f0d29dedbcf8ccdfa0ccd4cf" -[[package]] -name = "base64" -version = "0.21.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d297deb1925b89f2ccc13d7635fa0714f12c87adce1c75356b39ca9b7178567" - [[package]] name = "base64" version = "0.22.1" @@ -927,16 +921,6 @@ version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" -[[package]] -name = "core-foundation" -version = "0.9.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" -dependencies = [ - "core-foundation-sys", - "libc", -] - [[package]] name = "core-foundation" version = "0.10.1" @@ -1345,22 +1329,6 @@ dependencies = [ "pin-project-lite", ] -[[package]] -name = "eventsource-client" -version = "0.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43ddc25e1ad2cc0106d5e2d967397b4fb2068a66677ee9b0eea4600e5cfe8fb4" -dependencies = [ - "futures", - "hyper 0.14.32", - "hyper-rustls 0.24.2", - "hyper-timeout", - "log", - "pin-project", - "rand 0.8.6", - "tokio", -] - [[package]] name = "eventsource-stream" version = "0.2.3" @@ -1666,25 +1634,6 @@ dependencies = [ "subtle", ] -[[package]] -name = "h2" -version = "0.3.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0beca50380b1fc32983fc1cb4587bfa4bb9e78fc259aad4a0032d2080309222d" -dependencies = [ - "bytes", - "fnv", - "futures-core", - "futures-sink", - "futures-util", - "http 0.2.12", - "indexmap", - "slab", - "tokio", - "tokio-util", - "tracing", -] - [[package]] name = "h2" version = "0.4.15" @@ -1877,30 +1826,6 @@ dependencies = [ "typenum", ] -[[package]] -name = "hyper" -version = "0.14.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41dfc780fdec9373c01bae43289ea34c972e40ee3c9f6b3c8801a35f35586ce7" -dependencies = [ - "bytes", - "futures-channel", - "futures-core", - "futures-util", - "h2 0.3.27", - "http 0.2.12", - "http-body 0.4.6", - "httparse", - "httpdate", - "itoa", - "pin-project-lite", - "socket2 0.5.10", - "tokio", - "tower-service", - "tracing", - "want", -] - [[package]] name = "hyper" version = "1.9.0" @@ -1911,7 +1836,7 @@ dependencies = [ "bytes", "futures-channel", "futures-core", - "h2 0.4.15", + "h2", "http 1.4.0", "http-body 1.0.1", "httparse", @@ -1923,22 +1848,6 @@ dependencies = [ "want", ] -[[package]] -name = "hyper-rustls" -version = "0.24.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec3efd23720e2049821a693cbc7e65ea87c72f1c58ff2f9522ff332b1491e590" -dependencies = [ - "futures-util", - "http 0.2.12", - "hyper 0.14.32", - "log", - "rustls 0.21.12", - "rustls-native-certs 0.6.3", - "tokio", - "tokio-rustls 0.24.1", -] - [[package]] name = "hyper-rustls" version = "0.27.9" @@ -1946,46 +1855,34 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" dependencies = [ "http 1.4.0", - "hyper 1.9.0", + "hyper", "hyper-util", - "rustls 0.23.40", - "rustls-native-certs 0.8.3", + "rustls", + "rustls-native-certs", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tower-service", "webpki-roots 1.0.7", ] -[[package]] -name = "hyper-timeout" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bbb958482e8c7be4bc3cf272a766a2b0bf1a6755e7a6ae777f017a31d11b13b1" -dependencies = [ - "hyper 0.14.32", - "pin-project-lite", - "tokio", - "tokio-io-timeout", -] - [[package]] name = "hyper-util" version = "0.1.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" dependencies = [ - "base64 0.22.1", + "base64", "bytes", "futures-channel", "futures-util", "http 1.4.0", "http-body 1.0.1", - "hyper 1.9.0", + "hyper", "ipnet", "libc", "percent-encoding", "pin-project-lite", - "socket2 0.6.3", + "socket2", "tokio", "tower-service", "tracing", @@ -2267,7 +2164,7 @@ version = "10.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eba32bfb4ffdeaca3e34431072faf01745c9b26d25504aa7a6cf5684334fc4fc" dependencies = [ - "base64 0.22.1", + "base64", "ed25519-dalek", "getrandom 0.2.17", "hmac 0.12.1", @@ -2447,7 +2344,7 @@ version = "0.5.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a56634e7ed4b9464dfa2e90e367def6d566813c3b4ea9e009b9b992d2758428" dependencies = [ - "base64 0.22.1", + "base64", "dotenvy", "eventsource-stream", "futures", @@ -2639,12 +2536,6 @@ dependencies = [ "libc", ] -[[package]] -name = "openssl-probe" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e" - [[package]] name = "openssl-probe" version = "0.2.1" @@ -2716,7 +2607,7 @@ version = "3.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d30c53c26bc5b31a98cd02d20f25a7c8567146caf63ed593a9d87b2775291be" dependencies = [ - "base64 0.22.1", + "base64", "serde_core", ] @@ -2753,26 +2644,6 @@ dependencies = [ "siphasher", ] -[[package]] -name = "pin-project" -version = "1.1.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f1749c7ed4bcaf4c3d0a3efc28538844fb29bcdd7d2b67b2be7e20ba861ff517" -dependencies = [ - "pin-project-internal", -] - -[[package]] -name = "pin-project-internal" -version = "1.1.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d9b20ed30f105399776b9c883e68e536ef602a16ae6f596d2c473591d6ad64c6" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - [[package]] name = "pin-project-lite" version = "0.2.17" @@ -2944,8 +2815,8 @@ dependencies = [ "quinn-proto", "quinn-udp", "rustc-hash 2.1.2", - "rustls 0.23.40", - "socket2 0.6.3", + "rustls", + "socket2", "thiserror 2.0.18", "tokio", "tracing", @@ -2965,7 +2836,7 @@ dependencies = [ "rand_pcg", "ring", "rustc-hash 2.1.2", - "rustls 0.23.40", + "rustls", "rustls-pki-types", "slab", "thiserror 2.0.18", @@ -2983,7 +2854,7 @@ dependencies = [ "cfg_aliases", "libc", "once_cell", - "socket2 0.6.3", + "socket2", "tracing", "windows-sys 0.60.2", ] @@ -3153,17 +3024,17 @@ version = "0.12.28" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" dependencies = [ - "base64 0.22.1", + "base64", "bytes", "encoding_rs", "futures-core", "futures-util", - "h2 0.4.15", + "h2", "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.9.0", - "hyper-rustls 0.27.9", + "hyper", + "hyper-rustls", "hyper-util", "js-sys", "log", @@ -3171,14 +3042,14 @@ dependencies = [ "percent-encoding", "pin-project-lite", "quinn", - "rustls 0.23.40", + "rustls", "rustls-pki-types", "serde", "serde_json", "serde_urlencoded", "sync_wrapper", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tokio-util", "tower", "tower-http", @@ -3332,18 +3203,6 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "rustls" -version = "0.21.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f56a14d1f48b391359b22f731fd4bd7e43c97f3c50eee276f3aa09c94784d3e" -dependencies = [ - "log", - "ring", - "rustls-webpki 0.101.7", - "sct", -] - [[package]] name = "rustls" version = "0.23.40" @@ -3353,42 +3212,21 @@ dependencies = [ "once_cell", "ring", "rustls-pki-types", - "rustls-webpki 0.103.13", + "rustls-webpki", "subtle", "zeroize", ] -[[package]] -name = "rustls-native-certs" -version = "0.6.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9aace74cb666635c918e9c12bc0d348266037aa8eb599b5cba565709a8dff00" -dependencies = [ - "openssl-probe 0.1.6", - "rustls-pemfile", - "schannel", - "security-framework 2.11.1", -] - [[package]] name = "rustls-native-certs" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63" dependencies = [ - "openssl-probe 0.2.1", + "openssl-probe", "rustls-pki-types", "schannel", - "security-framework 3.7.0", -] - -[[package]] -name = "rustls-pemfile" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c74cae0a4cf6ccbbf5f359f08efdf8ee7e1dc532573bf0db71968cb56b1448c" -dependencies = [ - "base64 0.21.7", + "security-framework", ] [[package]] @@ -3401,16 +3239,6 @@ dependencies = [ "zeroize", ] -[[package]] -name = "rustls-webpki" -version = "0.101.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b6275d1ee7a1cd780b64aca7726599a1dbc893b1e64144529e55c3c2f745765" -dependencies = [ - "ring", - "untrusted", -] - [[package]] name = "rustls-webpki" version = "0.103.13" @@ -3449,16 +3277,6 @@ version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" -[[package]] -name = "sct" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da046153aa2352493d6cb7da4b6e5c0c057d8a1d0a9aa8560baffdd945acd414" -dependencies = [ - "ring", - "untrusted", -] - [[package]] name = "sec1" version = "0.7.3" @@ -3483,19 +3301,6 @@ dependencies = [ "zeroize", ] -[[package]] -name = "security-framework" -version = "2.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "897b2245f0b511c87893af39b033e5ca9cce68824c4d7e7630b5a1d339658d02" -dependencies = [ - "bitflags 2.11.1", - "core-foundation 0.9.4", - "core-foundation-sys", - "libc", - "security-framework-sys", -] - [[package]] name = "security-framework" version = "3.7.0" @@ -3503,7 +3308,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" dependencies = [ "bitflags 2.11.1", - "core-foundation 0.10.1", + "core-foundation", "core-foundation-sys", "libc", "security-framework-sys", @@ -3737,16 +3542,6 @@ dependencies = [ "serde", ] -[[package]] -name = "socket2" -version = "0.5.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e22376abed350d73dd1cd119b57ffccad95b4e585a7cda43e286245ce23c0678" -dependencies = [ - "libc", - "windows-sys 0.52.0", -] - [[package]] name = "socket2" version = "0.6.3" @@ -3804,7 +3599,7 @@ version = "0.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ee6798b1838b6a0f69c007c133b8df5866302197e404e8b6ee8ed3e3a5e68dc6" dependencies = [ - "base64 0.22.1", + "base64", "bytes", "chrono", "crc", @@ -3822,7 +3617,7 @@ dependencies = [ "memchr", "once_cell", "percent-encoding", - "rustls 0.23.40", + "rustls", "serde", "serde_json", "sha2 0.10.9", @@ -3881,7 +3676,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "aa003f0038df784eb8fecbbac13affe3da23b45194bd57dba231c8f48199c526" dependencies = [ "atoi", - "base64 0.22.1", + "base64", "bitflags 2.11.1", "byteorder", "bytes", @@ -3925,7 +3720,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "db58fcd5a53cf07c184b154801ff91347e4c30d17a3562a635ff028ad5deda46" dependencies = [ "atoi", - "base64 0.22.1", + "base64", "bitflags 2.11.1", "byteorder", "chrono", @@ -4259,21 +4054,11 @@ dependencies = [ "parking_lot", "pin-project-lite", "signal-hook-registry", - "socket2 0.6.3", + "socket2", "tokio-macros", "windows-sys 0.61.2", ] -[[package]] -name = "tokio-io-timeout" -version = "1.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0bd86198d9ee903fedd2f9a2e72014287c0d9167e4ae43b5853007205dda1b76" -dependencies = [ - "pin-project-lite", - "tokio", -] - [[package]] name = "tokio-macros" version = "2.7.0" @@ -4285,23 +4070,13 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "tokio-rustls" -version = "0.24.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c28327cf380ac148141087fbfb9de9d7bd4e84ab5d2c28fbc911d753de8a7081" -dependencies = [ - "rustls 0.21.12", - "tokio", -] - [[package]] name = "tokio-rustls" version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" dependencies = [ - "rustls 0.23.40", + "rustls", "tokio", ] @@ -4325,11 +4100,11 @@ checksum = "edc5f74e248dc973e0dbb7b74c7e0d6fcc301c694ff50049504004ef4d0cdcd9" dependencies = [ "futures-util", "log", - "rustls 0.23.40", - "rustls-native-certs 0.8.3", + "rustls", + "rustls-native-certs", "rustls-pki-types", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tungstenite 0.24.0", ] @@ -4570,7 +4345,7 @@ dependencies = [ "httparse", "log", "rand 0.8.6", - "rustls 0.23.40", + "rustls", "rustls-pki-types", "sha1 0.10.6", "thiserror 1.0.69", @@ -4909,7 +4684,7 @@ dependencies = [ "aes-gcm", "anyhow", "axum", - "base64 0.22.1", + "base64", "chrono", "jsonwebtoken", "rand 0.8.6", @@ -4948,7 +4723,7 @@ dependencies = [ "anyhow", "async-trait", "axum", - "base64 0.22.1", + "base64", "bytes", "chrono", "futures", @@ -5026,12 +4801,12 @@ dependencies = [ "anyhow", "async-trait", "axum", - "base64 0.22.1", + "base64", "chrono", "clap", "clap_complete", "dotenvy", - "eventsource-client", + "eventsource-stream", "futures", "open", "reqwest", @@ -5083,7 +4858,7 @@ version = "0.2.0" dependencies = [ "anyhow", "async-trait", - "base64 0.22.1", + "base64", "bytes", "chrono", "chrono-tz", @@ -5099,7 +4874,7 @@ dependencies = [ "regex", "reqwest", "reqwest-middleware", - "rustls-native-certs 0.8.3", + "rustls-native-certs", "serde", "serde_json", "sha1 0.10.6", @@ -5107,7 +4882,7 @@ dependencies = [ "similar", "thiserror 2.0.18", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tracing", "url", "uuid", @@ -5121,7 +4896,7 @@ dependencies = [ "anyhow", "async-trait", "axum", - "base64 0.22.1", + "base64", "bytes", "chrono", "futures", @@ -5157,7 +4932,7 @@ version = "0.2.0" dependencies = [ "anyhow", "axum", - "base64 0.22.1", + "base64", "chrono", "dotenvy", "futures", @@ -5186,7 +4961,7 @@ dependencies = [ "anyhow", "async-trait", "axum", - "base64 0.22.1", + "base64", "bytes", "clap", "futures", @@ -5199,7 +4974,7 @@ dependencies = [ "reqwest-middleware", "serde", "serde_json", - "socket2 0.6.3", + "socket2", "tempfile", "thiserror 2.0.18", "tikv-jemallocator", @@ -5248,7 +5023,7 @@ version = "0.2.0" dependencies = [ "anyhow", "async-trait", - "base64 0.22.1", + "base64", "chrono", "serde", "serde_json", @@ -5269,7 +5044,7 @@ dependencies = [ "anyhow", "async-trait", "axum", - "base64 0.22.1", + "base64", "chrono", "chrono-tz", "cron", @@ -5313,7 +5088,7 @@ version = "0.2.0" dependencies = [ "anyhow", "async-trait", - "base64 0.22.1", + "base64", "bytes", "chrono", "flate2", @@ -5387,7 +5162,7 @@ version = "0.2.0" dependencies = [ "anyhow", "async-trait", - "base64 0.22.1", + "base64", "http 1.4.0", "inventory", "reqwest", @@ -5406,7 +5181,7 @@ version = "0.2.0" dependencies = [ "anyhow", "axum", - "base64 0.22.1", + "base64", "clap", "futures", "hex", diff --git a/catalog/google/access/metadata.json b/catalog/google/access/metadata.json index ea15691a..8254695d 100644 --- a/catalog/google/access/metadata.json +++ b/catalog/google/access/metadata.json @@ -35,6 +35,18 @@ } }, "permissions": [ + { + "id": "openid", + "label": "Know who you are", + "description": "Confirm which Google account this is (asked on every connection, whatever else is picked).", + "always": true + }, + { + "id": "https://www.googleapis.com/auth/userinfo.email", + "label": "See your email address", + "description": "Read the account's email address, which names the connection (asked on every connection, whatever else is picked).", + "always": true + }, { "id": "https://www.googleapis.com/auth/drive.file", "label": "Files you pick", @@ -59,17 +71,17 @@ { "id": "https://www.googleapis.com/auth/gmail.metadata", "label": "Read mail headers", - "description": "Read subjects, senders and message ids, never message bodies (this is what threading a reply needs)." + "description": "Read subjects, senders and message ids, never message bodies (enough to thread a reply)." }, { "id": "https://www.googleapis.com/auth/gmail.readonly", "label": "Read mail", - "description": "Read the account's email (needed by the new-email trigger and message reads)." + "description": "Read the account's email (needed by the new-email trigger and message reads; sending from another address needs this or Organize mail)." }, { "id": "https://www.googleapis.com/auth/gmail.modify", "label": "Organize mail", - "description": "Add and remove labels on the account's email." + "description": "Read the account's email and add and remove its labels." }, { "id": "https://www.googleapis.com/auth/calendar.events", @@ -95,11 +107,11 @@ } ], "test": { - "url": "https://www.googleapis.com/drive/v3/about?fields=user", + "url": "https://openidconnect.googleapis.com/v1/userinfo", "captures": [ { "name": "user_email", - "path": "user.emailAddress" + "path": "email" } ] }, diff --git a/catalog/google/gmail.rs b/catalog/google/gmail.rs index 2ac032e7..bff43fdc 100644 --- a/catalog/google/gmail.rs +++ b/catalog/google/gmail.rs @@ -44,8 +44,9 @@ pub struct OutAttachment { /// What one outgoing message carries, before assembly. pub struct OutMessage<'a> { - /// The account's own address: Gmail sends as the signed-in - /// account, and an RFC 5322 message must name its sender. + /// The sender: the account's own address or one of its Send mail + /// as aliases (Gmail rewrites any other From to the account's own), + /// and an RFC 5322 message must name it. pub from: &'a str, pub to: &'a [String], pub cc: &'a [String], @@ -77,7 +78,7 @@ pub fn build_mime(m: &OutMessage<'_>) -> WeftResult> { } let mut builder = lettre::Message::builder() - .from(mailbox(m.from, "the account's own address")?) + .from(mailbox(m.from, "the From address")?) .subject(m.subject) .keep_bcc(); for addr in m.to { diff --git a/catalog/google/gmail_send/metadata.json b/catalog/google/gmail_send/metadata.json index 1c967982..6ddab46d 100644 --- a/catalog/google/gmail_send/metadata.json +++ b/catalog/google/gmail_send/metadata.json @@ -17,7 +17,13 @@ "requiresScopes": [ "https://www.googleapis.com/auth/gmail.send" ], - "description": "Needs send-mail; a reply also needs read-headers to thread onto the original." + "description": "Needs Send mail. Sending from another address also needs Read mail or Organize mail; replying to a message needs one of those or Read mail headers. Without them the node fails before sending, naming what to tick." + }, + { + "name": "from", + "type": "String", + "label": "From", + "description": "Send from this address instead of the account's own; it must be one of the account's Send mail as addresses set up in Gmail, and anything else fails naming the allowed ones. An address other than the account's own needs Read mail or Organize mail on the connection, to read that list." }, { "name": "to", @@ -56,7 +62,7 @@ "name": "replyTo", "type": "String", "label": "Reply to message id", - "description": "Gmail message id to reply to in its thread (as a Gmail trigger or Get emits)." + "description": "Gmail message id to reply to in its thread (as a Gmail trigger or Get emits). Needs Read mail, Organize mail or Read mail headers on the connection, to read the original's headers." }, { "name": "attachments", diff --git a/catalog/google/gmail_send/mod.rs b/catalog/google/gmail_send/mod.rs index 245abab1..79203144 100644 --- a/catalog/google/gmail_send/mod.rs +++ b/catalog/google/gmail_send/mod.rs @@ -1,7 +1,12 @@ -//! GmailSend: send mail as the connected account, with attachments -//! and reply-in-thread. Replying to a message id fetches its -//! Message-ID + thread and stamps In-Reply-To / References, so mail -//! clients thread it correctly. +//! GmailSend: send mail as the connected account (or one of its Send +//! mail as aliases), with attachments and reply-in-thread. Replying to +//! a message id fetches its Message-ID + thread and stamps In-Reply-To +//! / References, so mail clients thread it correctly. Both of those +//! read the mailbox, which Send mail alone cannot. Several Gmail +//! permissions can (Read mail, Organize mail, and Read mail headers for +//! a reply), and a requirement is one exact name, so the node asks for +//! none of them up front: Gmail refuses the read before anything is +//! sent, and the attempt's name says which permissions work. use async_trait::async_trait; use serde_json::{json, Value}; @@ -34,7 +39,8 @@ impl Node for GmailSendNode { let subject: String = ctx.inputs.get_or("subject", String::new())?; let text: Option = ctx.inputs.opt("text")?; let html: Option = ctx.inputs.opt("html")?; - let reply_to: Option = ctx.inputs.opt("replyTo")?; + let reply_to = ctx.inputs.opt::("replyTo")?.filter(|r| !r.trim().is_empty()); + let alias = ctx.inputs.opt::("from")?.map(|a| a.trim().to_string()).filter(|a| !a.is_empty()); let handles: Vec = ctx.inputs.list("attachments")?; if to.is_empty() && cc.is_empty() && bcc.is_empty() { @@ -44,25 +50,35 @@ impl Node for GmailSendNode { } // Gmail sends as the signed-in account; its address is the // connection's recorded identity, and the message must name it. - let Some(from) = account.identity().map(str::to_string) else { + let Some(own) = account.identity().map(str::to_string) else { weft::node_bail!( "gmail: the Google connection recorded no account address to send from; \ reconnect it on the access node" ) }; + let other_address = alias.as_ref().is_some_and(|a| !a.eq_ignore_ascii_case(&own)); let http = ctx.client(&account).await?; + let from = match alias { + None => own, + Some(alias) if other_address => { + send_as_check(&http, &own, &alias).await?; + alias + } + Some(alias) => alias, + }; // Reply threading: the original's Message-ID header + thread. let mut in_reply_to = None; let mut thread_id = None; - if let Some(orig_id) = reply_to.filter(|r| !r.trim().is_empty()) { + if let Some(orig_id) = reply_to { let orig: Value = get_json( &http, &format!( "{API}/messages/{}?format=metadata&metadataHeaders=Message-ID", super::api::segment(&orig_id) ), - "gmail: read the replied-to message", + "gmail: read the replied-to message (replying needs 'Read mail', \ + 'Organize mail' or 'Read mail headers' ticked on the Google access node)", ) .await?; // No Message-ID means the reply cannot thread. Sending it @@ -71,10 +87,7 @@ impl Node for GmailSendNode { let mid = header(&orig["payload"], "Message-ID").ok_or_else(|| { weft::node_error(format!( "gmail: message {orig_id} carries no Message-ID header, so this reply \ - cannot be threaded onto it. If the account connection was made before \ - replies were used, reconnect it on the access node and tick \ - 'Read mail headers'; otherwise send this as a new message instead of \ - a reply." + cannot be threaded onto it; send this as a new message instead of a reply." )) })?; in_reply_to = Some(mid.to_string()); @@ -123,3 +136,48 @@ impl Node for GmailSendNode { .await } } + +/// Gmail honours a From only when it is one of the account's Send mail +/// as addresses, and silently sends from the account's own address +/// otherwise. So a `from` that is not the account's own address is +/// checked against that list first, and a miss fails naming the +/// addresses that would work. Reading the list needs 'Read mail' or +/// 'Organize mail' on the connection (send-mail alone cannot). +async fn send_as_check( + http: &weft::reqwest_middleware::ClientWithMiddleware, + own: &str, + alias: &str, +) -> WeftResult<()> { + // A connection with send-mail alone is refused here (403, + // "insufficient authentication scopes"); the attempt's name says + // which permission to tick so that refusal reads as its own fix. + let answer = get_json( + http, + &format!("{API}/settings/sendAs"), + "list the account's Send mail as addresses (sending from an alias needs \ + 'Read mail' or 'Organize mail' ticked on the Google access node)", + ) + .await?; + // The primary address is on the list too. Gmail sets + // `verificationStatus` only on a custom alias, and one still + // "pending" is rewritten to the primary exactly like an unknown + // address; an entry without the field (the primary, a Workspace + // domain alias) is usable as is. + let allowed: Vec<&str> = answer["sendAs"] + .as_array() + .map(Vec::as_slice) + .unwrap_or_default() + .iter() + .filter(|s| s["verificationStatus"].as_str().is_none_or(|v| v == "accepted")) + .filter_map(|s| s["sendAsEmail"].as_str()) + .collect(); + if allowed.iter().any(|a| a.eq_ignore_ascii_case(alias)) { + return Ok(()); + } + Err(weft::WeftError::Input(format!( + "gmail: '{alias}' is not one of this account's Send mail as addresses, so Gmail \ + would send from {own} instead. Allowed: {}. Add the address under Gmail's \ + Settings > Accounts > Send mail as, or pick one of these.", + if allowed.is_empty() { own.to_string() } else { allowed.join(", ") } + ))) +} diff --git a/catalog/google/gmail_send/tests.rs b/catalog/google/gmail_send/tests.rs index 2717940d..a5b145b5 100644 --- a/catalog/google/gmail_send/tests.rs +++ b/catalog/google/gmail_send/tests.rs @@ -20,10 +20,19 @@ pub fn tests() -> Vec { vec![ NodeTest::fake("sends_the_assembled_raw_message", sends), NodeTest::fake("attachments_and_reply_threading_ride_the_raw_message", attachments_and_reply), + NodeTest::fake("from_an_accepted_alias_names_it_as_the_sender", from_alias), + NodeTest::fake("from_an_address_off_the_send_as_list_fails_naming_the_allowed_ones", from_unknown), NodeTest::fake("no_recipient_refuses_before_any_call", no_recipient), NodeTest::fake("a_refused_send_fails_the_run_when_error_is_unwired", refused_unwired), NodeTest::fake("a_refused_send_comes_out_on_error_when_it_is_wired", refused_wired), NodeTest::fake("a_missing_recipient_is_never_caught_by_error", mistake_not_caught), + NodeTest::fake("a_plain_send_asks_only_for_send_mail", plain_send_permissions), + NodeTest::fake("a_reply_on_send_mail_alone_fails_naming_read_mail", reply_needs_read_mail), + NodeTest::fake( + "sending_from_another_address_on_send_mail_alone_fails_naming_read_mail", + alias_needs_read_mail, + ), + NodeTest::fake("a_reply_on_organize_mail_is_not_asked_for_read_mail", reply_on_organize_mail), NodeTest::live("one_real_send", "google", live_send).with_fixture(fixture_spec( "GMAIL_TO", "Recipient address", @@ -116,6 +125,59 @@ async fn sends(rig: FakeRig) -> WeftResult<()> { assert!(mime.contains("To: ada@example.com"), "{mime}"); assert!(mime.contains("Subject: hello"), "{mime}"); assert!(mime.contains("body text"), "{mime}"); + // No `from`: the connection's own address, and no Send mail as read. + assert!(mime.contains("From: me@example.com"), "{mime}"); + assert_eq!(rig.requests().len(), 1, "only the send"); + Ok(()) +} + +/// The account's Send mail as list as Gmail answers it: the primary +/// (no verificationStatus), a verified alias, and one still pending. +fn send_as_list(rig: &FakeRig) { + rig.respond( + "GET", + "/gmail/v1/users/me/settings/sendAs", + json!({ "sendAs": [ + { "sendAsEmail": "me@example.com", "isPrimary": true }, + { "sendAsEmail": "Team@Example.com", "verificationStatus": "accepted" }, + { "sendAsEmail": "pending@example.com", "verificationStatus": "pending" }, + ]}), + ); +} + +async fn from_alias(rig: FakeRig) -> WeftResult<()> { + send_as_list(&rig); + rig.respond( + "POST", + "/gmail/v1/users/me/messages/send", + json!({ "id": "m3", "threadId": "t3" }), + ); + let mut mail = a_mail(); + // Matched case-insensitively; the header keeps it as given. + mail["from"] = json!("team@example.com"); + rig.run(&GmailSendNode, mail).await.ok()?; + + let sent = rig.requests(); + assert_eq!(sent.len(), 2, "read the Send mail as list, then send"); + let body = sent[1].body.clone().expect("send body"); + let mime = base64::engine::general_purpose::URL_SAFE_NO_PAD + .decode(body["raw"].as_str().expect("raw message")) + .expect("raw is base64url"); + let mime = String::from_utf8_lossy(&mime); + assert!(mime.contains("From: team@example.com"), "{mime}"); + Ok(()) +} + +async fn from_unknown(rig: FakeRig) -> WeftResult<()> { + send_as_list(&rig); + let mut mail = a_mail(); + // A pending alias would be rewritten to the primary, so it is + // refused like an address Gmail has never heard of. + mail["from"] = json!("pending@example.com"); + let err = rig.run(&GmailSendNode, mail).await.failure()?; + assert!(err.contains("pending@example.com"), "{err}"); + assert!(err.contains("Allowed: me@example.com, Team@Example.com."), "{err}"); + assert_eq!(rig.requests().len(), 1, "nothing was sent"); Ok(()) } @@ -142,7 +204,7 @@ fn refuse_the_send(rig: &FakeRig) { ); } -fn a_mail(rig: &FakeRig) -> serde_json::Value { +fn a_mail() -> serde_json::Value { json!({ "account": google_account(), "to": "ada@example.com", @@ -153,7 +215,7 @@ fn a_mail(rig: &FakeRig) -> serde_json::Value { async fn refused_unwired(rig: FakeRig) -> WeftResult<()> { refuse_the_send(&rig); - let err = rig.run(&GmailSendNode, a_mail(&rig)).await.failure()?; + let err = rig.run(&GmailSendNode, a_mail()).await.failure()?; assert!(err.contains("Daily sending quota exceeded"), "{err}"); Ok(()) } @@ -161,7 +223,7 @@ async fn refused_unwired(rig: FakeRig) -> WeftResult<()> { async fn refused_wired(rig: FakeRig) -> WeftResult<()> { refuse_the_send(&rig); rig.wire_output("error"); - let outcome = rig.run(&GmailSendNode, a_mail(&rig)).await.ok()?; + let outcome = rig.run(&GmailSendNode, a_mail()).await.ok()?; let error = outcome.output("error")?.as_str().expect("error is a string").to_string(); assert!(error.contains("Daily sending quota exceeded"), "{error}"); for port in ["id", "threadId"] { @@ -185,6 +247,103 @@ async fn mistake_not_caught(rig: FakeRig) -> WeftResult<()> { Ok(()) } +const SEND_MAIL: &str = "https://www.googleapis.com/auth/gmail.send"; +const ORGANIZE_MAIL: &str = "https://www.googleapis.com/auth/gmail.modify"; + +async fn plain_send_permissions(rig: FakeRig) -> WeftResult<()> { + rig.connection_permissions("google", &[SEND_MAIL]); + rig.respond( + "POST", + "/gmail/v1/users/me/messages/send", + json!({ "id": "m4", "threadId": "t4" }), + ); + let mut mail = a_mail(); + // The account's own address, in another case, is no other address. + mail["from"] = json!("ME@example.com"); + rig.run(&GmailSendNode, mail).await.ok()?; + assert_eq!(rig.requests().len(), 1, "only the send"); + Ok(()) +} + +/// The replied-to message, as Gmail answers its metadata read. +fn original_message(rig: &FakeRig) { + rig.respond( + "GET", + "/gmail/v1/users/me/messages/orig-1?format=metadata&metadataHeaders=Message-ID", + json!({ + "threadId": "t-orig", + "payload": { "headers": [{ "name": "Message-ID", "value": "" }] } + }), + ); +} + +/// Organize mail reads everything Read mail does, so a connection +/// holding it and Send mail replies without Read mail being asked for. +async fn reply_on_organize_mail(rig: FakeRig) -> WeftResult<()> { + rig.connection_permissions("google", &[SEND_MAIL, ORGANIZE_MAIL]); + original_message(&rig); + rig.respond( + "POST", + "/gmail/v1/users/me/messages/send", + json!({ "id": "m5", "threadId": "t-orig" }), + ); + let mut mail = a_mail(); + mail["replyTo"] = json!("orig-1"); + rig.run(&GmailSendNode, mail).await.ok()?; + assert_eq!(rig.requests().len(), 2, "read the original, then send"); + Ok(()) +} + +/// Gmail's answer to a read on a token holding Send mail alone. +fn insufficient_scopes(rig: &FakeRig, path: &str) { + rig.respond_status( + "GET", + path, + 403, + json!({ "error": { + "code": 403, + "message": "Request had insufficient authentication scopes.", + "status": "PERMISSION_DENIED", + }}), + ); +} + +/// A connection holding Send mail alone gets Gmail's refusal on the +/// read, named with the permissions that would work, and nothing is +/// sent. +async fn refused_for_read_mail( + rig: &FakeRig, + mail: serde_json::Value, + read_path: &str, +) -> WeftResult<()> { + rig.connection_permissions("google", &[SEND_MAIL]); + insufficient_scopes(rig, read_path); + let err = rig.run(&GmailSendNode, mail).await.failure()?; + assert!(err.contains("insufficient authentication scopes"), "{err}"); + assert!(err.contains("'Read mail'") && err.contains("'Organize mail'"), "{err}"); + let sent = rig.requests(); + assert_eq!(sent.len(), 1, "only the refused read: {sent:?}"); + assert_eq!(sent[0].method, "GET", "nothing was sent"); + Ok(()) +} + +async fn reply_needs_read_mail(rig: FakeRig) -> WeftResult<()> { + let mut mail = a_mail(); + mail["replyTo"] = json!("orig-1"); + refused_for_read_mail( + &rig, + mail, + "/gmail/v1/users/me/messages/orig-1?format=metadata&metadataHeaders=Message-ID", + ) + .await +} + +async fn alias_needs_read_mail(rig: FakeRig) -> WeftResult<()> { + let mut mail = a_mail(); + mail["from"] = json!("team@example.com"); + refused_for_read_mail(&rig, mail, "/gmail/v1/users/me/settings/sendAs").await +} + async fn live_send(rig: LiveRig) -> WeftResult<()> { // A mailbox cannot be self-provisioned: the recipient is a // fixture. The sent mail stays (the artifact IS the proof). diff --git a/catalog/human/query/metadata.json b/catalog/human/query/metadata.json index 269f8779..b1896944 100644 --- a/catalog/human/query/metadata.json +++ b/catalog/human/query/metadata.json @@ -1,7 +1,7 @@ { "type": "HumanQuery", "label": "Human", - "description": "Pauses the run until a person submits a form, then continues with their answers; use it for review or approval mid-run. A run that cannot pause, such as a route still answering its caller, holds its worker instead, up to its trigger's `holdSecs`, then fails saying it gave up. Each form field adds input and/or output ports by its kind.", + "description": "Pauses the run until a person submits a form, then continues with their answers; use it for review or approval mid-run. A run that cannot pause, such as a route still answering its caller, holds its worker instead, up to its trigger's `holdSecs`, then fails saying it gave up. A person who skips the form ends this step skipped, with every output closed. Each form field adds input and/or output ports by its kind.", "tags": [ "human", "form", diff --git a/catalog/human/query/tests.rs b/catalog/human/query/tests.rs index 92d3c40c..3680b9af 100644 --- a/catalog/human/query/tests.rs +++ b/catalog/human/query/tests.rs @@ -12,6 +12,7 @@ pub fn tests() -> Vec { NodeTest::fake("an_image_field_parks_the_stored_file_for_the_files_door", image_parked), NodeTest::fake("a_submission_resumes_and_maps_to_ports", submission_maps), NodeTest::fake("a_fields_label_and_placeholder_reach_the_form", label_and_placeholder), + NodeTest::fake("a_skipped_form_sends_nothing_on", skipped_form), ] } @@ -77,6 +78,27 @@ async fn label_and_placeholder(rig: FakeRig) -> WeftResult<()> { Ok(()) } +/// A person skipping the form: the wait ends skipped, which ends the step +/// skipped with every output closed, whatever its fields are (an +/// approve/reject one included). +async fn skipped_form(rig: FakeRig) -> WeftResult<()> { + rig.output_type("decision_approved", WeftType::parse("Boolean").expect("Boolean parses")); + rig.output_type("decision_rejected", WeftType::parse("Boolean").expect("Boolean parses")); + rig.signal_skipped(); + let outcome = rig + .run( + &HumanQueryNode, + json!({ + "title": "Ship it?", + "fields": [{ "kind": "approve_reject", "key": "decision" }], + }), + ) + .await; + assert!(matches!(outcome.result, Err(weft::WeftError::WaitSkipped(_))), "{:?}", outcome.result); + assert!(outcome.outputs.is_empty(), "nothing sent on: {:?}", outcome.outputs); + Ok(()) +} + async fn submission_maps(rig: FakeRig) -> WeftResult<()> { rig.output_type("decision_approved", WeftType::parse("Boolean").expect("Boolean parses")); rig.output_type("decision_rejected", WeftType::parse("Boolean").expect("Boolean parses")); diff --git a/catalog/instances/activate_instance_triggers/metadata.json b/catalog/instances/activate_instance_triggers/metadata.json deleted file mode 100644 index e007405e..00000000 --- a/catalog/instances/activate_instance_triggers/metadata.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "type": "ActivateInstanceTriggers", - "label": "Activate instance triggers", - "description": "Turn on one instance's per-instance triggers. A trigger reading the instance's infra copy needs that copy running first; triggers already on are left alone.", - "tags": [ - "instances", - "triggers", - "activate" - ], - "icon": "Zap", - "color": "#0ea5e9", - "inputs": [ - { - "name": "instance", - "type": "String", - "required": true, - "description": "The instance id your program gives it (`user-42`)." - }, - { - "name": "only", - "type": "List[String]", - "description": "Only these triggers, named as the program writes them. Unwired: every per-instance trigger." - } - ], - "outputs": [ - { - "name": "done", - "type": "Boolean", - "description": "True once the triggers are on." - } - ], - "validate": [ - { - "when": { - "kind": "not", - "of": { "kind": "input_names", "port": "only", "names": { "node": { "role": "trigger", "per_instance": true } } } - }, - "then": { - "message": "ActivateInstanceTriggers '{id}' names {names} in `only`, which is no per-instance trigger of this program. Name each the way the program writes it (`inbox`, `work.inbox` inside a group, or `one.inbox` inside an included file).", - "level": "structural", - "severity": "error", - "port": "only" - } - } - ] -} diff --git a/catalog/instances/activate_instance_triggers/mod.rs b/catalog/instances/activate_instance_triggers/mod.rs deleted file mode 100644 index a6afd745..00000000 --- a/catalog/instances/activate_instance_triggers/mod.rs +++ /dev/null @@ -1,32 +0,0 @@ -//! ActivateInstanceTriggers: turn on one instance's own triggers, every -//! one of them or the ones `only` names. - -use async_trait::async_trait; - -use weft::node::NodeOutput; -use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; - -use super::lifecycle::instance; - -#[derive(NodeManifest)] -pub struct ActivateInstanceTriggersNode; - -#[cfg(feature = "node-tests")] -mod tests; - -#[async_trait] -impl Node for ActivateInstanceTriggersNode { - #[cfg(feature = "node-tests")] - fn tests(&self) -> Vec { - tests::tests() - } - - async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { - let instance = instance(&ctx)?; - let only: Vec = ctx.inputs.opt("only")?.unwrap_or_default(); - // None named is every per-instance trigger of the instance. - let calls = ctx.triggers().only(only).instance(instance); - calls.activate().await?; - ctx.pulse_downstream(NodeOutput::new().set("done", true)).await - } -} diff --git a/catalog/instances/activate_instance_triggers/tests.rs b/catalog/instances/activate_instance_triggers/tests.rs deleted file mode 100644 index f2880f88..00000000 --- a/catalog/instances/activate_instance_triggers/tests.rs +++ /dev/null @@ -1,37 +0,0 @@ -//! ActivateInstanceTriggers self-tests: every per-instance trigger of the -//! instance by default, only the named ones when `only` is wired. - -use serde_json::json; - -use weft::program::ProgramCall; -use weft::{FakeRig, NodeTest, WeftResult}; - -use super::ActivateInstanceTriggersNode; - -pub fn tests() -> Vec { - vec![ - NodeTest::fake("every_trigger_of_the_instance", every), - NodeTest::fake("only_the_named_ones", only), - ] -} - -async fn every(rig: FakeRig) -> WeftResult<()> { - rig.run(&ActivateInstanceTriggersNode, json!({ "instance": "ada" })).await.ok()?; - match &rig.program_calls()[0].0 { - ProgramCall::TriggerActivate { scope } => { - assert!(scope.triggers.is_empty()); - assert_eq!(scope.instance.as_ref().unwrap().as_str(), "ada"); - } - other => panic!("unexpected call {other:?}"), - } - Ok(()) -} - -async fn only(rig: FakeRig) -> WeftResult<()> { - rig.run(&ActivateInstanceTriggersNode, json!({ "instance": "ada", "only": ["receive"] })).await.ok()?; - match &rig.program_calls()[0].0 { - ProgramCall::TriggerActivate { scope } => assert_eq!(scope.triggers, vec!["receive".to_string()]), - other => panic!("unexpected call {other:?}"), - } - Ok(()) -} diff --git a/catalog/instances/activate_triggers/metadata.json b/catalog/instances/activate_triggers/metadata.json new file mode 100644 index 00000000..0583e0c9 --- /dev/null +++ b/catalog/instances/activate_triggers/metadata.json @@ -0,0 +1,97 @@ +{ + "type": "ActivateTriggers", + "label": "Activate triggers", + "description": "Turn on triggers: the program's own, or with `instance` that instance's per-instance triggers. A trigger reading an infra copy needs that copy running first; triggers already on are left alone.", + "tags": [ + "instances", + "triggers", + "activate" + ], + "icon": "Zap", + "color": "#0ea5e9", + "inputs": [ + { + "name": "instance", + "type": "String", + "requiredWhenWired": true, + "description": "Whose triggers: an instance id your program gives it (`user-42`), for that instance's per-instance triggers. Unwired: the program's own triggers, the ones shared by every instance. Wired, and no id arrives (the step feeding it did not run): this node skips." + }, + { + "name": "only", + "type": "List[String]", + "description": "Only these triggers, named as the program writes them. Unwired: all of them (every shared trigger, or every per-instance trigger of the instance)." + } + ], + "outputs": [ + { + "name": "done", + "type": "Boolean", + "description": "True once the triggers are on." + } + ], + "validate": [ + { + "when": { + "kind": "all", + "of": [ + { + "kind": "input_satisfied", + "port": "instance" + }, + { + "kind": "not", + "of": { + "kind": "input_names", + "port": "only", + "names": { + "node": { + "role": "trigger", + "per_instance": true + } + } + } + } + ] + }, + "then": { + "message": "ActivateTriggers '{id}' names {names} in `only`, which is no per-instance trigger of this program. It is given an `instance`, so it acts on that instance's triggers: name each the way the program writes it (`inbox`, `work.inbox` inside a group, or `one.inbox` inside an included file), or leave `instance` unwired to act on the program's own triggers.", + "level": "structural", + "severity": "error", + "port": "only" + } + }, + { + "when": { + "kind": "all", + "of": [ + { + "kind": "not", + "of": { + "kind": "input_satisfied", + "port": "instance" + } + }, + { + "kind": "not", + "of": { + "kind": "input_names", + "port": "only", + "names": { + "node": { + "role": "trigger", + "per_instance": false + } + } + } + } + ] + }, + "then": { + "message": "ActivateTriggers '{id}' names {names} in `only`, which is no shared trigger of this program. It has no `instance`, so it acts on the program's own triggers: name each the way the program writes it (`inbox`, `work.inbox` inside a group, or `one.inbox` inside an included file), or wire `instance` to act on an instance's per-instance triggers.", + "level": "structural", + "severity": "error", + "port": "only" + } + } + ] +} diff --git a/catalog/instances/activate_triggers/mod.rs b/catalog/instances/activate_triggers/mod.rs new file mode 100644 index 00000000..3d9cdc9f --- /dev/null +++ b/catalog/instances/activate_triggers/mod.rs @@ -0,0 +1,37 @@ +//! ActivateTriggers: turn on the program's own triggers or, given +//! `instance`, that instance's, every one of them or the ones `only` +//! names. + +use async_trait::async_trait; + +use weft::node::NodeOutput; +use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; + +use super::lifecycle::instance_if_given; + +#[derive(NodeManifest)] +pub struct ActivateTriggersNode; + +#[cfg(feature = "node-tests")] +mod tests; + +#[async_trait] +impl Node for ActivateTriggersNode { + #[cfg(feature = "node-tests")] + fn tests(&self) -> Vec { + tests::tests() + } + + async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { + let instance = instance_if_given(&ctx)?; + let only: Vec = ctx.inputs.opt("only")?.unwrap_or_default(); + // None named is every trigger of the owner: every shared trigger, + // or every per-instance trigger of the instance. + let mut calls = ctx.triggers().only(only); + if let Some(id) = instance { + calls = calls.instance(id); + } + calls.activate().await?; + ctx.pulse_downstream(NodeOutput::new().set("done", true)).await + } +} diff --git a/catalog/instances/activate_triggers/tests.rs b/catalog/instances/activate_triggers/tests.rs new file mode 100644 index 00000000..28eb27ff --- /dev/null +++ b/catalog/instances/activate_triggers/tests.rs @@ -0,0 +1,51 @@ +//! ActivateTriggers self-tests: every per-instance trigger of the +//! instance by default, every shared trigger when no instance is given, +//! only the named ones when `only` is wired. + +use serde_json::json; + +use weft::program::ProgramCall; +use weft::{FakeRig, NodeTest, WeftResult}; + +use super::ActivateTriggersNode; + +pub fn tests() -> Vec { + vec![ + NodeTest::fake("every_trigger_of_the_instance", every), + NodeTest::fake("only_the_named_ones", only), + NodeTest::fake("no_instance_is_the_programs_own_triggers", the_programs_own), + ] +} + +async fn every(rig: FakeRig) -> WeftResult<()> { + rig.run(&ActivateTriggersNode, json!({ "instance": "ada" })).await.ok()?; + match &rig.program_calls()[0].0 { + ProgramCall::TriggerActivate { scope } => { + assert!(scope.triggers.is_empty()); + assert_eq!(scope.instance.as_ref().unwrap().as_str(), "ada"); + } + other => panic!("unexpected call {other:?}"), + } + Ok(()) +} + +async fn only(rig: FakeRig) -> WeftResult<()> { + rig.run(&ActivateTriggersNode, json!({ "instance": "ada", "only": ["receive"] })).await.ok()?; + match &rig.program_calls()[0].0 { + ProgramCall::TriggerActivate { scope } => assert_eq!(scope.triggers, vec!["receive".to_string()]), + other => panic!("unexpected call {other:?}"), + } + Ok(()) +} + +async fn the_programs_own(rig: FakeRig) -> WeftResult<()> { + rig.run(&ActivateTriggersNode, json!({ "only": ["inbox"] })).await.ok()?; + match &rig.program_calls()[0].0 { + ProgramCall::TriggerActivate { scope } => { + assert_eq!(scope.triggers, vec!["inbox".to_string()]); + assert!(scope.instance.is_none(), "the program's own triggers"); + } + other => panic!("unexpected call {other:?}"), + } + Ok(()) +} diff --git a/catalog/instances/deactivate_instance_triggers/metadata.json b/catalog/instances/deactivate_instance_triggers/metadata.json deleted file mode 100644 index 8d61dd05..00000000 --- a/catalog/instances/deactivate_instance_triggers/metadata.json +++ /dev/null @@ -1,136 +0,0 @@ -{ - "type": "DeactivateInstanceTriggers", - "label": "Deactivate instance triggers", - "description": "Turn off one instance's per-instance triggers (park, hibernate or wipe), with `running` deciding what happens to its runs in flight.", - "tags": [ - "instances", - "triggers", - "deactivate" - ], - "icon": "ZapOff", - "color": "#0ea5e9", - "inputs": [ - { - "name": "instance", - "type": "String", - "required": true, - "description": "The instance id your program gives it (`user-42`)." - }, - { - "name": "only", - "type": "List[String]", - "description": "Only these triggers. Unwired: every per-instance trigger." - }, - { - "name": "triggers", - "type": "String", - "widget": { - "kind": "select", - "options": [ - "park", - "hibernate", - "wipe" - ] - }, - "default": "park", - "label": "Triggers", - "description": "Park keeps everything that arrives (answers, events, the trigger's own fires) and runs it once the trigger is back, while a caller on a route is told to try again in a moment; hibernate does the same for `graceMinutes` with its questions hidden, then takes nothing more; wipe drops it all." - }, - { - "name": "running", - "type": "String", - "widget": { - "kind": "select", - "options": [ - "wait", - "cancel" - ] - }, - "default": "wait", - "label": "Running work", - "description": "The instance's runs still going: wait lets them finish, cancel stops them. `wipe` needs cancel." - }, - { - "name": "graceMinutes", - "type": "Number", - "default": 15, - "label": "Grace minutes", - "description": "Hibernate only: how long it keeps accepting.", - "widget": { - "kind": "number", - "min": 0, - "step": 1 - } - }, - { - "name": "includeSelf", - "type": "Boolean", - "default": false, - "label": "Stop this run too", - "description": "On: this run is cancelled too when it is one of the runs reached." - } - ], - "outputs": [ - { - "name": "done", - "type": "Boolean", - "description": "True once the triggers are down; under `running: wait` their runs may still be finishing." - } - ], - "validate": [ - { - "when": { - "kind": "not", - "of": { - "kind": "input_names", - "port": "only", - "names": { - "node": { - "role": "trigger", - "per_instance": true - } - } - } - }, - "then": { - "message": "DeactivateInstanceTriggers '{id}' names {names} in `only`, which is no per-instance trigger of this program. Name each the way the program writes it (`inbox`, `work.inbox` inside a group, or `one.inbox` inside an included file).", - "level": "structural", - "severity": "error", - "port": "only" - } - }, - { - "when": { - "kind": "all", - "of": [ - { - "kind": "config_equals", - "field": "triggers", - "equals": "wipe" - }, - { - "kind": "not", - "of": { - "kind": "config_equals", - "field": "running", - "equals": "cancel" - } - }, - { - "kind": "not", - "of": { - "kind": "input_wired", - "port": "running" - } - } - ] - }, - "then": { - "message": "DeactivateInstanceTriggers '{id}' wipes the triggers but lets running work finish. Wiping drops what the runs still need, so set `running` to cancel, or pick park or hibernate for `triggers`.", - "level": "structural", - "severity": "error", - "field": "running" - } - } - ] -} diff --git a/catalog/instances/deactivate_instance_triggers/mod.rs b/catalog/instances/deactivate_instance_triggers/mod.rs deleted file mode 100644 index 4fe83371..00000000 --- a/catalog/instances/deactivate_instance_triggers/mod.rs +++ /dev/null @@ -1,34 +0,0 @@ -//! DeactivateInstanceTriggers: turn off one instance's own triggers, -//! every one of them or the ones `only` names, the way the node's choices -//! say. - -use async_trait::async_trait; - -use weft::node::NodeOutput; -use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; - -use super::lifecycle::{instance, stop_self, take_down_spec}; - -#[derive(NodeManifest)] -pub struct DeactivateInstanceTriggersNode; - -#[cfg(feature = "node-tests")] -mod tests; - -#[async_trait] -impl Node for DeactivateInstanceTriggersNode { - #[cfg(feature = "node-tests")] - fn tests(&self) -> Vec { - tests::tests() - } - - async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { - let instance = instance(&ctx)?; - let only: Vec = ctx.inputs.opt("only")?.unwrap_or_default(); - let spec = take_down_spec(&ctx)?; - // None named is every per-instance trigger of the instance. - let calls = ctx.triggers().only(only).instance(instance); - calls.deactivate(spec, stop_self(&ctx)?).await?; - ctx.pulse_downstream(NodeOutput::new().set("done", true)).await - } -} diff --git a/catalog/instances/deactivate_instance_triggers/tests.rs b/catalog/instances/deactivate_instance_triggers/tests.rs deleted file mode 100644 index e5d11fbe..00000000 --- a/catalog/instances/deactivate_instance_triggers/tests.rs +++ /dev/null @@ -1,30 +0,0 @@ -//! DeactivateInstanceTriggers self-tests: the instance's triggers go down -//! with the node's choices. - -use serde_json::json; - -use weft::program::ProgramCall; -use weft::{DeactivationMode, FakeRig, NodeTest, StopSelf, WeftResult}; - -use super::DeactivateInstanceTriggersNode; - -pub fn tests() -> Vec { - vec![NodeTest::fake("the_instances_triggers_go_down", down)] -} - -async fn down(rig: FakeRig) -> WeftResult<()> { - rig.run( - &DeactivateInstanceTriggersNode, - json!({ "instance": "ada", "triggers": "wipe", "running": "cancel", "includeSelf": true }), - ) - .await - .ok()?; - match &rig.program_calls()[0] { - (ProgramCall::TriggerDeactivate { scope, spec }, StopSelf::Include) => { - assert_eq!(scope.instance.as_ref().unwrap().as_str(), "ada"); - assert_eq!(spec.mode, DeactivationMode::Wipe); - } - other => panic!("unexpected call {other:?}"), - } - Ok(()) -} diff --git a/catalog/instances/deactivate_triggers/metadata.json b/catalog/instances/deactivate_triggers/metadata.json new file mode 100644 index 00000000..54fa33ff --- /dev/null +++ b/catalog/instances/deactivate_triggers/metadata.json @@ -0,0 +1,178 @@ +{ + "type": "DeactivateTriggers", + "label": "Deactivate triggers", + "description": "Turn off triggers (park, hibernate or wipe): the program's own, or with `instance` that instance's per-instance triggers, with `running` deciding what happens to the runs they fired that are still going.", + "tags": [ + "instances", + "triggers", + "deactivate" + ], + "icon": "ZapOff", + "color": "#0ea5e9", + "inputs": [ + { + "name": "instance", + "type": "String", + "requiredWhenWired": true, + "description": "Whose triggers: an instance id your program gives it (`user-42`), for that instance's per-instance triggers. Unwired: the program's own triggers, the ones shared by every instance. Wired, and no id arrives (the step feeding it did not run): this node skips." + }, + { + "name": "only", + "type": "List[String]", + "description": "Only these triggers, named as the program writes them. Unwired: all of them (every shared trigger, or every per-instance trigger of the instance)." + }, + { + "name": "triggers", + "type": "String", + "widget": { + "kind": "select", + "options": [ + "park", + "hibernate", + "wipe" + ] + }, + "default": "park", + "label": "Triggers", + "description": "Park keeps everything that arrives (answers, events, the trigger's own fires) and runs it once the trigger is back, while a caller on a route is told to try again in a moment; hibernate does the same for `graceMinutes` with its questions hidden, then takes nothing more; wipe drops it all." + }, + { + "name": "running", + "type": "String", + "widget": { + "kind": "select", + "options": [ + "wait", + "cancel" + ] + }, + "default": "wait", + "label": "Running work", + "description": "The runs they fired that are still going: wait lets them finish, cancel stops them. `wipe` needs cancel." + }, + { + "name": "graceMinutes", + "type": "Number", + "default": 15, + "label": "Grace minutes", + "description": "Hibernate only: how long it keeps accepting.", + "widget": { + "kind": "number", + "min": 0, + "step": 1 + } + }, + { + "name": "includeSelf", + "type": "Boolean", + "default": false, + "label": "Stop this run too", + "description": "On: this run is cancelled too when it is one of the runs reached." + } + ], + "outputs": [ + { + "name": "done", + "type": "Boolean", + "description": "True once the triggers are down; under `running: wait` their runs may still be finishing." + } + ], + "validate": [ + { + "when": { + "kind": "all", + "of": [ + { + "kind": "input_satisfied", + "port": "instance" + }, + { + "kind": "not", + "of": { + "kind": "input_names", + "port": "only", + "names": { + "node": { + "role": "trigger", + "per_instance": true + } + } + } + } + ] + }, + "then": { + "message": "DeactivateTriggers '{id}' names {names} in `only`, which is no per-instance trigger of this program. It is given an `instance`, so it acts on that instance's triggers: name each the way the program writes it (`inbox`, `work.inbox` inside a group, or `one.inbox` inside an included file), or leave `instance` unwired to act on the program's own triggers.", + "level": "structural", + "severity": "error", + "port": "only" + } + }, + { + "when": { + "kind": "all", + "of": [ + { + "kind": "not", + "of": { + "kind": "input_satisfied", + "port": "instance" + } + }, + { + "kind": "not", + "of": { + "kind": "input_names", + "port": "only", + "names": { + "node": { + "role": "trigger", + "per_instance": false + } + } + } + } + ] + }, + "then": { + "message": "DeactivateTriggers '{id}' names {names} in `only`, which is no shared trigger of this program. It has no `instance`, so it acts on the program's own triggers: name each the way the program writes it (`inbox`, `work.inbox` inside a group, or `one.inbox` inside an included file), or wire `instance` to act on an instance's per-instance triggers.", + "level": "structural", + "severity": "error", + "port": "only" + } + }, + { + "when": { + "kind": "all", + "of": [ + { + "kind": "config_equals", + "field": "triggers", + "equals": "wipe" + }, + { + "kind": "not", + "of": { + "kind": "config_equals", + "field": "running", + "equals": "cancel" + } + }, + { + "kind": "not", + "of": { + "kind": "input_wired", + "port": "running" + } + } + ] + }, + "then": { + "message": "DeactivateTriggers '{id}' wipes the triggers but lets running work finish. Wiping drops what the runs still need, so set `running` to cancel, or pick park or hibernate for `triggers`.", + "level": "structural", + "severity": "error", + "field": "running" + } + } + ] +} diff --git a/catalog/instances/deactivate_triggers/mod.rs b/catalog/instances/deactivate_triggers/mod.rs new file mode 100644 index 00000000..ed6cc0d2 --- /dev/null +++ b/catalog/instances/deactivate_triggers/mod.rs @@ -0,0 +1,38 @@ +//! DeactivateTriggers: turn off the program's own triggers or, given +//! `instance`, that instance's, every one of them or the ones `only` +//! names, the way the node's choices say. + +use async_trait::async_trait; + +use weft::node::NodeOutput; +use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; + +use super::lifecycle::{instance_if_given, stop_self, take_down_spec}; + +#[derive(NodeManifest)] +pub struct DeactivateTriggersNode; + +#[cfg(feature = "node-tests")] +mod tests; + +#[async_trait] +impl Node for DeactivateTriggersNode { + #[cfg(feature = "node-tests")] + fn tests(&self) -> Vec { + tests::tests() + } + + async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { + let instance = instance_if_given(&ctx)?; + let only: Vec = ctx.inputs.opt("only")?.unwrap_or_default(); + let spec = take_down_spec(&ctx)?; + // None named is every trigger of the owner: every shared trigger, + // or every per-instance trigger of the instance. + let mut calls = ctx.triggers().only(only); + if let Some(id) = instance { + calls = calls.instance(id); + } + calls.deactivate(spec, stop_self(&ctx)?).await?; + ctx.pulse_downstream(NodeOutput::new().set("done", true)).await + } +} diff --git a/catalog/instances/deactivate_triggers/tests.rs b/catalog/instances/deactivate_triggers/tests.rs new file mode 100644 index 00000000..8a205bb2 --- /dev/null +++ b/catalog/instances/deactivate_triggers/tests.rs @@ -0,0 +1,47 @@ +//! DeactivateTriggers self-tests: the instance's triggers go down +//! with the node's choices, and the program's own when no instance is +//! given. + +use serde_json::json; + +use weft::program::ProgramCall; +use weft::{DeactivationMode, FakeRig, NodeTest, StopSelf, WeftResult}; + +use super::DeactivateTriggersNode; + +pub fn tests() -> Vec { + vec![ + NodeTest::fake("the_instances_triggers_go_down", down), + NodeTest::fake("no_instance_is_the_programs_own_triggers", the_programs_own), + ] +} + +async fn down(rig: FakeRig) -> WeftResult<()> { + rig.run( + &DeactivateTriggersNode, + json!({ "instance": "ada", "triggers": "wipe", "running": "cancel", "includeSelf": true }), + ) + .await + .ok()?; + match &rig.program_calls()[0] { + (ProgramCall::TriggerDeactivate { scope, spec }, StopSelf::Include) => { + assert_eq!(scope.instance.as_ref().unwrap().as_str(), "ada"); + assert_eq!(spec.mode, DeactivationMode::Wipe); + } + other => panic!("unexpected call {other:?}"), + } + Ok(()) +} + +async fn the_programs_own(rig: FakeRig) -> WeftResult<()> { + rig.run(&DeactivateTriggersNode, json!({})).await.ok()?; + match &rig.program_calls()[0] { + (ProgramCall::TriggerDeactivate { scope, spec }, StopSelf::Keep) => { + assert!(scope.triggers.is_empty(), "every shared trigger"); + assert!(scope.instance.is_none(), "the program's own triggers"); + assert_eq!(spec.mode, DeactivationMode::Park); + } + other => panic!("unexpected call {other:?}"), + } + Ok(()) +} diff --git a/catalog/instances/lifecycle.rs b/catalog/instances/lifecycle.rs index b59e5fe3..e8ca2074 100644 --- a/catalog/instances/lifecycle.rs +++ b/catalog/instances/lifecycle.rs @@ -1,7 +1,9 @@ -//! Shared by the instance nodes that take something down: how they read -//! what happens to the triggers and runs they reach, and whether the run -//! doing it goes too. One reader, so every such node offers the same -//! choices with the same defaults. +//! Shared by the instance nodes: how they read an instance id (one they +//! always need, or one that, left out, means the program's own copies), +//! and, for the ones that take something down, what happens to the +//! triggers and runs they reach and whether the run doing it goes too. +//! One reader, so every such node offers the same choices with the same +//! defaults. use weft::instance::InstanceId; use weft::node::NodeOutput; @@ -19,6 +21,21 @@ pub fn instance(ctx: &ExecutionContext) -> WeftResult { } } +/// The instance id a node acting on either kind of copy was handed, or +/// `None` when `instance` is unwired, which means the program's own +/// copies. A given id is checked like [`instance`], so an empty one (a +/// value that never arrived) fails rather than reaching the program's +/// own copies by accident. +pub fn instance_if_given(ctx: &ExecutionContext) -> WeftResult> { + match ctx.inputs.opt::("instance")? { + None => Ok(None), + Some(given) => match InstanceId::new(given) { + Ok(id) => Ok(Some(id)), + Err(why) => node_bail!("instance: {why}"), + }, + } +} + /// What a take-down does to the triggers it reaches (`triggers`: park, /// hibernate or wipe) and to the runs they fired (`running`: wait or /// cancel). Waiting before a wipe is refused, as everywhere else. @@ -49,10 +66,10 @@ pub fn stop_self(ctx: &ExecutionContext) -> WeftResult { Ok(if ctx.inputs.get::("includeSelf")? { StopSelf::Include } else { StopSelf::Keep }) } -/// What a node that takes one instance's copy down sends on: `done` when -/// the copy is down (taken down now, or already down), `noCopy` when that -/// instance has no copy of the node (a mistyped id, or one terminated -/// before), so a program answering a person can tell them which. +/// What a node that takes a copy down sends on: `done` when the copy is +/// down (taken down now, or already down), `noCopy` when there is no such +/// copy (never started, terminated before, or a mistyped instance id), so +/// a program answering a person can tell them which. pub fn taken_down(answer: InfraDownAnswer) -> NodeOutput { match answer { InfraDownAnswer::TakenDown | InfraDownAnswer::AlreadyDown => NodeOutput::new().set("done", true), diff --git a/catalog/instances/package.toml b/catalog/instances/package.toml index ef06d7c1..41882fce 100644 --- a/catalog/instances/package.toml +++ b/catalog/instances/package.toml @@ -2,8 +2,10 @@ name = "instances" # Nodes for a program that runs several instances: separate running -# copies of part of the program, each under its own id. Bringing an -# instance's own infra up and down, turning its triggers on and off, +# copies of part of the program, each under its own id. Bringing infra up +# and down and turning triggers on and off (`StartInfra`, `StopInfra`, +# `TerminateInfra`, `ActivateTriggers`, `DeactivateTriggers`: an +# instance's copies when given one, the program's own copies otherwise), # handing a browser an instance token, reading and setting the values an # instance was given, reading which instance a run is for, what an # instance spent, which infra copies exist and every instance weft holds diff --git a/catalog/instances/start_infra/metadata.json b/catalog/instances/start_infra/metadata.json new file mode 100644 index 00000000..dbed5145 --- /dev/null +++ b/catalog/instances/start_infra/metadata.json @@ -0,0 +1,105 @@ +{ + "type": "StartInfra", + "label": "Start infra", + "description": "Bring up a copy of an infra node: the program's own, or with `instance` that instance's copy of a `@per_instance` node. `done` fires once it answers, so the triggers reading it can be activated next. The run waits without holding a worker; a copy that fails to come up fails the node with the reason; an already-running copy is left as is. If you want to answer a caller fast (a route replying 202), turn `waitUntilRunning` off and reply after this node: it finishes as soon as weft accepts the start, and a status read right after shows the copy as `provisioning`.", + "tags": [ + "instances", + "infra", + "start" + ], + "icon": "Play", + "color": "#0ea5e9", + "inputs": [ + { + "name": "node", + "type": "String", + "required": true, + "description": "The infra node, named as the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file): a shared one when `instance` is unwired, one marked `@per_instance` when it is set." + }, + { + "name": "instance", + "type": "String", + "requiredWhenWired": true, + "description": "Whose copy: an instance id your program gives it (`user-42`), for that instance's copy of a `@per_instance` node. Unwired: the program's own copy. Wired, and no id arrives (the step feeding it did not run): this node skips." + }, + { + "name": "waitUntilRunning", + "type": "Boolean", + "default": true, + "label": "Wait until running", + "description": "On: `done` fires once the copy runs, and a copy that fails to come up fails the node. Off: `done` fires once weft accepts the start (a start refused for a passing reason, such as a build in progress, is asked again until it is), and whether the copy then comes up is for a status read to tell." + } + ], + "outputs": [ + { + "name": "done", + "type": "Boolean", + "description": "True once the copy is running, or once weft accepted the start when `waitUntilRunning` is off." + } + ], + "validate": [ + { + "when": { + "kind": "all", + "of": [ + { + "kind": "input_satisfied", + "port": "instance" + }, + { + "kind": "not", + "of": { + "kind": "input_names", + "port": "node", + "names": { + "node": { + "role": "infra", + "per_instance": true + } + } + } + } + ] + }, + "then": { + "message": "StartInfra '{id}' names {names} in `node`, which is no infra node of this program marked `@per_instance`. It is given an `instance`, so it acts on that instance's copy: name one the way the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file), or leave `instance` unwired to act on the program's own copy of a shared one.", + "level": "structural", + "severity": "error", + "port": "node" + } + }, + { + "when": { + "kind": "all", + "of": [ + { + "kind": "not", + "of": { + "kind": "input_satisfied", + "port": "instance" + } + }, + { + "kind": "not", + "of": { + "kind": "input_names", + "port": "node", + "names": { + "node": { + "role": "infra", + "per_instance": false + } + } + } + } + ] + }, + "then": { + "message": "StartInfra '{id}' names {names} in `node`, which is no shared infra node of this program. It has no `instance`, so it acts on the program's own copy: name one the way the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file), or wire `instance` to act on an instance's copy of a `@per_instance` one.", + "level": "structural", + "severity": "error", + "port": "node" + } + } + ] +} diff --git a/catalog/instances/start_instance_infra/mod.rs b/catalog/instances/start_infra/mod.rs similarity index 59% rename from catalog/instances/start_instance_infra/mod.rs rename to catalog/instances/start_infra/mod.rs index 52039684..0a672c22 100644 --- a/catalog/instances/start_instance_infra/mod.rs +++ b/catalog/instances/start_infra/mod.rs @@ -1,22 +1,23 @@ -//! StartInstanceInfra: bring up one instance's copy of a `@per_instance` -//! infra node, and fire `done` once the copy answers, or, with -//! `waitUntilRunning` off, once weft accepted the start. +//! StartInfra: bring up a copy of an infra node, the program's own or, +//! given `instance`, that instance's copy of a `@per_instance` node, and +//! fire `done` once the copy answers, or, with `waitUntilRunning` off, +//! once weft accepted the start. use async_trait::async_trait; use weft::node::NodeOutput; use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; -use super::lifecycle::instance; +use super::lifecycle::instance_if_given; #[derive(NodeManifest)] -pub struct StartInstanceInfraNode; +pub struct StartInfraNode; #[cfg(feature = "node-tests")] mod tests; #[async_trait] -impl Node for StartInstanceInfraNode { +impl Node for StartInfraNode { #[cfg(feature = "node-tests")] fn tests(&self) -> Vec { tests::tests() @@ -25,7 +26,10 @@ impl Node for StartInstanceInfraNode { async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { let node: String = ctx.inputs.get("node")?; let wait_until_running: bool = ctx.inputs.get("waitUntilRunning")?; - let copy = ctx.infra(node).instance(instance(&ctx)?); + let mut copy = ctx.infra(node); + if let Some(id) = instance_if_given(&ctx)? { + copy = copy.instance(id); + } if wait_until_running { copy.start().await?; } else { diff --git a/catalog/instances/start_instance_infra/tests.rs b/catalog/instances/start_infra/tests.rs similarity index 67% rename from catalog/instances/start_instance_infra/tests.rs rename to catalog/instances/start_infra/tests.rs index 841358de..9b718d5f 100644 --- a/catalog/instances/start_instance_infra/tests.rs +++ b/catalog/instances/start_infra/tests.rs @@ -1,21 +1,25 @@ -//! StartInstanceInfra self-tests: the start is asked for that instance's -//! copy of that node and `done` waits for the copy to run, a copy that +//! StartInfra self-tests: the start is asked for that instance's copy +//! of that node, or the program's own copy when `instance` is unwired, +//! and `done` waits for the copy to run, a copy that //! fails fails the node, a copy somebody stops (or that never came to be) //! while the node waits fails it instead of being started again, a start //! refused for a passing reason is asked again, a node told not to wait //! finishes once the start is accepted without looking at the copy, and a -//! blank instance is refused before anything is asked. +//! blank or empty instance is refused before anything is asked (a value +//! that never arrived must not reach the program's own copy). use serde_json::json; use weft::program::ProgramCall; use weft::{FakeRig, NodeTest, StopSelf, WeftResult}; -use super::StartInstanceInfraNode; +use super::StartInfraNode; pub fn tests() -> Vec { vec![ NodeTest::fake("starts_the_instances_copy", starts_the_instances_copy), + NodeTest::fake("no_instance_starts_the_programs_own_copy", starts_the_programs_own_copy), + NodeTest::fake("an_empty_instance_is_refused", empty_instance), NodeTest::fake("waits_until_the_copy_runs", waits_until_the_copy_runs), NodeTest::fake("a_copy_that_fails_fails_the_node", a_copy_that_fails), NodeTest::fake("a_copy_stopped_while_waiting_fails_the_node", stopped_while_waiting), @@ -29,7 +33,7 @@ pub fn tests() -> Vec { async fn starts_the_instances_copy(rig: FakeRig) -> WeftResult<()> { rig.answer_program_call("weft.infra.status", running()); - let outcome = rig.run(&StartInstanceInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.ok()?; + let outcome = rig.run(&StartInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.ok()?; let calls = rig.program_calls(); assert_eq!(calls.len(), 2, "the start, then one look that found it running"); match &calls[0] { @@ -43,8 +47,33 @@ async fn starts_the_instances_copy(rig: FakeRig) -> WeftResult<()> { Ok(()) } +async fn starts_the_programs_own_copy(rig: FakeRig) -> WeftResult<()> { + rig.answer_program_call("weft.infra.status", json!({ "node": "bridge", "status": "running" })); + let outcome = rig.run(&StartInfraNode, json!({ "node": "bridge" })).await.ok()?; + let calls = rig.program_calls(); + assert_eq!(calls.len(), 2, "the start, then one look that found it running: {calls:?}"); + for (call, _) in &calls { + match call { + ProgramCall::InfraStart { node, instance } | ProgramCall::InfraStatus { node, instance } => { + assert_eq!(node, "bridge"); + assert!(instance.is_none(), "the program's own copy: {call:?}"); + } + other => panic!("unexpected call {other:?}"), + } + } + assert_eq!(outcome.outputs["done"], json!(true)); + Ok(()) +} + +async fn empty_instance(rig: FakeRig) -> WeftResult<()> { + let err = rig.run(&StartInfraNode, json!({ "node": "bridge", "instance": "", "waitUntilRunning": false })).await.failure()?; + assert!(err.contains("is not a valid instance id"), "{err}"); + assert!(rig.program_calls().is_empty()); + Ok(()) +} + async fn blank_instance(rig: FakeRig) -> WeftResult<()> { - let err = rig.run(&StartInstanceInfraNode, json!({ "node": "bridge", "instance": " " })).await.failure()?; + let err = rig.run(&StartInfraNode, json!({ "node": "bridge", "instance": " " })).await.failure()?; assert!(err.contains("is not a valid instance id"), "{err}"); assert!(rig.program_calls().is_empty()); Ok(()) @@ -58,7 +87,7 @@ async fn waits_until_the_copy_runs(rig: FakeRig) -> WeftResult<()> { rig.answer_program_call("weft.infra.status", json!({ "node": "bridge", "instance": "user-42", "status": "provisioning" })); rig.answer_program_call("weft.infra.status", running()); rig.signal(json!({})); - let outcome = rig.run(&StartInstanceInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.ok()?; + let outcome = rig.run(&StartInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.ok()?; assert_eq!(rig.awaited_signals().len(), 1, "one wait between the two looks"); assert_eq!(outcome.outputs["done"], json!(true)); Ok(()) @@ -69,7 +98,7 @@ async fn a_copy_that_fails(rig: FakeRig) -> WeftResult<()> { "weft.infra.status", json!({ "node": "bridge", "instance": "user-42", "status": "failed", "failure": "image pull failed" }), ); - let err = rig.run(&StartInstanceInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.failure()?; + let err = rig.run(&StartInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.failure()?; assert!(err.contains("image pull failed"), "{err}"); Ok(()) } @@ -78,7 +107,7 @@ async fn stopped_while_waiting(rig: FakeRig) -> WeftResult<()> { rig.answer_program_call("weft.infra.status", json!({ "node": "bridge", "instance": "user-42", "status": "provisioning" })); rig.answer_program_call("weft.infra.status", json!({ "node": "bridge", "instance": "user-42", "status": "stopped" })); rig.signal(json!({})); - let err = rig.run(&StartInstanceInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.failure()?; + let err = rig.run(&StartInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.failure()?; assert!(err.contains("took it down"), "{err}"); let starts = rig.program_calls().iter().filter(|(call, _)| matches!(call, ProgramCall::InfraStart { .. })).count(); assert_eq!(starts, 1, "a stop while waiting is never undone by a second start"); @@ -87,7 +116,7 @@ async fn stopped_while_waiting(rig: FakeRig) -> WeftResult<()> { async fn no_copy(rig: FakeRig) -> WeftResult<()> { rig.answer_program_call("weft.infra.status", json!(null)); - let err = rig.run(&StartInstanceInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.failure()?; + let err = rig.run(&StartInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.failure()?; assert!(err.contains("did not come up"), "{err}"); Ok(()) } @@ -97,7 +126,7 @@ async fn refused_for_now(rig: FakeRig) -> WeftResult<()> { rig.answer_program_call("weft.infra.start", json!({ "answer": "started" })); rig.answer_program_call("weft.infra.status", running()); rig.signal(json!({})); - let outcome = rig.run(&StartInstanceInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.ok()?; + let outcome = rig.run(&StartInfraNode, json!({ "node": "bridge", "instance": "user-42" })).await.ok()?; let starts = rig.program_calls().iter().filter(|(call, _)| matches!(call, ProgramCall::InfraStart { .. })).count(); assert_eq!(starts, 2, "asked again after the wait"); assert_eq!(rig.awaited_signals().len(), 1); @@ -107,7 +136,7 @@ async fn refused_for_now(rig: FakeRig) -> WeftResult<()> { async fn without_waiting(rig: FakeRig) -> WeftResult<()> { let outcome = rig - .run(&StartInstanceInfraNode, json!({ "node": "bridge", "instance": "user-42", "waitUntilRunning": false })) + .run(&StartInfraNode, json!({ "node": "bridge", "instance": "user-42", "waitUntilRunning": false })) .await .ok()?; let calls = rig.program_calls(); @@ -123,7 +152,7 @@ async fn without_waiting_refused_for_now(rig: FakeRig) -> WeftResult<()> { rig.answer_program_call("weft.infra.start", json!({ "answer": "already_starting" })); rig.signal(json!({})); let outcome = rig - .run(&StartInstanceInfraNode, json!({ "node": "bridge", "instance": "user-42", "waitUntilRunning": false })) + .run(&StartInfraNode, json!({ "node": "bridge", "instance": "user-42", "waitUntilRunning": false })) .await .ok()?; let calls = rig.program_calls(); diff --git a/catalog/instances/start_instance_infra/metadata.json b/catalog/instances/start_instance_infra/metadata.json deleted file mode 100644 index 36bc7283..00000000 --- a/catalog/instances/start_instance_infra/metadata.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "type": "StartInstanceInfra", - "label": "Start instance infra", - "description": "Bring up one instance's copy of a `@per_instance` infra node; `done` fires once it answers, so the instance's triggers reading it can be activated next. The run waits without holding a worker; a copy that fails to come up fails the node with the reason; an already-running copy is left as is. If you want to answer a caller fast (a route replying 202), turn `waitUntilRunning` off and reply after this node: it finishes as soon as weft accepts the start, and a status read right after shows the copy as `provisioning`.", - "tags": [ - "instances", - "infra", - "start" - ], - "icon": "Play", - "color": "#0ea5e9", - "inputs": [ - { - "name": "node", - "type": "String", - "required": true, - "description": "The `@per_instance` infra node, named as the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file)." - }, - { - "name": "instance", - "type": "String", - "required": true, - "description": "The instance id your program gives it (`user-42`)." - }, - { - "name": "waitUntilRunning", - "type": "Boolean", - "default": true, - "label": "Wait until running", - "description": "On: `done` fires once the copy runs, and a copy that fails to come up fails the node. Off: `done` fires once weft accepts the start (a start refused for a passing reason, such as a build in progress, is asked again until it is), and whether the copy then comes up is for a status read to tell." - } - ], - "outputs": [ - { - "name": "done", - "type": "Boolean", - "description": "True once the copy is running, or once weft accepted the start when `waitUntilRunning` is off." - } - ], - "validate": [ - { - "when": { - "kind": "not", - "of": { "kind": "input_names", "port": "node", "names": { "node": { "role": "infra", "per_instance": true } } } - }, - "then": { - "message": "StartInstanceInfra '{id}' names {names} in `node`, which is no infra node of this program marked `@per_instance`. Name one the way the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file).", - "level": "structural", - "severity": "error", - "port": "node" - } - } - ] -} diff --git a/catalog/instances/stop_infra/metadata.json b/catalog/instances/stop_infra/metadata.json new file mode 100644 index 00000000..37c6f115 --- /dev/null +++ b/catalog/instances/stop_infra/metadata.json @@ -0,0 +1,184 @@ +{ + "type": "StopInfra", + "label": "Stop infra", + "description": "Scale a copy of an infra node down, keeping its disk: the program's own copy, or with `instance` that instance's copy of a `@per_instance` node. The triggers reading the copy go down per `triggers`; the runs using it follow `running`.", + "tags": [ + "instances", + "infra", + "stop" + ], + "icon": "Pause", + "color": "#0ea5e9", + "inputs": [ + { + "name": "node", + "type": "String", + "required": true, + "description": "The infra node, named as the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file): a shared one when `instance` is unwired, one marked `@per_instance` when it is set." + }, + { + "name": "instance", + "type": "String", + "requiredWhenWired": true, + "description": "Whose copy: an instance id your program gives it (`user-42`), for that instance's copy of a `@per_instance` node. Unwired: the program's own copy. Wired, and no id arrives (the step feeding it did not run): this node skips." + }, + { + "name": "triggers", + "type": "String", + "widget": { + "kind": "select", + "options": [ + "park", + "hibernate", + "wipe" + ] + }, + "default": "park", + "label": "Triggers", + "description": "The triggers reading the copy: park keeps everything that arrives (answers, events, the trigger's own fires) and runs it once the trigger is back, while a caller on a route is told to try again in a moment; hibernate does the same for `graceMinutes` with its questions hidden, then takes nothing more; wipe drops it all. Starting the copy again brings back the ones it took down." + }, + { + "name": "running", + "type": "String", + "widget": { + "kind": "select", + "options": [ + "wait", + "cancel" + ] + }, + "default": "wait", + "label": "Running work", + "description": "The runs still going that use the copy: wait lets them finish, cancel stops them. `wipe` needs cancel." + }, + { + "name": "graceMinutes", + "type": "Number", + "default": 15, + "label": "Grace minutes", + "description": "Hibernate only: how long it keeps accepting.", + "widget": { + "kind": "number", + "min": 0, + "step": 1 + } + }, + { + "name": "includeSelf", + "type": "Boolean", + "default": false, + "label": "Stop this run too", + "description": "On: this run is cancelled too when it is one of the runs reached." + } + ], + "outputs": [ + { + "name": "done", + "type": "Boolean", + "description": "True once the triggers reading the copy are down and the stop is handed off (or the copy was already down); the copy scales down afterwards." + }, + { + "name": "noCopy", + "type": "Boolean", + "description": "True instead of `done` when the copy does not exist: it was never started or was terminated before, or the instance id names no instance that has one. Wire it to tell whoever asked." + } + ], + "validate": [ + { + "when": { + "kind": "all", + "of": [ + { + "kind": "input_satisfied", + "port": "instance" + }, + { + "kind": "not", + "of": { + "kind": "input_names", + "port": "node", + "names": { + "node": { + "role": "infra", + "per_instance": true + } + } + } + } + ] + }, + "then": { + "message": "StopInfra '{id}' names {names} in `node`, which is no infra node of this program marked `@per_instance`. It is given an `instance`, so it acts on that instance's copy: name one the way the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file), or leave `instance` unwired to act on the program's own copy of a shared one.", + "level": "structural", + "severity": "error", + "port": "node" + } + }, + { + "when": { + "kind": "all", + "of": [ + { + "kind": "not", + "of": { + "kind": "input_satisfied", + "port": "instance" + } + }, + { + "kind": "not", + "of": { + "kind": "input_names", + "port": "node", + "names": { + "node": { + "role": "infra", + "per_instance": false + } + } + } + } + ] + }, + "then": { + "message": "StopInfra '{id}' names {names} in `node`, which is no shared infra node of this program. It has no `instance`, so it acts on the program's own copy: name one the way the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file), or wire `instance` to act on an instance's copy of a `@per_instance` one.", + "level": "structural", + "severity": "error", + "port": "node" + } + }, + { + "when": { + "kind": "all", + "of": [ + { + "kind": "config_equals", + "field": "triggers", + "equals": "wipe" + }, + { + "kind": "not", + "of": { + "kind": "config_equals", + "field": "running", + "equals": "cancel" + } + }, + { + "kind": "not", + "of": { + "kind": "input_wired", + "port": "running" + } + } + ] + }, + "then": { + "message": "StopInfra '{id}' wipes the triggers but lets running work finish. Wiping drops what the runs still need, so set `running` to cancel, or pick park or hibernate for `triggers`.", + "level": "structural", + "severity": "error", + "field": "running" + } + } + ] +} diff --git a/catalog/instances/stop_infra/mod.rs b/catalog/instances/stop_infra/mod.rs new file mode 100644 index 00000000..a0fb4a01 --- /dev/null +++ b/catalog/instances/stop_infra/mod.rs @@ -0,0 +1,34 @@ +//! StopInfra: scale a copy of an infra node down, the program's own or, +//! given `instance`, that instance's copy, keeping its disk, and take the +//! triggers reading it down the way the node's choices say. + +use async_trait::async_trait; + +use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; + +use super::lifecycle::{instance_if_given, stop_self, take_down_spec, taken_down}; + +#[derive(NodeManifest)] +pub struct StopInfraNode; + +#[cfg(feature = "node-tests")] +mod tests; + +#[async_trait] +impl Node for StopInfraNode { + #[cfg(feature = "node-tests")] + fn tests(&self) -> Vec { + tests::tests() + } + + async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { + let node: String = ctx.inputs.get("node")?; + let spec = take_down_spec(&ctx)?; + let mut copy = ctx.infra(node); + if let Some(id) = instance_if_given(&ctx)? { + copy = copy.instance(id); + } + let answer = copy.stop(spec, stop_self(&ctx)?).await?; + ctx.pulse_downstream(taken_down(answer)).await + } +} diff --git a/catalog/instances/stop_instance_infra/tests.rs b/catalog/instances/stop_infra/tests.rs similarity index 67% rename from catalog/instances/stop_instance_infra/tests.rs rename to catalog/instances/stop_infra/tests.rs index ec47a7b4..c8e9b479 100644 --- a/catalog/instances/stop_instance_infra/tests.rs +++ b/catalog/instances/stop_infra/tests.rs @@ -1,4 +1,6 @@ -//! StopInstanceInfra self-tests: the choices become the take-down spec, +//! StopInfra self-tests: the stop is asked for that instance's copy, or +//! the program's own when no instance is given, the choices become the +//! take-down spec, //! the self choice follows the toggle, and a wait before a wipe is //! refused before anything is asked. @@ -7,11 +9,12 @@ use serde_json::json; use weft::program::ProgramCall; use weft::{DeactivationMode, FakeRig, NodeTest, RunningPolicy, StopSelf, WeftResult}; -use super::StopInstanceInfraNode; +use super::StopInfraNode; pub fn tests() -> Vec { vec![ NodeTest::fake("parks_and_waits_by_default", defaults), + NodeTest::fake("no_instance_stops_the_programs_own_copy", the_programs_own), NodeTest::fake("the_choices_reach_the_call", choices), NodeTest::fake("waiting_before_a_wipe_is_refused", wipe_wait), NodeTest::fake("no_copy_is_told_apart_from_done", no_copy), @@ -21,7 +24,7 @@ pub fn tests() -> Vec { async fn no_copy(rig: FakeRig) -> WeftResult<()> { rig.answer_program_call("weft.infra.stop", json!({ "copy": "no_copy" })); - let out = rig.run(&StopInstanceInfraNode, json!({ "node": "bridge", "instance": "adaa" })).await.ok()?; + let out = rig.run(&StopInfraNode, json!({ "node": "bridge", "instance": "adaa" })).await.ok()?; assert_eq!(out.output("noCopy")?, &json!(true)); assert!(out.output("done").is_err(), "nothing was taken down"); Ok(()) @@ -29,14 +32,14 @@ async fn no_copy(rig: FakeRig) -> WeftResult<()> { async fn already_down(rig: FakeRig) -> WeftResult<()> { rig.answer_program_call("weft.infra.stop", json!({ "copy": "already_down" })); - let out = rig.run(&StopInstanceInfraNode, json!({ "node": "bridge", "instance": "ada" })).await.ok()?; + let out = rig.run(&StopInfraNode, json!({ "node": "bridge", "instance": "ada" })).await.ok()?; assert_eq!(out.output("done")?, &json!(true)); assert!(out.output("noCopy").is_err()); Ok(()) } async fn defaults(rig: FakeRig) -> WeftResult<()> { - rig.run(&StopInstanceInfraNode, json!({ "node": "bridge", "instance": "ada" })).await.ok()?; + rig.run(&StopInfraNode, json!({ "node": "bridge", "instance": "ada" })).await.ok()?; match &rig.program_calls()[0] { (ProgramCall::InfraStop { node, instance, spec }, StopSelf::Keep) => { assert_eq!(node, "bridge"); @@ -49,9 +52,22 @@ async fn defaults(rig: FakeRig) -> WeftResult<()> { Ok(()) } +async fn the_programs_own(rig: FakeRig) -> WeftResult<()> { + let out = rig.run(&StopInfraNode, json!({ "node": "pg" })).await.ok()?; + match &rig.program_calls()[0] { + (ProgramCall::InfraStop { node, instance, .. }, StopSelf::Keep) => { + assert_eq!(node, "pg"); + assert!(instance.is_none(), "the program's own copy"); + } + other => panic!("unexpected call {other:?}"), + } + assert_eq!(out.output("done")?, &json!(true)); + Ok(()) +} + async fn choices(rig: FakeRig) -> WeftResult<()> { rig.run( - &StopInstanceInfraNode, + &StopInfraNode, json!({ "node": "bridge", "instance": "ada", "triggers": "hibernate", "graceMinutes": 30, "running": "cancel", "includeSelf": true }), ) .await @@ -69,7 +85,7 @@ async fn choices(rig: FakeRig) -> WeftResult<()> { async fn wipe_wait(rig: FakeRig) -> WeftResult<()> { let err = rig - .run(&StopInstanceInfraNode, json!({ "node": "bridge", "instance": "ada", "triggers": "wipe", "running": "wait" })) + .run(&StopInfraNode, json!({ "node": "bridge", "instance": "ada", "triggers": "wipe", "running": "wait" })) .await .failure()?; assert!(err.contains("wipe requires"), "{err}"); diff --git a/catalog/instances/stop_instance_infra/metadata.json b/catalog/instances/stop_instance_infra/metadata.json deleted file mode 100644 index 400e24af..00000000 --- a/catalog/instances/stop_instance_infra/metadata.json +++ /dev/null @@ -1,142 +0,0 @@ -{ - "type": "StopInstanceInfra", - "label": "Stop instance infra", - "description": "Scale one instance's copy of a `@per_instance` infra node down, keeping its disk. The instance's triggers reading it go down per `triggers`; its runs follow `running`.", - "tags": [ - "instances", - "infra", - "stop" - ], - "icon": "Pause", - "color": "#0ea5e9", - "inputs": [ - { - "name": "node", - "type": "String", - "required": true, - "description": "The `@per_instance` infra node, named as the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file)." - }, - { - "name": "instance", - "type": "String", - "required": true, - "description": "The instance id your program gives it (`user-42`)." - }, - { - "name": "triggers", - "type": "String", - "widget": { - "kind": "select", - "options": [ - "park", - "hibernate", - "wipe" - ] - }, - "default": "park", - "label": "Triggers", - "description": "The instance's triggers reading it: park keeps everything that arrives (answers, events, the trigger's own fires) and runs it once the trigger is back, while a caller on a route is told to try again in a moment; hibernate does the same for `graceMinutes` with its questions hidden, then takes nothing more; wipe drops it all. Starting the copy again brings back the ones it took down." - }, - { - "name": "running", - "type": "String", - "widget": { - "kind": "select", - "options": [ - "wait", - "cancel" - ] - }, - "default": "wait", - "label": "Running work", - "description": "The instance's runs still going: wait lets them finish, cancel stops them. `wipe` needs cancel." - }, - { - "name": "graceMinutes", - "type": "Number", - "default": 15, - "label": "Grace minutes", - "description": "Hibernate only: how long it keeps accepting.", - "widget": { - "kind": "number", - "min": 0, - "step": 1 - } - }, - { - "name": "includeSelf", - "type": "Boolean", - "default": false, - "label": "Stop this run too", - "description": "On: this run is cancelled too when it is one of the runs reached." - } - ], - "outputs": [ - { - "name": "done", - "type": "Boolean", - "description": "True once the triggers reading the copy are down and the stop is handed off (or the copy was already down); the copy scales down afterwards." - }, - { - "name": "noCopy", - "type": "Boolean", - "description": "True instead of `done` when this instance has no copy of the node: the id names no instance that has one, or its copy was terminated before. Wire it to tell whoever asked." - } - ], - "validate": [ - { - "when": { - "kind": "not", - "of": { - "kind": "input_names", - "port": "node", - "names": { - "node": { - "role": "infra", - "per_instance": true - } - } - } - }, - "then": { - "message": "StopInstanceInfra '{id}' names {names} in `node`, which is no infra node of this program marked `@per_instance`. Name one the way the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file).", - "level": "structural", - "severity": "error", - "port": "node" - } - }, - { - "when": { - "kind": "all", - "of": [ - { - "kind": "config_equals", - "field": "triggers", - "equals": "wipe" - }, - { - "kind": "not", - "of": { - "kind": "config_equals", - "field": "running", - "equals": "cancel" - } - }, - { - "kind": "not", - "of": { - "kind": "input_wired", - "port": "running" - } - } - ] - }, - "then": { - "message": "StopInstanceInfra '{id}' wipes the triggers but lets running work finish. Wiping drops what the runs still need, so set `running` to cancel, or pick park or hibernate for `triggers`.", - "level": "structural", - "severity": "error", - "field": "running" - } - } - ] -} diff --git a/catalog/instances/stop_instance_infra/mod.rs b/catalog/instances/stop_instance_infra/mod.rs deleted file mode 100644 index b3a9c6f4..00000000 --- a/catalog/instances/stop_instance_infra/mod.rs +++ /dev/null @@ -1,30 +0,0 @@ -//! StopInstanceInfra: scale one instance's copy of an infra node down, -//! keeping its disk, taking the instance's triggers reading it down the -//! way the node's choices say. - -use async_trait::async_trait; - -use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; - -use super::lifecycle::{instance, stop_self, take_down_spec, taken_down}; - -#[derive(NodeManifest)] -pub struct StopInstanceInfraNode; - -#[cfg(feature = "node-tests")] -mod tests; - -#[async_trait] -impl Node for StopInstanceInfraNode { - #[cfg(feature = "node-tests")] - fn tests(&self) -> Vec { - tests::tests() - } - - async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { - let node: String = ctx.inputs.get("node")?; - let spec = take_down_spec(&ctx)?; - let answer = ctx.infra(node).instance(instance(&ctx)?).stop(spec, stop_self(&ctx)?).await?; - ctx.pulse_downstream(taken_down(answer)).await - } -} diff --git a/catalog/instances/terminate_infra/metadata.json b/catalog/instances/terminate_infra/metadata.json new file mode 100644 index 00000000..c8485141 --- /dev/null +++ b/catalog/instances/terminate_infra/metadata.json @@ -0,0 +1,185 @@ +{ + "type": "TerminateInfra", + "label": "Terminate infra", + "description": "Delete a copy of an infra node and its disk: the program's own copy, or with `instance` that instance's copy of a `@per_instance` node. The triggers reading the copy go down per `triggers`; the runs using it follow `running`.", + "tags": [ + "instances", + "infra", + "terminate", + "delete" + ], + "icon": "Trash2", + "color": "#0ea5e9", + "inputs": [ + { + "name": "node", + "type": "String", + "required": true, + "description": "The infra node, named as the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file): a shared one when `instance` is unwired, one marked `@per_instance` when it is set." + }, + { + "name": "instance", + "type": "String", + "requiredWhenWired": true, + "description": "Whose copy: an instance id your program gives it (`user-42`), for that instance's copy of a `@per_instance` node. Unwired: the program's own copy. Wired, and no id arrives (the step feeding it did not run): this node skips." + }, + { + "name": "triggers", + "type": "String", + "widget": { + "kind": "select", + "options": [ + "park", + "hibernate", + "wipe" + ] + }, + "default": "park", + "label": "Triggers", + "description": "The triggers reading the copy: park keeps everything that arrives (answers, events, the trigger's own fires) and runs it once the trigger is back, while a caller on a route is told to try again in a moment; hibernate does the same for `graceMinutes` with its questions hidden, then takes nothing more; wipe drops it all." + }, + { + "name": "running", + "type": "String", + "widget": { + "kind": "select", + "options": [ + "wait", + "cancel" + ] + }, + "default": "wait", + "label": "Running work", + "description": "The runs still going that use the copy: wait lets them finish, cancel stops them. `wipe` needs cancel." + }, + { + "name": "graceMinutes", + "type": "Number", + "default": 15, + "label": "Grace minutes", + "description": "Hibernate only: how long it keeps accepting.", + "widget": { + "kind": "number", + "min": 0, + "step": 1 + } + }, + { + "name": "includeSelf", + "type": "Boolean", + "default": false, + "label": "Stop this run too", + "description": "On: this run is cancelled too when it is one of the runs reached." + } + ], + "outputs": [ + { + "name": "done", + "type": "Boolean", + "description": "True once the triggers reading the copy are down and the delete is handed off (or the copy was already going); the copy is deleted afterwards." + }, + { + "name": "noCopy", + "type": "Boolean", + "description": "True instead of `done` when the copy does not exist: it was never started or was terminated before, or the instance id names no instance that has one. Wire it to tell whoever asked." + } + ], + "validate": [ + { + "when": { + "kind": "all", + "of": [ + { + "kind": "input_satisfied", + "port": "instance" + }, + { + "kind": "not", + "of": { + "kind": "input_names", + "port": "node", + "names": { + "node": { + "role": "infra", + "per_instance": true + } + } + } + } + ] + }, + "then": { + "message": "TerminateInfra '{id}' names {names} in `node`, which is no infra node of this program marked `@per_instance`. It is given an `instance`, so it acts on that instance's copy: name one the way the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file), or leave `instance` unwired to act on the program's own copy of a shared one.", + "level": "structural", + "severity": "error", + "port": "node" + } + }, + { + "when": { + "kind": "all", + "of": [ + { + "kind": "not", + "of": { + "kind": "input_satisfied", + "port": "instance" + } + }, + { + "kind": "not", + "of": { + "kind": "input_names", + "port": "node", + "names": { + "node": { + "role": "infra", + "per_instance": false + } + } + } + } + ] + }, + "then": { + "message": "TerminateInfra '{id}' names {names} in `node`, which is no shared infra node of this program. It has no `instance`, so it acts on the program's own copy: name one the way the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file), or wire `instance` to act on an instance's copy of a `@per_instance` one.", + "level": "structural", + "severity": "error", + "port": "node" + } + }, + { + "when": { + "kind": "all", + "of": [ + { + "kind": "config_equals", + "field": "triggers", + "equals": "wipe" + }, + { + "kind": "not", + "of": { + "kind": "config_equals", + "field": "running", + "equals": "cancel" + } + }, + { + "kind": "not", + "of": { + "kind": "input_wired", + "port": "running" + } + } + ] + }, + "then": { + "message": "TerminateInfra '{id}' wipes the triggers but lets running work finish. Wiping drops what the runs still need, so set `running` to cancel, or pick park or hibernate for `triggers`.", + "level": "structural", + "severity": "error", + "field": "running" + } + } + ] +} diff --git a/catalog/instances/terminate_infra/mod.rs b/catalog/instances/terminate_infra/mod.rs new file mode 100644 index 00000000..c434d6b8 --- /dev/null +++ b/catalog/instances/terminate_infra/mod.rs @@ -0,0 +1,34 @@ +//! TerminateInfra: delete a copy of an infra node and its disk, the +//! program's own or, given `instance`, that instance's copy, and take the +//! triggers reading it down the way the node's choices say. + +use async_trait::async_trait; + +use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; + +use super::lifecycle::{instance_if_given, stop_self, take_down_spec, taken_down}; + +#[derive(NodeManifest)] +pub struct TerminateInfraNode; + +#[cfg(feature = "node-tests")] +mod tests; + +#[async_trait] +impl Node for TerminateInfraNode { + #[cfg(feature = "node-tests")] + fn tests(&self) -> Vec { + tests::tests() + } + + async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { + let node: String = ctx.inputs.get("node")?; + let spec = take_down_spec(&ctx)?; + let mut copy = ctx.infra(node); + if let Some(id) = instance_if_given(&ctx)? { + copy = copy.instance(id); + } + let answer = copy.terminate(spec, stop_self(&ctx)?).await?; + ctx.pulse_downstream(taken_down(answer)).await + } +} diff --git a/catalog/instances/terminate_instance_infra/tests.rs b/catalog/instances/terminate_infra/tests.rs similarity index 51% rename from catalog/instances/terminate_instance_infra/tests.rs rename to catalog/instances/terminate_infra/tests.rs index 7bac6a89..cca9ec98 100644 --- a/catalog/instances/terminate_instance_infra/tests.rs +++ b/catalog/instances/terminate_infra/tests.rs @@ -1,20 +1,24 @@ -//! TerminateInstanceInfra self-tests: the terminate is asked for that -//! instance's copy with the node's choices. +//! TerminateInfra self-tests: the terminate is asked for that +//! instance's copy with the node's choices, or for the program's own copy +//! when no instance is given. use serde_json::json; use weft::program::ProgramCall; use weft::{DeactivationMode, FakeRig, NodeTest, RunningPolicy, StopSelf, WeftResult}; -use super::TerminateInstanceInfraNode; +use super::TerminateInfraNode; pub fn tests() -> Vec { - vec![NodeTest::fake("terminates_the_instances_copy", terminates)] + vec![ + NodeTest::fake("terminates_the_instances_copy", terminates), + NodeTest::fake("no_instance_terminates_the_programs_own_copy", the_programs_own), + ] } async fn terminates(rig: FakeRig) -> WeftResult<()> { rig.run( - &TerminateInstanceInfraNode, + &TerminateInfraNode, json!({ "node": "one.bridge", "instance": "ada", "triggers": "wipe", "running": "cancel" }), ) .await @@ -31,3 +35,15 @@ async fn terminates(rig: FakeRig) -> WeftResult<()> { } Ok(()) } + +async fn the_programs_own(rig: FakeRig) -> WeftResult<()> { + rig.run(&TerminateInfraNode, json!({ "node": "pg" })).await.ok()?; + match &rig.program_calls()[0] { + (ProgramCall::InfraTerminate { node, instance, .. }, StopSelf::Keep) => { + assert_eq!(node, "pg"); + assert!(instance.is_none(), "the program's own copy"); + } + other => panic!("unexpected call {other:?}"), + } + Ok(()) +} diff --git a/catalog/instances/terminate_instance_infra/metadata.json b/catalog/instances/terminate_instance_infra/metadata.json deleted file mode 100644 index 841e3937..00000000 --- a/catalog/instances/terminate_instance_infra/metadata.json +++ /dev/null @@ -1,143 +0,0 @@ -{ - "type": "TerminateInstanceInfra", - "label": "Terminate instance infra", - "description": "Delete one instance's copy of a `@per_instance` infra node and its disk. The instance's triggers reading it go down per `triggers`; its runs follow `running`.", - "tags": [ - "instances", - "infra", - "terminate", - "delete" - ], - "icon": "Trash2", - "color": "#0ea5e9", - "inputs": [ - { - "name": "node", - "type": "String", - "required": true, - "description": "The `@per_instance` infra node, named as the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file)." - }, - { - "name": "instance", - "type": "String", - "required": true, - "description": "The instance id your program gives it (`user-42`)." - }, - { - "name": "triggers", - "type": "String", - "widget": { - "kind": "select", - "options": [ - "park", - "hibernate", - "wipe" - ] - }, - "default": "park", - "label": "Triggers", - "description": "The instance's triggers reading it: park keeps everything that arrives (answers, events, the trigger's own fires) and runs it once the trigger is back, while a caller on a route is told to try again in a moment; hibernate does the same for `graceMinutes` with its questions hidden, then takes nothing more; wipe drops it all." - }, - { - "name": "running", - "type": "String", - "widget": { - "kind": "select", - "options": [ - "wait", - "cancel" - ] - }, - "default": "wait", - "label": "Running work", - "description": "The instance's runs still going: wait lets them finish, cancel stops them. `wipe` needs cancel." - }, - { - "name": "graceMinutes", - "type": "Number", - "default": 15, - "label": "Grace minutes", - "description": "Hibernate only: how long it keeps accepting.", - "widget": { - "kind": "number", - "min": 0, - "step": 1 - } - }, - { - "name": "includeSelf", - "type": "Boolean", - "default": false, - "label": "Stop this run too", - "description": "On: this run is cancelled too when it is one of the runs reached." - } - ], - "outputs": [ - { - "name": "done", - "type": "Boolean", - "description": "True once the triggers reading the copy are down and the delete is handed off (or the copy was already going); the copy is deleted afterwards." - }, - { - "name": "noCopy", - "type": "Boolean", - "description": "True instead of `done` when this instance has no copy of the node: the id names no instance that has one, or its copy was terminated before. Wire it to tell whoever asked." - } - ], - "validate": [ - { - "when": { - "kind": "not", - "of": { - "kind": "input_names", - "port": "node", - "names": { - "node": { - "role": "infra", - "per_instance": true - } - } - } - }, - "then": { - "message": "TerminateInstanceInfra '{id}' names {names} in `node`, which is no infra node of this program marked `@per_instance`. Name one the way the program writes it (`bridge`, `work.bridge` inside a group, or `one.bridge` inside an included file).", - "level": "structural", - "severity": "error", - "port": "node" - } - }, - { - "when": { - "kind": "all", - "of": [ - { - "kind": "config_equals", - "field": "triggers", - "equals": "wipe" - }, - { - "kind": "not", - "of": { - "kind": "config_equals", - "field": "running", - "equals": "cancel" - } - }, - { - "kind": "not", - "of": { - "kind": "input_wired", - "port": "running" - } - } - ] - }, - "then": { - "message": "TerminateInstanceInfra '{id}' wipes the triggers but lets running work finish. Wiping drops what the runs still need, so set `running` to cancel, or pick park or hibernate for `triggers`.", - "level": "structural", - "severity": "error", - "field": "running" - } - } - ] -} diff --git a/catalog/instances/terminate_instance_infra/mod.rs b/catalog/instances/terminate_instance_infra/mod.rs deleted file mode 100644 index 2c956855..00000000 --- a/catalog/instances/terminate_instance_infra/mod.rs +++ /dev/null @@ -1,30 +0,0 @@ -//! TerminateInstanceInfra: delete one instance's copy of an infra node -//! and its disk, taking the instance's triggers reading it down the way -//! the node's choices say. - -use async_trait::async_trait; - -use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; - -use super::lifecycle::{instance, stop_self, take_down_spec, taken_down}; - -#[derive(NodeManifest)] -pub struct TerminateInstanceInfraNode; - -#[cfg(feature = "node-tests")] -mod tests; - -#[async_trait] -impl Node for TerminateInstanceInfraNode { - #[cfg(feature = "node-tests")] - fn tests(&self) -> Vec { - tests::tests() - } - - async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { - let node: String = ctx.inputs.get("node")?; - let spec = take_down_spec(&ctx)?; - let answer = ctx.infra(node).instance(instance(&ctx)?).terminate(spec, stop_self(&ctx)?).await?; - ctx.pulse_downstream(taken_down(answer)).await - } -} diff --git a/crates/weft-access-store/src/flows.rs b/crates/weft-access-store/src/flows.rs index 4cd0c4d8..4024b468 100644 --- a/crates/weft-access-store/src/flows.rs +++ b/crates/weft-access-store/src/flows.rs @@ -987,6 +987,10 @@ pub async fn begin_oauth( }; let client_id = registration.client_id.clone(); + // The ticked permissions, with what the service asks for always (what + // its test call reads the account by): a grant is checked against + // the whole of it once it lands. + let requested = spec.requested_permissions(&req.permissions); let state = uuid::Uuid::new_v4().simple().to_string(); let verifier = pkce.then(|| { format!("{}{}", uuid::Uuid::new_v4().simple(), uuid::Uuid::new_v4().simple()) @@ -1001,9 +1005,9 @@ pub async fn begin_oauth( q.append_pair("client_id", &client_id); q.append_pair("redirect_uri", redirect_uri); q.append_pair("state", &state); - if !req.permissions.is_empty() { + if !requested.is_empty() { let delim = scope_delimiter.as_deref().unwrap_or(" "); - q.append_pair("scope", &req.permissions.join(delim)); + q.append_pair("scope", &requested.join(delim)); } for (k, v) in auth_params { q.append_pair(k, v); @@ -1028,7 +1032,7 @@ pub async fn begin_oauth( .bind(crate::seal_json(&serde_json::to_value(registration)?)?) .bind(req.project_id) .bind(serde_json::to_value(spec)?) - .bind(serde_json::to_value(&req.permissions)?) + .bind(serde_json::to_value(&requested)?) .bind(verifier.as_deref().map(crate::seal_str)) .bind(door_str(req.door)) .bind(req.upgrade_grant_id) diff --git a/crates/weft-access-store/tests/store_flows.rs b/crates/weft-access-store/tests/store_flows.rs index b159b2af..0282b289 100644 --- a/crates/weft-access-store/tests/store_flows.rs +++ b/crates/weft-access-store/tests/store_flows.rs @@ -442,6 +442,27 @@ async fn a_poll_on_a_state_with_nothing_live_answers_gone(pool: PgPool) { assert!(take_connect_result(&pool, TENANT_A, &started.state).await.unwrap().is_none()); } +/// A permission the service marks `always` rides every consent on top +/// of what was ticked, is recorded with it, and is held to the +/// provider's echo like a ticked one. +#[sqlx::test] +async fn a_consent_asks_for_what_the_service_always_needs(pool: PgPool) { + weft_task_store::apply_groups(&pool, &[&weft_access_store::GROUP]).await.unwrap(); + let fake = FakeProvider::new(); + let base = fake.serve().await; + let mut spec = oauth_spec(&base, "coexisting"); + spec.permissions.insert( + 0, + serde_json::from_value(json!({ "id": "whoami", "label": "Who", "description": "Know the account.", "always": true })).unwrap(), + ); + let grant = full_consent(&pool, TENANT_A, &spec, &["write"], None).await.unwrap(); + assert_eq!(grant.scopes, vec!["whoami", "write"], "asked for, and recorded, with the ticked one"); + + *fake.scope_echo.lock().unwrap() = Some("write".into()); + let err = full_consent(&pool, TENANT_A, &spec, &["write"], None).await.unwrap_err(); + assert!(err.to_string().contains("missing 'whoami'"), "{err}"); +} + #[sqlx::test] async fn oauth_consent_records_granted_scopes_and_enforces_the_echo(pool: PgPool) { weft_task_store::apply_groups(&pool, &[&weft_access_store::GROUP]).await.unwrap(); diff --git a/crates/weft-broker-client/src/protocol.rs b/crates/weft-broker-client/src/protocol.rs index 0bcc69f3..0b95fc34 100644 --- a/crates/weft-broker-client/src/protocol.rs +++ b/crates/weft-broker-client/src/protocol.rs @@ -182,11 +182,12 @@ pub struct RunAnswersResponse { pub answers: Vec, } -/// One answer waiting for a run: the wait it resolves, and the value. +/// One answer waiting for a run: the wait it resolves, and the answer (a +/// value, or a person's skip). #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct RunAnswer { pub token: String, - pub value: Value, + pub answer: weft_core::primitive::WaitAnswer, } /// `POST /v1/run/cancels`: the cancels waiting for the runs the calling @@ -2218,10 +2219,11 @@ pub enum LetGo { #[derive(Debug, Clone, Serialize, Deserialize)] pub struct DoorLetGo {} -/// `POST /v1/door/tick`: once a second, a worker says it is alive (its -/// `worker_lease`) and how many runs it drives per trigger, states what it -/// counted at its door this minute, and hears what the project's other -/// copies counted. +/// `POST /v1/door/tick`: once a second while it has work under way, a +/// worker says it is alive (its `worker_lease`) and how many runs it +/// drives per trigger, states what it counted at its door this minute, and +/// hears what the project's other copies counted. With nothing under way +/// it sends none, and its lease lapses. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct DoorTickRequest { /// The image it runs (the program's binary hash): a drain of the runs diff --git a/crates/weft-broker/src/handlers.rs b/crates/weft-broker/src/handlers.rs index fb2163db..c21dfcda 100644 --- a/crates/weft-broker/src/handlers.rs +++ b/crates/weft-broker/src/handlers.rs @@ -161,7 +161,7 @@ pub async fn execution_stop_tagged( tenant_id: execution_id_scope.tenant, payload: serde_json::to_value(&payload).map_err(internal)?, }; - state.tasks.enqueue_dedup(task).await.map_err(internal)?; + weft_task_store::tasks::enqueue(&state.pool, task).await.map_err(internal)?; Ok(Json(ExecutionStopTaggedResponse { stops_asker })) } @@ -2293,7 +2293,7 @@ pub(crate) fn unavailable_or_internal(e: anyhow::Error) -> (StatusCode, String) tracing::warn!(target: "weft_broker", "no connection to the database could be had: {e:#}"); return (StatusCode::SERVICE_UNAVAILABLE, NOT_DONE.into()); } - if causes().any(database_unreachable) { + if causes().any(weft_task_store::db::unreachable) { tracing::warn!(target: "weft_broker", "could not reach the database: {e:#}"); return (StatusCode::SERVICE_UNAVAILABLE, "the database connection dropped, or the database is unavailable; ask again".into()); } @@ -2306,23 +2306,6 @@ fn no_connection(e: &sqlx::Error) -> bool { matches!(e, sqlx::Error::PoolTimedOut | sqlx::Error::PoolClosed) } -/// Whether a database error means the server could not be reached or -/// is restarting, rather than that the statement itself failed: no -/// connection to be had, a connection that broke, or the server -/// refusing because it is shutting down or starting up (SQLSTATE class -/// 08, and 57P01 to 57P03). -fn database_unreachable(e: &sqlx::Error) -> bool { - match e { - sqlx::Error::PoolTimedOut | sqlx::Error::PoolClosed | sqlx::Error::Io(_) | sqlx::Error::Protocol(_) => { - true - } - sqlx::Error::Database(db) => db - .code() - .is_some_and(|code| code.starts_with("08") || matches!(&*code, "57P01" | "57P02" | "57P03")), - _ => false, - } -} - #[cfg(test)] mod tests { use super::*; diff --git a/crates/weft-broker/src/records.rs b/crates/weft-broker/src/records.rs index 0d1e21ea..2b183b4f 100644 --- a/crates/weft-broker/src/records.rs +++ b/crates/weft-broker/src/records.rs @@ -122,7 +122,7 @@ async fn run_answers(State(state): State>, AuthedCaller(caller) if !answers.is_empty() || !heard.woken_before(deadline, parked).await.map_err(|e| unavailable_or_internal(e.context("wait for a run's answers")))? { - return Ok(Json(RunAnswersResponse { answers: answers.into_iter().map(|(token, value)| RunAnswer { token, value }).collect() })); + return Ok(Json(RunAnswersResponse { answers: answers.into_iter().map(|(token, answer)| RunAnswer { token, answer }).collect() })); } } } diff --git a/crates/weft-broker/tests/door_let_go.rs b/crates/weft-broker/tests/door_let_go.rs index e2e7ee22..bf53be11 100644 --- a/crates/weft-broker/tests/door_let_go.rs +++ b/crates/weft-broker/tests/door_let_go.rs @@ -139,7 +139,7 @@ async fn an_answer_handed_to_a_run_that_parks_reaches_its_record(pool: PgPool) { schema(&pool).await; let execution_id = driven_by(&pool, "w1").await; let mut tx = pool.begin().await.unwrap(); - weft_task_store::parked_fires::hand_answer_in(&mut tx, "form", execution_id, &serde_json::json!("yes")).await.unwrap(); + weft_task_store::parked_fires::hand_answer_in(&mut tx, "form", execution_id, &weft_core::primitive::WaitAnswer::Given { value: serde_json::json!("yes") }).await.unwrap(); tx.commit().await.unwrap(); assert!(let_go(&pool, worker("w1"), execution_id, LetGo::Parked).await.unwrap()); assert_eq!(standing(&pool, execution_id).await, ("queued".into(), None)); diff --git a/crates/weft-cli/Cargo.toml b/crates/weft-cli/Cargo.toml index 68533b31..c19c115f 100644 --- a/crates/weft-cli/Cargo.toml +++ b/crates/weft-cli/Cargo.toml @@ -20,7 +20,8 @@ weft-compiler = { workspace = true } weft-catalog = { workspace = true } uuid = { workspace = true } futures = { workspace = true } -eventsource-client = "0.13" +# Reads the live updates (SSE) off an answer of the one HTTP client. +eventsource-stream = "0.2" tokio = { workspace = true } reqwest = { workspace = true } serde = { workspace = true } diff --git a/crates/weft-cli/src/client.rs b/crates/weft-cli/src/client.rs index fc637b65..07d3dd74 100644 --- a/crates/weft-cli/src/client.rs +++ b/crates/weft-cli/src/client.rs @@ -5,23 +5,172 @@ use std::sync::{Arc, RwLock}; use anyhow::Context; use weft_core::net::EmptyBody; -/// In an error's chain when the store refused because the upload is being -/// completed by another caller right now -/// ([`weft_core::storage::COMPLETING_HEADER`]): the file is about to land. +/// In an error's chain when the install answered with a failing status: +/// it was reached, and it (or something in front of it) said no. Reads +/// as the reason the answer gave (see `failure_text`). #[derive(Debug)] -pub struct StoreCompleting; +pub struct Refused { + status: reqwest::StatusCode, + message: String, + /// What the refusal's marker header says, when it carries one. + marked: Option, +} + +/// What a refusal's marker header says about why the install said no, +/// beyond its status. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Marked { + /// The upload is being completed by another caller right now + /// ([`weft_core::storage::COMPLETING_HEADER`]): the file is about to + /// land. + StoreCompleting, + /// Another ask of the project is starting its image builds right now + /// ([`weft_core::builds::BUILD_BUSY_HEADER`]) and lets go within + /// moments. + BuildBusy, +} + +impl Refused { + pub fn marked(&self) -> Option { + self.marked + } + + /// The status the answer carried. + pub fn status(&self) -> reqwest::StatusCode { + self.status + } +} + +impl std::fmt::Display for Refused { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(&self.message) + } +} + +impl std::error::Error for Refused {} + +/// The refusal in `e`'s chain, when the install answered one. +pub fn refusal(e: &anyhow::Error) -> Option<&Refused> { + e.chain().find_map(|cause| cause.downcast_ref::()) +} + +/// Whether `e` is a request that failed on its way rather than being +/// answered: its connection could not be made ([`InstallUnreachable`]), +/// or broke before the answer was whole. Nothing said no, so the same +/// request may get through later; whether sending it again is safe is the +/// caller's call. A status, however bad, is an answer ([`Refused`]), and so +/// is a TLS failure (a certificate refused, a handshake that is not TLS): +/// the address or the setup is wrong, and waiting will not change it. +/// The one reading of a failed connection, for requests to the install and +/// to the store alike. +pub fn connection_failed(e: &anyhow::Error) -> bool { + e.chain().filter_map(|cause| cause.downcast_ref::()).any(failed_on_the_way) +} + +/// [`connection_failed`] for one request's error. +fn failed_on_the_way(cause: &reqwest::Error) -> bool { + cause.status().is_none() + && !tls_failure(cause) + && (cause.is_connect() || cause.is_timeout() || cause.is_request() || cause.is_body() || cause.is_decode()) +} + +/// Whether the connection was never made: refused, or not made in time. +fn no_connection(cause: &reqwest::Error) -> bool { + !tls_failure(cause) && (cause.is_connect() || cause.is_timeout()) +} + +/// Whether `cause` is TLS refusing the other end, which is how rustls +/// reports a refused certificate or a bad handshake. +fn tls_failure(cause: &reqwest::Error) -> bool { + let mut source = std::error::Error::source(cause); + while let Some(err) = source { + if err.downcast_ref::().is_some_and(|io| io.kind() == std::io::ErrorKind::InvalidData) { + return true; + } + source = err.source(); + } + false +} + +/// In an error's chain when a request could not reach the install: the +/// connection could not be made, or was not made in time. Nothing was +/// refused, so what the person can do depends on where the install is +/// (`crate::progress::error_detail`): start the daemon when it is this +/// machine's, check the network when it is a remote one. +#[derive(Debug)] +pub struct InstallUnreachable { + /// The install's address. + pub url: String, + /// The target that named it (`--on `), when one did. + pub target: Option, + /// Why the request did not get through. + pub cause: reqwest::Error, +} + +impl InstallUnreachable { + /// Whether the install is this machine's: its address is a loopback + /// one, which only the local daemon answers at. + pub fn is_local(&self) -> bool { + is_loopback_address(&self.url) + } + + /// How the install is named to a person: its target and address, or + /// its address alone. + pub fn named(&self) -> String { + match &self.target { + Some(target) => format!("the install `{target}` ({})", self.url), + None => format!("the install at {}", self.url), + } + } +} + +/// Whether `url` names this machine (a loopback host). +fn is_loopback_address(url: &str) -> bool { + let host = reqwest::Url::parse(url).ok().and_then(|url| url.host().map(|host| host.to_owned())); + match host { + Some(url::Host::Domain(name)) => name.eq_ignore_ascii_case("localhost"), + Some(url::Host::Ipv4(ip)) => ip.is_loopback(), + Some(url::Host::Ipv6(ip)) => ip.is_loopback(), + None => false, + } +} -impl std::fmt::Display for StoreCompleting { +impl std::fmt::Display for InstallUnreachable { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.write_str("the upload is being completed by another caller") + write!(f, "{} did not answer", self.named()) } } -impl std::error::Error for StoreCompleting {} +impl std::error::Error for InstallUnreachable { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(&self.cause) + } +} + +/// How long making a connection to the install may take before the +/// request fails as unreachable. An install answers a connection in well +/// under a second on this machine and in a few seconds on a cloud that +/// is waking up; a path that is down never does, and without a limit +/// the request would hang on it for minutes. +const CONNECT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(15); + +/// How long a connection may sit with nothing on it before the system +/// checks that the other end is still there, and how often after that. +/// A request held open on a quiet connection (a verb waiting on the +/// install's answer) keeps its path alive this way, and one whose path +/// died fails within a minute or so instead of waiting for ever. There is +/// no limit on a request's whole length: some verbs wait as long as the +/// person's own work runs. +const KEEPALIVE_IDLE: std::time::Duration = std::time::Duration::from_secs(30); +const KEEPALIVE_INTERVAL: std::time::Duration = std::time::Duration::from_secs(10); +const KEEPALIVE_RETRIES: u32 = 3; #[derive(Clone)] pub struct DispatcherClient { base: String, + /// The target this install was named by (`--on `), for an + /// error that has to say which install did not answer. + target: Option, /// The bearer every request carries: the operator key, when this /// person holds one for the install (see `crate::credentials`; the /// local install needs none), or an instance token (`with_bearer`). @@ -33,7 +182,23 @@ pub struct DispatcherClient { impl DispatcherClient { pub fn new(base: impl Into, operator_key: Option) -> Self { - Self { base: base.into(), bearer: operator_key.map(|key| Arc::new(RwLock::new(key))), http: reqwest::Client::new() } + // The one connection setup every request to the install uses, + // live-update streams included. + let http = reqwest::Client::builder() + .connect_timeout(CONNECT_TIMEOUT) + .tcp_keepalive(KEEPALIVE_IDLE) + .tcp_keepalive_interval(KEEPALIVE_INTERVAL) + .tcp_keepalive_retries(KEEPALIVE_RETRIES) + .build() + .expect("an HTTP client with only timeouts set always builds"); + Self { base: base.into(), target: None, bearer: operator_key.map(|key| Arc::new(RwLock::new(key))), http } + } + + /// The same client, naming the install by `target` when it does not + /// answer. + pub fn for_target(mut self, target: Option<&str>) -> Self { + self.target = target.map(str::to_string); + self } /// The same dispatcher, with every request carrying `token` as its @@ -41,7 +206,7 @@ impl DispatcherClient { /// that answers to a token rather than to the operator (the /// instance door). pub fn with_bearer(&self, token: &str) -> Self { - Self::new(self.base.clone(), Some(token.to_string())) + Self { bearer: Some(Arc::new(RwLock::new(token.to_string()))), ..self.clone() } } /// Swap the bearer this client and every clone of it carry from the @@ -64,20 +229,28 @@ impl DispatcherClient { &self.base } - /// A live-updates (SSE) stream at `path` on this install, carrying - /// the same bearer as every other request. The SSE library owns its - /// own connection, so this is the one door out of this client that - /// is not `send`, and it still cannot leave without the key. - pub fn event_stream(&self, path: &str) -> anyhow::Result { - let url = format!("{}{}", self.base, path); - let mut builder = eventsource_client::ClientBuilder::for_url(&url).context("build sse client")?; - if let Some(commit) = CLI_COMMIT { - builder = builder.header(weft_core::install::CLI_COMMIT_HEADER, commit).context("build sse client")?; - } - match self.current_bearer()? { - Some(key) => builder.header("Authorization", &format!("Bearer {key}")).context("build sse client"), - None => Ok(builder), - } + /// The live updates (SSE) at `path` on this install, each event as it + /// comes. Opened like every other request (`send`, `check`), so it + /// carries the key, uses the same connection setup, and a refusal or + /// an install out of reach reads the same as anywhere else. Once open, + /// the install was reached: a stream that breaks afterwards is never + /// "unreachable", since the install itself ends a subscriber that fell + /// behind by cutting the stream, which reads the same as a dropped + /// connection. + pub async fn event_stream( + &self, + path: &str, + ) -> anyhow::Result> + use<>> { + use eventsource_stream::{EventStreamError, Eventsource}; + use futures::StreamExt; + let resp = self.check(self.send(reqwest::Method::GET, path, None).await?).await?; + let url = resp.url().clone(); + Ok(resp.bytes_stream().eventsource().map(move |event| { + event.map_err(|e| match e { + EventStreamError::Transport(cause) => anyhow::Error::new(cause).context(format!("live updates from {url} stopped")), + other => anyhow::Error::new(other).context(format!("read live updates from {url}")), + }) + })) } /// Every request to the install goes out here, so none can leave @@ -102,11 +275,39 @@ impl DispatcherClient { None if [reqwest::Method::POST, reqwest::Method::PUT, reqwest::Method::PATCH].contains(&method) => builder.empty_body(), None => builder, }; - let resp = builder.send().await.with_context(|| format!("{method} {url}"))?; + let resp = builder.send().await.map_err(|cause| self.failed(cause, format!("{method} {url}")))?; warn_once_on_version_note(&resp); Ok(resp) } + /// The error a request or the reading of its answer failed with, + /// under `doing`: [`InstallUnreachable`] when no connection could be + /// made, the cause as it came otherwise. + fn failed(&self, cause: reqwest::Error, doing: String) -> anyhow::Error { + if no_connection(&cause) { + return anyhow::Error::new(InstallUnreachable { url: self.base.clone(), target: self.target.clone(), cause }).context(doing); + } + anyhow::Error::new(cause).context(doing) + } + + /// The whole body of `resp`, as it came. + async fn body(&self, resp: reqwest::Response) -> anyhow::Result> { + let url = resp.url().clone(); + let body = resp.bytes().await.map_err(|cause| self.failed(cause, format!("read the answer from {url}")))?; + Ok(body.into()) + } + + /// The body of `resp` as text. + async fn text(&self, resp: reqwest::Response) -> anyhow::Result { + Ok(String::from_utf8_lossy(&self.body(resp).await?).into_owned()) + } + + /// A successful answer's JSON body, or `check`'s error. + async fn answer(&self, resp: reqwest::Response) -> anyhow::Result { + let resp = self.check(resp).await?; + serde_json::from_slice(&self.body(resp).await?).context("parse response") + } + /// The ONE place any client method turns an HTTP failure into an /// error. On non-2xx, surface the dispatcher's BODY text (its /// handlers return the reason as the body, e.g. "project is @@ -116,29 +317,32 @@ impl DispatcherClient { /// the bare status only when the body is empty, and names an HTML /// page rather than printing it (see `failure_text`). Every verb /// routes through this so they all get the same message quality. - async fn check(resp: reqwest::Response) -> anyhow::Result { + /// The error is a [`Refused`], so a caller can tell which status said no. + async fn check(&self, resp: reqwest::Response) -> anyhow::Result { let status = resp.status(); if status.is_success() { return Ok(resp); } - let completing = status == reqwest::StatusCode::CONFLICT - && resp.headers().contains_key(weft_core::storage::COMPLETING_HEADER); + let marked = match status { + reqwest::StatusCode::CONFLICT if resp.headers().contains_key(weft_core::storage::COMPLETING_HEADER) => { + Some(Marked::StoreCompleting) + } + reqwest::StatusCode::CONFLICT if resp.headers().contains_key(weft_core::builds::BUILD_BUSY_HEADER) => Some(Marked::BuildBusy), + _ => None, + }; let content_type = resp .headers() .get(reqwest::header::CONTENT_TYPE) .and_then(|v| v.to_str().ok()) .map(str::to_string); - let body = resp.text().await.with_context(|| format!("read the body of a {status} answer"))?; - let msg = failure_text(status, content_type.as_deref(), &body); - if completing { - return Err(anyhow::Error::new(StoreCompleting).context(msg)); - } - anyhow::bail!(msg) + let body = self.text(resp).await.with_context(|| format!("read the body of a {status} answer"))?; + let message = failure_text(status, content_type.as_deref(), &body); + Err(anyhow::Error::new(Refused { status, message, marked })) } pub async fn get_json(&self, path: &str) -> anyhow::Result { let resp = self.send(reqwest::Method::GET, path, None).await?; - Self::check(resp).await?.json().await.context("parse response") + self.answer(resp).await } /// GET where "no such thing" is an answer, not an error: `None` on @@ -154,17 +358,17 @@ impl DispatcherClient { { return Ok(None); } - Ok(Some(Self::check(resp).await?.json().await.context("parse response")?)) + Ok(Some(self.answer(resp).await?)) } pub async fn post_json(&self, path: &str, body: &serde_json::Value) -> anyhow::Result { let resp = self.send(reqwest::Method::POST, path, Some(body)).await?; - Self::check(resp).await?.json().await.context("parse response") + self.answer(resp).await } pub async fn delete(&self, path: &str) -> anyhow::Result<()> { let resp = self.send(reqwest::Method::DELETE, path, None).await?; - Self::check(resp).await?; + self.check(resp).await?; Ok(()) } @@ -180,7 +384,7 @@ impl DispatcherClient { { return Ok(()); } - Self::check(resp).await?; + self.check(resp).await?; Ok(()) } @@ -191,32 +395,32 @@ impl DispatcherClient { if resp.status() == reqwest::StatusCode::NOT_FOUND && resp.headers().contains_key("x-weft-not-found") { return Ok(None); } - Ok(Some(Self::check(resp).await?.json().await.context("parse response")?)) + Ok(Some(self.answer(resp).await?)) } pub async fn post_empty(&self, path: &str) -> anyhow::Result<()> { let resp = self.send(reqwest::Method::POST, path, None).await?; - Self::check(resp).await?; + self.check(resp).await?; Ok(()) } /// PUT with a JSON body, returning JSON. pub async fn put_json(&self, path: &str, body: &serde_json::Value) -> anyhow::Result { let resp = self.send(reqwest::Method::PUT, path, Some(body)).await?; - Self::check(resp).await?.json().await.context("parse response") + self.answer(resp).await } /// PUT with a JSON body, discard the response (a 204). pub async fn put_with_body(&self, path: &str, body: &serde_json::Value) -> anyhow::Result<()> { let resp = self.send(reqwest::Method::PUT, path, Some(body)).await?; - Self::check(resp).await?; + self.check(resp).await?; Ok(()) } /// DELETE returning JSON (a prune answers what it removed). pub async fn delete_json(&self, path: &str) -> anyhow::Result { let resp = self.send(reqwest::Method::DELETE, path, None).await?; - Self::check(resp).await?.json().await.context("parse response") + self.answer(resp).await } /// POST with a JSON body, returning JSON, or `Ok(Err(refusal))` when @@ -234,9 +438,9 @@ impl DispatcherClient { if resp.status() == reqwest::StatusCode::PRECONDITION_REQUIRED && resp.headers().contains_key(weft_core::TRIGGER_CHOICE_REQUIRED_HEADER) { - return Ok(Err(resp.text().await.unwrap_or_default().trim().to_string())); + return Ok(Err(self.text(resp).await?.trim().to_string())); } - Ok(Ok(Self::check(resp).await?.json().await.context("parse response")?)) + Ok(Ok(self.answer(resp).await?)) } /// POST with a JSON body, handing back the status and the body @@ -246,8 +450,17 @@ impl DispatcherClient { pub async fn post_json_status(&self, path: &str, body: &serde_json::Value) -> anyhow::Result<(u16, String)> { let resp = self.send(reqwest::Method::POST, path, Some(body)).await?; let status = resp.status(); - let text = resp.text().await.unwrap_or_default(); - Ok((status.as_u16(), text)) + Ok((status.as_u16(), self.text(resp).await?)) + } + + /// POST with a JSON body, handing back the success status and the + /// body text: for a route whose successes mean different things (a + /// build answers 200 once the version registered, 202 while its images + /// build). A failure is `check`'s. + pub async fn post_json_success(&self, path: &str, body: &serde_json::Value) -> anyhow::Result<(u16, String)> { + let resp = self.check(self.send(reqwest::Method::POST, path, Some(body)).await?).await?; + let status = resp.status(); + Ok((status.as_u16(), self.text(resp).await?)) } /// DELETE carrying a JSON body and returning JSON (the storage @@ -258,7 +471,7 @@ impl DispatcherClient { body: &serde_json::Value, ) -> anyhow::Result { let resp = self.send(reqwest::Method::DELETE, path, Some(body)).await?; - Self::check(resp).await?.json().await.context("parse response") + self.answer(resp).await } /// POST with a JSON body, discard the response. For endpoints @@ -269,7 +482,7 @@ impl DispatcherClient { body: &serde_json::Value, ) -> anyhow::Result<()> { let resp = self.send(reqwest::Method::POST, path, Some(body)).await?; - Self::check(resp).await?; + self.check(resp).await?; Ok(()) } } @@ -330,9 +543,139 @@ pub(crate) fn refusal_text(body: &str) -> Option { } } +/// A stand-in install for tests: every connection gets the same scripted +/// end, so a test can make each way a request fails happen for real. +#[cfg(test)] +pub(crate) mod fake { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + #[derive(Clone, Copy)] + pub enum Then { + /// Write these bytes as the answer, then close. + Answer(&'static str), + /// Read the request, then reset the connection without a word. + Reset, + } + + /// The address of a stand-in install doing `then` to every request. + pub async fn install(then: Then) -> String { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let base = format!("http://{}", listener.local_addr().unwrap()); + tokio::spawn(async move { + loop { + let Ok((mut conn, _)) = listener.accept().await else { return }; + let mut request = vec![0u8; 64 * 1024]; + let _ = conn.read(&mut request).await; + match then { + Then::Answer(raw) => { + let _ = conn.write_all(raw.as_bytes()).await; + let _ = conn.shutdown().await; + } + Then::Reset => { + #[allow(deprecated)] + conn.set_linger(Some(std::time::Duration::ZERO)).unwrap(); + } + } + } + }); + base + } + + /// The address of a port nothing listens on: a connection there is refused. + pub fn closed() -> String { + format!("http://{}", std::net::TcpListener::bind("127.0.0.1:0").unwrap().local_addr().unwrap()) + } +} + #[cfg(test)] mod tests { - use super::{failure_text, refusal_text}; + use futures::StreamExt; + use super::fake::{closed, install, Then}; + use super::{connection_failed, failure_text, is_loopback_address, refusal, refusal_text, DispatcherClient, InstallUnreachable}; + + fn unreachable(e: &anyhow::Error) -> bool { + e.chain().any(|cause| cause.is::()) + } + + /// A connection that cannot be made is the install not answering. + #[tokio::test] + async fn a_refused_connection_is_unreachable() { + let e = DispatcherClient::new(closed(), None).get_json("/install").await.unwrap_err(); + assert!(unreachable(&e), "{e:#}"); + assert!(connection_failed(&e)); + } + + /// A connection that breaks once the request is on it, before the + /// answer or halfway through it, failed on its way whichever verb was + /// reading, but never as "the install is not there", since it was. + #[tokio::test] + async fn a_connection_that_breaks_mid_request_failed_on_its_way() { + let reset = DispatcherClient::new(install(Then::Reset).await, None); + let e = reset.get_json("/install").await.unwrap_err(); + assert!(connection_failed(&e) && !unreachable(&e), "reset before the answer: {e:#}"); + + let cut = DispatcherClient::new( + install(Then::Answer("HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: 100\r\n\r\n{\"state\"")).await, + None, + ); + let e = cut.get_json("/install").await.unwrap_err(); + assert!(connection_failed(&e) && !unreachable(&e), "cut halfway through the answer: {e:#}"); + let e = cut.post_json_status("/run", &serde_json::json!({})).await.unwrap_err(); + assert!(connection_failed(&e), "a status read never swallows its body: {e:#}"); + } + + /// A status the install answered is a refusal carrying it, never a + /// connection that failed. + #[tokio::test] + async fn an_answered_status_is_a_refusal() { + let forbidden = DispatcherClient::new( + install(Then::Answer("HTTP/1.1 403 Forbidden\r\ncontent-length: 14\r\n\r\nnot your build")).await, + None, + ); + let e = forbidden.get_json("/projects/p/builds/b").await.unwrap_err(); + assert!(!connection_failed(&e), "{e:#}"); + assert_eq!(e.to_string(), "not your build"); + assert_eq!(refusal(&e).map(|r| r.status()), Some(reqwest::StatusCode::FORBIDDEN)); + } + + /// Live updates go through the same door: a refusal to open them is + /// the install's refusal, and a stream that cannot connect is unreachable. + #[tokio::test] + async fn live_updates_fail_like_every_other_request() { + let missing = DispatcherClient::new( + install(Then::Answer("HTTP/1.1 404 Not Found\r\ncontent-length: 10\r\n\r\nno such id")).await, + None, + ); + let e = missing.event_stream("/events/project/p").await.err().unwrap(); + assert_eq!(e.to_string(), "no such id"); + let e = DispatcherClient::new(closed(), None).event_stream("/events/project/p").await.err().unwrap(); + assert!(unreachable(&e), "{e:#}"); + // Opened, then cut before its end (how the install ends a + // subscriber that fell behind): reached, so never "unreachable". + let cut = DispatcherClient::new( + install(Then::Answer("HTTP/1.1 200 OK\r\ncontent-type: text/event-stream\r\ntransfer-encoding: chunked\r\n\r\n9\r\ndata: 1\n\n\r\n")).await, + None, + ); + let mut stream = std::pin::pin!(cut.event_stream("/events/project/p").await.unwrap()); + assert_eq!(stream.next().await.unwrap().unwrap().data, "1"); + let e = stream.next().await.unwrap().err().unwrap(); + assert!(!unreachable(&e), "{e:#}"); + } + + /// Only a loopback address is this machine's daemon. + #[test] + fn only_a_loopback_address_is_local() { + for (url, local) in [ + ("http://127.0.0.1:9000", true), + ("http://localhost:9000", true), + ("http://[::1]:9000", true), + ("https://weft.example.com", false), + ("http://10.0.0.4:9000", false), + ("not a url", false), + ] { + assert_eq!(is_loopback_address(url), local, "{url}"); + } + } #[test] fn a_structured_refusal_reads_as_its_errors() { diff --git a/crates/weft-cli/src/commands/activate.rs b/crates/weft-cli/src/commands/activate.rs index 8d83c2ca..e05c59af 100644 --- a/crates/weft-cli/src/commands/activate.rs +++ b/crates/weft-cli/src/commands/activate.rs @@ -140,7 +140,7 @@ fn left_out_note(left_out: &[String], on: Option<&str>) -> Option { Some(format!( "left off: {} run once per instance, so they stay off until an instance is named. \ Switch one instance's on with `{}`, or have the program switch \ - them on itself once it knows the instance (a node like ActivateInstanceTriggers does it)", + them on itself once it knows the instance (an ActivateTriggers node given the instance does it)", left_out.iter().map(|t| format!("'{t}'")).collect::>().join(", "), super::weft_on(on, "activate --instance ") )) diff --git a/crates/weft-cli/src/commands/assets.rs b/crates/weft-cli/src/commands/assets.rs index 2ab00119..e0be8d4d 100644 --- a/crates/weft-cli/src/commands/assets.rs +++ b/crates/weft-cli/src/commands/assets.rs @@ -371,10 +371,7 @@ impl AssetStore for DispatcherStore<'_> { } fn interrupted(&self, error: &anyhow::Error) -> bool { - error - .chain() - .filter_map(|cause| cause.downcast_ref::()) - .any(|e| e.is_connect() || e.is_timeout() || e.is_request() || e.is_body()) + crate::client::connection_failed(error) } } @@ -634,7 +631,8 @@ impl DispatcherStore<'_> { // the same bounded backoff the engine gives a completion. Otherwise // look once, without retrying the upload or interpreting error text. let Some(key) = upload_key else { return Err(error) }; - let completing = error.chain().any(|cause| cause.is::()); + let completing = + crate::client::refusal(&error).is_some_and(|refused| refused.marked() == Some(crate::client::Marked::StoreCompleting)); let mut attempt = 0; loop { let stored = self.client.get_json_if_found(&format!("/storage/files/meta/{key}")).await diff --git a/crates/weft-cli/src/commands/cancel_build.rs b/crates/weft-cli/src/commands/cancel_build.rs index d98aa8ab..30ac6a3c 100644 --- a/crates/weft-cli/src/commands/cancel_build.rs +++ b/crates/weft-cli/src/commands/cancel_build.rs @@ -1,11 +1,11 @@ //! `weft cancel-build [project]`. Cancel an in-flight build -//! (transition=building). Flips the transition to cancelling_build; -//! the dispatcher driving the build interrupts the builder job and -//! the verb that was building errs "cancelled". +//! (transition=building): the install stops every image build the +//! project started, stops waiting on the ones another project started +//! (they go on for it), and the verb following the build errs +//! "cancelled". Ctrl+C on that verb only stops following: the builds go +//! on, and running it again picks them up. //! -//! 412 from the dispatcher when no build is in flight (which is the -//! case when the build already ran locally before the verb; there, -//! Ctrl+C the CLI to cancel a build). +//! 412 from the dispatcher when no build is in flight. use super::Ctx; use crate::progress::ActionVerb; diff --git a/crates/weft-cli/src/commands/connect.rs b/crates/weft-cli/src/commands/connect.rs index 377bf65e..43f72543 100644 --- a/crates/weft-cli/src/commands/connect.rs +++ b/crates/weft-cli/src/commands/connect.rs @@ -1554,16 +1554,15 @@ fn ticked_permissions( opts: &ConnectOpts, interactive: bool, ) -> Result> { - // Own-account-only entries are capability declarations, never - // consent asks; they are set up in the provider's account and can - // never ride a consent URL, so they are not askable here either. - let tickable: Vec<_> = spec.permissions.iter().filter(|p| !p.own_only).collect(); + let tickable = spec.tickable_permissions(); let askable = |id: &str| tickable.iter().any(|p| p.id == id); if let Some(ids) = &opts.permissions { for id in ids { if !askable(id) { let known: Vec<&str> = tickable.iter().map(|p| p.id.as_str()).collect(); - let why = if spec.declares_permission(id) { + let why = if spec.permissions.iter().any(|p| p.id == *id && p.always) { + format!("'{id}' is asked for on every connect already; leave it out") + } else if spec.declares_permission(id) { format!( "'{id}' is an own-account-only capability: it is set up inside \ the connected account, never asked for at connect" diff --git a/crates/weft-cli/src/commands/ensure.rs b/crates/weft-cli/src/commands/ensure.rs index a4ec48a5..0d914d5c 100644 --- a/crates/weft-cli/src/commands/ensure.rs +++ b/crates/weft-cli/src/commands/ensure.rs @@ -11,9 +11,11 @@ //! - Resolve the `@asset` refs only this machine can read (a file outside //! the project, a URL) and send them with the version. //! - `POST /projects/{id}/builds`: the install fetches the version back, -//! compiles it, builds every stale image with its own BuildKit, and -//! registers the result. Nothing is built or loaded on this machine, -//! and nothing this machine computed is believed. +//! compiles it, and starts a build of every stale image with its own +//! builder, answering at once with those builds. The install registers +//! the version itself once they are built; this command follows it +//! until it is registered or failed. Nothing is built or loaded on this +//! machine, and nothing this machine computed is believed. //! - Registering a build is UNCONDITIONAL even while executions run on //! the previous image: every worker task is stamped with the image it //! was enqueued for and only claimable by a process baked from it, so @@ -22,8 +24,8 @@ use anyhow::{Context, Result}; -use weft_core::builds::BuiltProgram; -use weft_core::projects::BuildState; +use weft_core::builds::{BuildsUnderway, BuiltProgram, ImageBuild, VersionBuildState, VersionBuildStatus}; +use weft_core::projects::{BuildState, BuildStateResponse}; use super::Ctx; use crate::client::DispatcherClient; @@ -121,38 +123,254 @@ impl ProjectHandle { } } -/// How often a build in progress is asked which images it is building. +/// How often a version build being followed is asked where it is, and +/// how often an ask the install could not be reached for is sent again. const BUILD_LOOK_EVERY: std::time::Duration = std::time::Duration::from_secs(5); /// How often a build that goes on says so. const BUILD_SAY_EVERY: std::time::Duration = std::time::Duration::from_secs(30); -/// An image seen building while the CLI waits on the install's build. -struct Seen { - /// The builder's id for its build. - build: String, - /// Whether the address of its log was said. - log_said: bool, - /// How long into the wait it was first seen. +/// The image builds a version build waits on, as this command follows +/// them: the ones still running, the images that finished, and how the +/// others failed. +#[derive(Default)] +struct Following { + /// By the name weft minted for each build. + pending: std::collections::BTreeMap, + /// The images this command saw built, in the order they finished. + built: Vec, + /// How each build that did not succeed ended, in the order they did. + failures: Vec, +} + +/// One image build being followed. +struct Followed { + image: String, + /// How long into the command it was first seen. since: std::time::Duration, + /// The builder's id for it, once said; whether its log was said too. + said: Option<(String, bool)>, + /// Its log's address, once known: a failure names it. + log_url: Option, +} + +/// What a look at one build has to be said. +#[derive(Debug, PartialEq)] +enum Said { + /// It runs on the builder (said again once its log's address comes). + Building(weft_core::projects::BuildInFlight), + /// It finished, about `seconds` after it was first seen. + Built { image: String, seconds: u64 }, +} + +impl Following { + /// Follow `builds`, the image builds a 202 named. + fn new(builds: Vec, elapsed: std::time::Duration) -> Self { + let pending = builds + .into_iter() + .map(|build| (build.name, Followed { image: build.image, since: elapsed, said: None, log_url: None })) + .collect(); + Self { pending, ..Self::default() } + } + + /// Take in what the install answered about the build `name`, and + /// answer what is to be said about it. + fn note(&mut self, name: &str, answer: &BuildStateResponse, elapsed: std::time::Duration) -> Vec { + let Some(followed) = self.pending.get_mut(name) else { return Vec::new() }; + let mut said = Vec::new(); + if answer.log_url.is_some() { + followed.log_url.clone_from(&answer.log_url); + } + if let Some(build) = &answer.build { + let with_log = answer.log_url.is_some(); + let say = match &followed.said { + None => true, + Some((_, log_said)) => !log_said && with_log, + }; + if say { + followed.said = Some((build.clone(), with_log)); + said.push(Said::Building(weft_core::projects::BuildInFlight { + image: followed.image.clone(), + build: build.clone(), + started_at_unix: answer.started_at_unix.unwrap_or_default(), + log_url: answer.log_url.clone(), + })); + } + } + let log = followed.log_url.as_ref().map(|url| format!("\nits log: {url}")).unwrap_or_default(); + let image = followed.image.clone(); + let ended = match answer.state { + Some(BuildState::Running) => return said, + Some(BuildState::Succeeded) => { + said.push(Said::Built { image: image.clone(), seconds: elapsed.saturating_sub(followed.since).as_secs() }); + self.built.push(image); + None + } + Some(BuildState::Failed) => Some(format!( + "the build of {image} failed:\n{}{log}", + answer.reason.as_deref().unwrap_or("the builder gave no reason") + )), + Some(BuildState::Cancelled) => Some(format!("the build of {image} was cancelled")), + None => Some(format!("the build of {image} is no longer on record")), + }; + self.failures.extend(ended); + self.pending.remove(name); + said + } + + /// The images building right now, as far as this command has said. + fn building(&self) -> Vec { + self.pending.values().filter(|f| f.said.is_some()).map(|f| f.image.clone()).collect() + } +} + +/// Follow the version build `build` of project `id` until the install +/// registered it, saying what each image build is doing on the way, and +/// answer what registered. A version that failed, was cancelled, or was +/// superseded by a newer build ends the command with why. While the +/// install is out of reach the builds go on there, so the command says so +/// and keeps looking, for as long as it takes; any other failure to look +/// (a refusal, a body it cannot read) ends the command with it. +async fn follow_version( + client: &DispatcherClient, + id: &str, + build: uuid::Uuid, + progress: &Progress, + following: &mut Following, + started: std::time::Instant, +) -> Result { + let mut said = started.elapsed(); + let mut out_of_reach = false; + loop { + tokio::time::sleep(BUILD_LOOK_EVERY).await; + let elapsed = started.elapsed(); + let due = elapsed.saturating_sub(said) >= BUILD_SAY_EVERY; + let answer = match version_state(client, id, build).await { + Ok(answer) => answer, + Err(e) => { + let why = look_again_after(e).map_err(|e| e.context("read where the build is"))?; + // Said at once when the install goes out of reach, then on + // the usual beat while it stays out. + if !out_of_reach || due { + said = elapsed; + progress.build_unreachable(elapsed, &why); + } + out_of_reach = true; + continue; + } + }; + for image in &answer.images { + for say in following.note(&image.build.name, &image.state, started.elapsed()) { + match say { + Said::Building(build) => progress.build_image(&build), + Said::Built { image, seconds } => progress.build_image_done(&image, seconds), + } + } + } + let why = || answer.reason.clone().unwrap_or_else(|| "the install gave no reason".to_string()); + match answer.state { + VersionBuildStatus::Registered => { + return answer.program.context("the install says the version registered, but answered no program"); + } + VersionBuildStatus::Waiting => {} + VersionBuildStatus::Failed if !following.failures.is_empty() => { + anyhow::bail!("the build failed:\n{}", following.failures.join("\n\n")) + } + VersionBuildStatus::Failed => anyhow::bail!("the build failed:\n{}", why()), + VersionBuildStatus::Cancelled => anyhow::bail!("the build was cancelled"), + VersionBuildStatus::Superseded => anyhow::bail!("this build will not be registered: {}", why()), + } + // Back in reach: said at once, so nothing goes on reading as out + // of reach. + let back = std::mem::take(&mut out_of_reach); + if back || due { + said = elapsed; + progress.build_wait(elapsed, &following.building(), &following.built); + } + } +} + +/// A request to the install that failed: `Ok` with why, to say, when the +/// install is only out of reach for now and the request is to be sent +/// again: its connection failed on the way +/// ([`crate::client::connection_failed`]), or the install (or the gateway +/// in front of it) answered that it cannot serve this right now (a 502, +/// 503 or 504: down, starting, or unable to read its database). Both +/// requests this is read on (asking for the build, which joins the one +/// already asked for, and reading where it is) are safe to send again. The +/// error itself when anything else answered. +fn look_again_after(e: anyhow::Error) -> Result { + use reqwest::StatusCode; + let busy = crate::client::refusal(&e).is_some_and(|refused| { + matches!(refused.status(), StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT) + }); + if busy || crate::client::connection_failed(&e) { + return Ok(format!("{e:#}")); + } + Err(e) } -/// Where the build `build` of project `id` is, `None` when the install has -/// no such build on record. -async fn build_state(client: &crate::client::DispatcherClient, id: &str, build: &str) -> Result> { +/// A build ask that failed: `Ok` with why, to say, when it is to be sent +/// again ([`look_again_after`], and, once it was `sent_again`, the first +/// ask still starting the builds, [`crate::client::Marked::BuildBusy`]); +/// the error itself otherwise. +fn ask_again_after(e: anyhow::Error, sent_again: bool) -> Result { + let first_still_starting = + sent_again && crate::client::refusal(&e).is_some_and(|refused| refused.marked() == Some(crate::client::Marked::BuildBusy)); + if first_still_starting { + return Ok(format!("{e:#}")); + } + look_again_after(e) +} + +/// Where the version build `build` of project `id` is. +async fn version_state(client: &DispatcherClient, id: &str, build: uuid::Uuid) -> Result { let answer = client.get_json(&format!("/projects/{id}/builds/{build}")).await?; - let answer: weft_core::projects::BuildStateResponse = serde_json::from_value(answer).context("read a build's state")?; - Ok(answer.state) + serde_json::from_value(answer).context("read a build's state") } -/// The image builds running for project `id`. Only ever used to tell a -/// person where a build is, so the caller warns about a status that -/// cannot be read rather than failing the build being waited on. -async fn builds_running(client: &crate::client::DispatcherClient, id: &str) -> Result> { - let Some(status) = client.get_json_if_found(&format!("/projects/{id}/status")).await? else { return Ok(Vec::new()) }; - let status: weft_core::projects::ProjectStatusResponse = - serde_json::from_value(status).context("read the project's status")?; - Ok(status.builds) +/// What the build ask answered. +enum Asked { + /// Every image was there: the version registered. + Registered(Box), + /// The install registers it once these builds end. + Underway(BuildsUnderway), +} + +/// Ask the install to build `body`, sending the ask again while the +/// install is out of reach ([`look_again_after`]): a second ask joins the +/// version build the first one may have started. Once an ask was sent +/// again, the install answering that the project's builds are being +/// started right now ([`crate::client::Marked::BuildBusy`]) is that first +/// ask still starting them, so it is waited out too; on the first ask +/// that answer is somebody else's build, and ends the command. +async fn ask_to_build( + client: &DispatcherClient, + path: &str, + body: &serde_json::Value, + progress: &Progress, + started: std::time::Instant, +) -> Result { + let mut said: Option = None; + let mut sent_again = false; + loop { + let asked = client.post_json_success(path, body).await.and_then(|(status, text)| match status { + 202 => Ok(Asked::Underway(serde_json::from_str(&text).context("read the builds the install started")?)), + _ => Ok(Asked::Registered(Box::new(serde_json::from_str(&text).context("read the build's answer")?))), + }); + let e = match asked { + Ok(asked) => return Ok(asked), + Err(e) => e, + }; + let why = ask_again_after(e, sent_again).map_err(|e| e.context("the build failed"))?; + sent_again = true; + let elapsed = started.elapsed(); + if said.is_none_or(|said| elapsed.saturating_sub(said) >= BUILD_SAY_EVERY) { + said = Some(elapsed); + progress.build_unreachable(elapsed, &why); + } + tokio::time::sleep(BUILD_LOOK_EVERY).await; + } } /// Make sure the install knows this project, WITHOUT building anything. @@ -316,101 +534,19 @@ pub async fn build_compiled( }; progress.build_start(&project.manifest.package.name); progress.dispatcher_call_start(&path); - let body = serde_json::to_value(&body)?; let started = std::time::Instant::now(); - // Each image once, and again when its log's address arrives: its - // build, whether its log was said, and when it was first seen. - let mut building: std::collections::BTreeMap = std::collections::BTreeMap::new(); - // The images this wait saw finish, in the order they did: between two - // images nothing is building, and the heartbeat says what is done. - let mut built_so_far: Vec = Vec::new(); - let (status, text) = { - // While the install builds, name each image it is building and - // where its log is, say when each is done, and say every - // `BUILD_SAY_EVERY` that it goes on: a build can take minutes, - // and a quiet terminal reads as a hang. - let mut said = std::time::Duration::ZERO; - // A status that cannot be read is said once, not every look; the - // same for an image's own state. - let mut warned = false; - let mut state_warned = false; - crate::progress::while_waiting(client.post_json_status(&path, &body), BUILD_LOOK_EVERY, async |elapsed| { - match builds_running(&client, &id).await { - Ok(builds) => { - let now: std::collections::BTreeSet<&str> = builds.iter().map(|b| b.image.as_str()).collect(); - let gone: Vec = building.keys().filter(|image| !now.contains(image.as_str())).cloned().collect(); - // An image no longer building may have failed: only one the - // ledger says succeeded is said built. A failure is the - // build's own answer, at its end. - for image in gone { - let seen = &building[&image]; - match build_state(&client, &id, &seen.build).await { - Ok(Some(BuildState::Running)) => continue, - Ok(Some(BuildState::Succeeded)) => { - progress.build_image_done(&image, elapsed.saturating_sub(seen.since).as_secs()); - built_so_far.push(image.clone()); - } - Ok(Some(BuildState::Failed | BuildState::Cancelled) | None) => {} - // Not said built, nor listed as building any more: - // what became of it is the build's own answer, at - // its end. - Err(e) => { - if !state_warned { - state_warned = true; - progress.warn(&format!("cannot tell whether {image} built ({e:#}); the build's answer will")); - } - } - } - building.remove(&image); - } - for build in &builds { - let with_log = build.log_url.is_some(); - match building.get_mut(&build.image) { - None => { - building.insert(build.image.clone(), Seen { build: build.build.clone(), log_said: with_log, since: elapsed }); - progress.build_image(build); - } - Some(seen) if !seen.log_said && with_log => { - seen.log_said = true; - progress.build_image(build); - } - Some(_) => {} - } - } - } - Err(e) if !warned => { - warned = true; - progress.warn(&format!( - "cannot show which images are building or where their logs are ({e:#}); the build goes on" - )); - } - Err(_) => {} - } - if elapsed.saturating_sub(said) >= BUILD_SAY_EVERY { - said = elapsed; - progress.build_wait(elapsed, &building.keys().cloned().collect::>(), &built_so_far); - } - }) - .await - .context("ask the install to build")? - }; - if !(200..300).contains(&status) { - anyhow::bail!( - "the build failed:\n{}", - crate::client::refusal_text(&text).unwrap_or_else(|| format!("the install answered {status}")) - ); - } - let built: BuiltProgram = serde_json::from_str(&text).context("read the build's answer")?; - // The last images to finish (the worker's, always among them) finish as - // the build answers, so no look saw them leave: they are said built - // here. - for (image, seen) in &building { - if built.built_images.contains(image) { - progress.build_image_done(image, started.elapsed().saturating_sub(seen.since).as_secs()); + // The ask answers at once: the version registered (200), or the image + // builds it waits on (202), which the install registers it after. + let (built, following) = match ask_to_build(&client, &path, &serde_json::to_value(&body)?, progress, started).await? { + Asked::Registered(built) => (*built, Following::default()), + Asked::Underway(underway) => { + let mut following = Following::new(underway.images, started.elapsed()); + let built = follow_version(&client, &id, underway.build, progress, &mut following, started).await?; + (built, following) } - } + }; progress.dispatcher_call_done(serde_json::json!({ "project_id": id })); - progress.build_done(&project.manifest.package.name, &built.built_images); + progress.build_done(&project.manifest.package.name, &following.built); if let Some(note) = replaced_infra_images_note(&built.replaced_infra_images, ctx.on()) { progress.warn(¬e); } @@ -438,3 +574,116 @@ pub async fn build_compiled( built, }) } + +#[cfg(test)] +mod tests { + use super::*; + + fn answer(state: Option) -> BuildStateResponse { + BuildStateResponse { state, ..Default::default() } + } + + fn secs(n: u64) -> std::time::Duration { + std::time::Duration::from_secs(n) + } + + /// A build is said once the builder names it, again once its log's + /// address comes, and built at its end, with how long it took. + #[test] + fn a_followed_build_is_said_as_it_goes() { + let mut following = Following::new(vec![ImageBuild { image: "worker".into(), name: "b1".into() }], secs(2)); + assert!(following.note("b1", &answer(Some(BuildState::Running)), secs(5)).is_empty(), "not on the builder yet"); + let named = BuildStateResponse { build: Some("cb-1".into()), started_at_unix: Some(7), ..answer(Some(BuildState::Running)) }; + let said = following.note("b1", &named, secs(10)); + assert!(matches!(&said[..], [Said::Building(b)] if b.build == "cb-1" && b.log_url.is_none()), "{said:?}"); + assert!(following.note("b1", &named, secs(15)).is_empty(), "said once"); + let with_log = BuildStateResponse { log_url: Some("https://log".into()), ..named.clone() }; + let said = following.note("b1", &with_log, secs(20)); + assert!(matches!(&said[..], [Said::Building(b)] if b.log_url.as_deref() == Some("https://log")), "{said:?}"); + assert_eq!(following.building(), ["worker"]); + let done = BuildStateResponse { state: Some(BuildState::Succeeded), ..with_log }; + assert_eq!(following.note("b1", &done, secs(62)), [Said::Built { image: "worker".into(), seconds: 60 }]); + assert!(following.pending.is_empty()); + assert_eq!(following.built, ["worker"]); + assert!(following.failures.is_empty()); + assert!(following.note("b1", &done, secs(67)).is_empty(), "a build that ended is said once"); + } + + /// Every way a build can end other than built is a failure that names + /// its image, the reason and the log when there are some. + #[test] + fn a_build_that_did_not_succeed_says_why() { + let mut following = Following::new( + vec![ + ImageBuild { image: "worker".into(), name: "b1".into() }, + ImageBuild { image: "db".into(), name: "b2".into() }, + ImageBuild { image: "cache".into(), name: "b3".into() }, + ], + secs(0), + ); + let failed = BuildStateResponse { + reason: Some("cargo: error[E0425]".into()), + log_url: Some("https://log/1".into()), + ..answer(Some(BuildState::Failed)) + }; + following.note("b1", &failed, secs(5)); + following.note("b2", &answer(Some(BuildState::Cancelled)), secs(5)); + following.note("b3", &answer(None), secs(5)); + assert!(following.pending.is_empty()); + assert_eq!(following.failures[0], "the build of worker failed:\ncargo: error[E0425]\nits log: https://log/1"); + assert_eq!(following.failures[1], "the build of db was cancelled"); + assert_eq!(following.failures[2], "the build of cache is no longer on record"); + } + + /// A look the install refused, or answered with something unreadable, + /// ends the follow with that answer; one that never reached the + /// install, or reached only its gateway saying it is down, is looked + /// at again. + #[tokio::test] + async fn a_refused_look_ends_the_follow_and_an_unreachable_one_looks_again() { + use crate::client::fake::{closed, install, Then}; + let build = uuid::Uuid::nil(); + let look = |base: String| async move { version_state(&DispatcherClient::new(base, None), "p", build).await.unwrap_err() }; + + let refused = look(install(Then::Answer("HTTP/1.1 401 Unauthorized\r\ncontent-length: 11\r\n\r\nwrong key\r\n")).await).await; + let e = look_again_after(refused).unwrap_err(); + assert_eq!(format!("{e:#}"), "wrong key"); + + let unreadable = look(install(Then::Answer("HTTP/1.1 200 OK\r\ncontent-length: 6\r\n\r\n{\"x\":1")).await).await; + assert!(look_again_after(unreadable).is_err(), "an answer it cannot read is not the install being away"); + + let why = look_again_after(look(closed()).await).unwrap(); + assert!(why.contains("did not answer"), "{why}"); + let reset = look(install(Then::Reset).await).await; + assert!(look_again_after(reset).is_ok()); + let down = look(install(Then::Answer("HTTP/1.1 502 Bad Gateway\r\ncontent-length: 0\r\n\r\n")).await).await; + assert!(look_again_after(down).is_ok()); + let busy = look(install(Then::Answer("HTTP/1.1 503 Service Unavailable\r\ncontent-length: 0\r\n\r\n")).await).await; + assert!(look_again_after(busy).is_ok()); + } + + /// The project's builds being started right now ends a first ask (it + /// is somebody else's build), and is waited out once the ask was sent + /// again (it is this command's own first ask, whose answer was lost). + #[tokio::test] + async fn a_busy_project_ends_a_first_ask_and_is_waited_out_once_sent_again() { + use crate::client::fake::{install, Then}; + let busy = || async { + let base = install(Then::Answer( + "HTTP/1.1 409 Conflict\r\nx-weft-build-busy: 1\r\ncontent-length: 16\r\n\r\ncannot build now", + )) + .await; + DispatcherClient::new(base, None).post_json_success("/projects/p/builds", &serde_json::json!({})).await.unwrap_err() + }; + assert!(ask_again_after(busy().await, false).is_err()); + assert!(ask_again_after(busy().await, true).is_ok()); + let cancelled = DispatcherClient::new( + install(Then::Answer("HTTP/1.1 409 Conflict\r\ncontent-length: 15\r\n\r\nbuild cancelled")).await, + None, + ) + .post_json_success("/projects/p/builds", &serde_json::json!({})) + .await + .unwrap_err(); + assert!(ask_again_after(cancelled, true).is_err(), "only the busy marker is waited out"); + } +} diff --git a/crates/weft-cli/src/commands/follow.rs b/crates/weft-cli/src/commands/follow.rs index b47786d8..a576a1ec 100644 --- a/crates/weft-cli/src/commands/follow.rs +++ b/crates/weft-cli/src/commands/follow.rs @@ -6,7 +6,6 @@ use anyhow::Context; use std::collections::HashSet; use serde_json::Value; -use eventsource_client::Client; use futures::StreamExt; use weft_core::live_event::LiveEvent; @@ -70,50 +69,41 @@ async fn follow_sse( target: FollowTarget<'_>, mut emit: impl FnMut(&LiveEvent) -> anyhow::Result<()>, ) -> anyhow::Result<()> { - let es = client - .event_stream(&target.path())? - // Reconnecting without recovering history would hide lost updates. - // A disconnected follower must tell the user it stopped watching. - .reconnect(eventsource_client::ReconnectOptions::reconnect(false).build()) - .build(); - let mut stream = es.stream(); + let inspect = || { + format!( + "the run may still be running. Inspect it with `{}` and `{}`", + super::weft_on(on, "executions"), + super::weft_on(on, "events ") + ) + }; + // Never reopened: reconnecting without recovering history would hide + // lost updates, so a follower that loses the stream says it stopped + // watching. + let mut stream = std::pin::pin!(client.event_stream(&target.path()).await.context("listen for live updates")?); // Only the finite initial history needs overlap detection. Do not grow // a seen-event cache for the lifetime of a potentially unbounded run. let mut history_ids = HashSet::new(); - while let Some(ev) = stream.next().await { - let inspect = || { - format!( - "the run may still be running. Inspect it with `{}` and `{}`", - super::weft_on(on, "executions"), - super::weft_on(on, "events ") - ) - }; - match ev.with_context(|| format!("live updates interrupted; {}", inspect()))? { - eventsource_client::SSE::Event(event) => { - let event: LiveEvent = serde_json::from_str(&event.data).context("decode execution event")?; - if history_ids.contains(event_identity(&event)?) { continue; } - emit(&event)?; - if target.finished(&event) { - return Ok(()); - } - } - eventsource_client::SSE::Comment(_) => {} - eventsource_client::SSE::Connected(_) => { - // The server is listening now. Recover everything that ran - // before attachment, including an already-finished run. - // New events remain buffered on the open stream meanwhile. - if let FollowTarget::Execution(execution_id) = target { - for event in super::versions::replay_rows(client, execution_id).await? { - if !history_ids.insert(event_identity(&event)?.to_owned()) { continue; } - emit(&event)?; - if target.finished(&event) { - return Ok(()); - } - } - } + // The server is listening now. Recover everything that ran before + // attachment, including an already-finished run. New events remain + // buffered on the open stream meanwhile. + if let FollowTarget::Execution(execution_id) = target { + for event in super::versions::replay_rows(client, execution_id).await? { + if !history_ids.insert(event_identity(&event)?.to_owned()) { continue; } + emit(&event)?; + if target.finished(&event) { + return Ok(()); } } } + while let Some(ev) = stream.next().await { + let ev = ev.with_context(|| format!("live updates interrupted; {}", inspect()))?; + let event: LiveEvent = serde_json::from_str(&ev.data).context("decode execution event")?; + if history_ids.contains(event_identity(&event)?) { continue; } + emit(&event)?; + if target.finished(&event) { + return Ok(()); + } + } anyhow::bail!( "live updates ended before following was finished; inspect the run with `{}` and `{}`", super::weft_on(on, "executions"), diff --git a/crates/weft-cli/src/commands/mod.rs b/crates/weft-cli/src/commands/mod.rs index ca33a382..9688e6a6 100644 --- a/crates/weft-cli/src/commands/mod.rs +++ b/crates/weft-cli/src/commands/mod.rs @@ -278,7 +278,7 @@ impl Ctx { /// resolves it, so an install with no address yet errors here. pub fn client(&self) -> anyhow::Result { let install = self.install()?; - Ok(crate::client::DispatcherClient::new(install.url.clone(), install.operator_key.clone())) + Ok(crate::client::DispatcherClient::new(install.url.clone(), install.operator_key.clone()).for_target(self.on())) } /// A client resolved afresh on every call, never cached: a long-lived @@ -291,7 +291,7 @@ impl Ctx { /// edit is seen. pub fn fresh_client(&self, project: Option<&Project>) -> anyhow::Result { let install = self.resolve_install(|| Ok(project.cloned()))?; - Ok(crate::client::DispatcherClient::new(install.url, install.operator_key)) + Ok(crate::client::DispatcherClient::new(install.url, install.operator_key).for_target(self.on())) } /// The cwd-discovered project, lazy-loaded and cached, keeping diff --git a/crates/weft-cli/src/commands/versions.rs b/crates/weft-cli/src/commands/versions.rs index d023d2ec..b25aad76 100644 --- a/crates/weft-cli/src/commands/versions.rs +++ b/crates/weft-cli/src/commands/versions.rs @@ -605,9 +605,9 @@ pub fn frames_key(frames: &weft_core::frames::LoopFrames) -> String { weft_core::frames::frames_text(frames) } -/// The outside facts a replay shows: every kick payload, every answer -/// a person gave (with the question the node showed them, which is -/// its firing input), and every live caller message. +/// The outside facts a replay shows: how every wait ended (a person's +/// answer or skip, or the run giving it up, with the question the node +/// showed, which is its firing input), and every live caller message. #[derive(Debug, Default, Clone)] pub struct OutsideFacts { pub answers: Vec, @@ -623,11 +623,13 @@ pub fn outside_facts(rows: &[LiveEvent]) -> Result { DispatcherEvent::NodeStarted { node, frames, input, .. } => { inputs.insert((node.clone(), frames_key(frames)), input.clone()); } - // A resume with no token is a boundary re-fire, not an answer. - DispatcherEvent::NodeResumed { node, frames, token: Some(_), value, .. } => { + // Every wait, the ones held in the worker included (those + // never suspend or resume the node, so only this event sees + // them): a replay ends each the same way. + DispatcherEvent::WaitEnded { node, frames, ended, .. } => { facts.answers.push(weft_core::run_spec::Answer { node: node.clone(), - payload: value.clone().unwrap_or(Value::Null), + ended: ended.clone(), question: inputs.get(&(node.clone(), frames_key(frames))).cloned(), frames: frames.clone(), }); @@ -807,6 +809,7 @@ mod tests { use super::*; use serde_json::json; use weft_core::versions::{Head, ManifestDiff}; + use weft_core::primitive::AwaitEnd; #[test] fn a_renamed_example_is_run_and_listed_by_its_filename() { @@ -1159,7 +1162,7 @@ mod tests { } #[test] - fn outside_facts_pair_each_answer_with_the_question_the_node_showed() { + fn outside_facts_record_how_each_wait_ended_with_the_question_the_node_showed() { // Each row as the replay answers it: the run's own fields around // the kind's. let row = |mut fields: Value| -> LiveEvent { @@ -1173,8 +1176,15 @@ mod tests { let rows = vec![ row(json!({ "kind": "node_started", "node": "review", "frames": [], "input": { "prompt": "ok?" }, "closed_ports": [] })), row(json!({ "kind": "node_suspended", "node": "review", "frames": [], "token": "t" })), - row(json!({ "kind": "node_resumed", "node": "review", "frames": [], "token": "t", "value": { "answer": "yes" } })), - row(json!({ "kind": "node_resumed", "node": "crashed", "frames": [], "token": null, "value": null })), + row(json!({ "kind": "wait_ended", "node": "review", "frames": [], "token": "t", "ended": { "type": "answered", "value": { "answer": "yes" } } })), + row(json!({ "kind": "node_resumed", "node": "review", "frames": [], "token": "t" })), + row(json!({ "kind": "node_resumed", "node": "crashed", "frames": [], "token": null })), + row(json!({ "kind": "wait_ended", "node": "skipped", "frames": [], "token": "t3", "ended": { "type": "skipped" } })), + // A wait held in the worker that gave up: no suspend, no + // resume, only its ending and the node failing with its error. + row(json!({ "kind": "node_started", "node": "held", "frames": [], "input": { "prompt": "still there?" }, "closed_ports": [] })), + row(json!({ "kind": "wait_ended", "node": "held", "frames": [], "token": "t2", "ended": { "type": "gave_up", "error": "node 'held' gave up its wait: quiet" } })), + row(json!({ "kind": "node_failed", "node": "held", "frames": [], "error": "node 'held' gave up its wait: quiet" })), // One window row carrying the whole short conversation, // which is how the journal writes it: the messages come out // of the row, not one row each. @@ -1184,9 +1194,14 @@ mod tests { ], "totals": [] })), ]; let facts = outside_facts(&rows).unwrap(); - assert_eq!(facts.answers.len(), 1, "a boundary re-fire is not an answer"); - assert_eq!(facts.answers[0].payload, json!({ "answer": "yes" })); + assert_eq!(facts.answers.len(), 3, "one per wait that ended; a boundary re-fire is none"); + assert_eq!(facts.answers[0].ended, AwaitEnd::Answered { value: json!({ "answer": "yes" }) }); assert_eq!(facts.answers[0].question, Some(json!({ "prompt": "ok?" }))); + assert_eq!(facts.answers[1].node, "skipped"); + assert_eq!(facts.answers[1].ended, AwaitEnd::Skipped); + assert_eq!(facts.answers[2].node, "held"); + assert_eq!(facts.answers[2].ended, AwaitEnd::GaveUp { error: "node 'held' gave up its wait: quiet".into() }); + assert_eq!(facts.answers[2].question, Some(json!({ "prompt": "still there?" })), "a wait given up keeps the question it asked"); assert_eq!(facts.caller.len(), 2, "both halves of the exchange, request AND answer"); let directions: Vec<&str> = facts.caller.iter().filter_map(|m| m["direction"].as_str()).collect(); diff --git a/crates/weft-cli/src/progress.rs b/crates/weft-cli/src/progress.rs index f7dac4eb..f68e1e60 100644 --- a/crates/weft-cli/src/progress.rs +++ b/crates/weft-cli/src/progress.rs @@ -41,9 +41,8 @@ pub enum ActionVerb { /// partial trigger registrations and flips the project back to /// Inactive. CancelActivate, - /// Cancel an in-flight build (transition=building). - /// Flips the transition to cancelling_build; the dispatcher driving - /// the build sees it and stops the build. + /// Cancel an in-flight build (transition=building): the install + /// stops the project's image builds. CancelBuild, Deactivate, /// Force-cancel running executions while a deactivate-with-wait @@ -91,6 +90,12 @@ pub enum Phase { /// `{ "elapsedSeconds", "images": [...], "built": [...] }`: the images /// still building, and the ones this build has finished so far. BuildWait, + /// The install cannot be reached to see where the build is (no + /// answer, or its gateway says it is down); the command keeps looking. + /// Detail carries `{ "elapsedSeconds", "message" }`: since the build + /// began, and why the last look failed. The next `BuildWait` or + /// `BuildImage` says the install answers again. + BuildUnreachable, /// HTTP request to the dispatcher started. DispatcherCallStart, /// HTTP request to the dispatcher finished. Body in `detail` @@ -141,15 +146,8 @@ struct Event<'a> { detail: Option<&'a Value>, } -/// The one report for a failure that reached `main` unreported: a verb -/// with no progress channel, or a failure before a channel existed. -/// Same shape as a channel's own error, so every verb fails the same -/// way: under `--json` one `phase: "error"` event on stdout, otherwise -/// one lowercase `error:` line on stderr. The cause chain rides along -/// in the message (`{e:#}`), since readers build their errors from -/// context and the chain is where the "why" lives. -/// What a failure that never reached the daemon says, first, before the -/// transport noise. +/// What a failure that never reached this machine's daemon says, first, +/// before the transport noise. /// /// Worth its own sentence because the three reasons a request does not /// arrive (the daemon is not started, this machine only has the editor @@ -162,17 +160,20 @@ struct Event<'a> { /// is a fact rather than wording, so this line is free to be rephrased. const DAEMON_UNREACHABLE: &str = "the weft daemon is not answering"; -/// True when the request never reached the daemon at all: nothing was -/// refused, nothing answered. A status the daemon returns, however bad, +/// The install a request never reached, when one did not: nothing was +/// refused, nothing answered. A status the install returns, however bad, /// is not this. -fn daemon_unreachable(e: &anyhow::Error) -> bool { - e.chain().any(|cause| { - cause - .downcast_ref::() - .is_some_and(|r| r.is_connect() || r.is_timeout()) - }) +fn unreachable_install(e: &anyhow::Error) -> Option<&crate::client::InstallUnreachable> { + e.chain().find_map(|cause| cause.downcast_ref::()) } +/// The one report for a failure that reached `main` unreported: a verb +/// with no progress channel, or a failure before a channel existed. +/// Same shape as a channel's own error, so every verb fails the same +/// way: under `--json` one `phase: "error"` event on stdout, otherwise +/// one lowercase `error:` line on stderr. The cause chain rides along +/// in the message (`{e:#}`), since readers build their errors from +/// context and the chain is where the "why" lives. pub fn report_plain_error(json: bool, e: &anyhow::Error) { let detail = error_detail(e); if json { @@ -187,21 +188,28 @@ pub fn report_plain_error(json: bool, e: &anyhow::Error) { /// that reports one: the message (with the cause chain, `{e:#}`, since /// that is where the "why" lives), plus the facts a host acts on /// without matching the wording. -// SYNC: daemonUnreachable <-> extension-vscode/src/cli.ts CliFailure +// SYNC: daemonUnreachable, installUnreachable <-> extension-vscode/src/cli.ts CliFailure // SYNC: needsTriggerChoice <-> extension-vscode/src/triggerChoice.ts isTriggerChoiceRefusal pub fn error_detail(e: &anyhow::Error) -> Value { - let unreachable = daemon_unreachable(e); - let message = if unreachable { - format!( + let unreachable = unreachable_install(e); + let local = unreachable.is_some_and(|install| install.is_local()); + let message = match unreachable { + Some(_) if local => format!( "{DAEMON_UNREACHABLE}. Start it with `weft daemon start`, or install it if this \ machine only has the editor extension. ({e:#})" - ) - } else { - format!("{e:#}") + ), + // A remote install: nothing on this machine would start it, so + // the daemon's advice would send the person the wrong way. The + // chain names it (`InstallUnreachable`'s own words). + Some(_) => format!("{e:#}; check the network and run the command again"), + None => format!("{e:#}"), }; serde_json::json!({ "message": message, - "daemonUnreachable": unreachable, + // This machine's daemon did not answer: a host may offer to start it. + "daemonUnreachable": local, + // Any install did not answer, this machine's or a remote one. + "installUnreachable": unreachable.is_some(), // The dispatcher needs to know how triggers come down and // nobody could be asked: a host that can ask opens its picker // and sends the verb again with the choice. @@ -319,6 +327,16 @@ impl Progress { ); } + /// The last look at the build could not reach the install, `why`; + /// `elapsed` since the build began. + /// SYNC: the build_unreachable detail <-> packages/weft-graph/src/webview/lib/components/project/ActionBar.svelte (build_unreachable) + pub fn build_unreachable(&self, elapsed: std::time::Duration, why: &str) { + self.emit( + Phase::BuildUnreachable, + Some(serde_json::json!({ "elapsedSeconds": elapsed.as_secs(), "message": why })), + ); + } + /// `built`: the image refs the install had to build (empty when every /// image was already there). pub fn build_done(&self, project: &str, built: &[String]) { @@ -506,6 +524,15 @@ fn human_line(ev: &Event<'_>) -> Option { }; build_wait_line(elapsed, &list("images"), &list("built")) } + Phase::BuildUnreachable => { + let elapsed = ev.detail.and_then(|d| d.get("elapsedSeconds")).and_then(|v| v.as_u64()).unwrap_or(0); + let why = ev.detail.and_then(|d| d.get("message")).and_then(|v| v.as_str()).unwrap_or("no answer"); + format!( + " cannot reach the install to see the build ({} so far): {why}\n \ + the build may still be going on there; looking again (Ctrl+C to stop following)", + elapsed_text(elapsed) + ) + } Phase::InfraProvisionStart => "provisioning infra".to_string(), Phase::DrainWait => { let cap = match ev.detail.and_then(|d| d.get("capSeconds")).and_then(|v| v.as_u64()) { @@ -616,4 +643,39 @@ mod tests { assert_eq!(other["needsTriggerChoice"], false); assert_eq!(other["message"], "project not registered"); } + + /// A request that never reached this machine's daemon says to start + /// it, and flags it for a host that can; one that never reached a + /// remote install names that install, says to check the network, and + /// never offers the daemon. + #[tokio::test] + async fn an_unreachable_install_is_named_by_where_it_is() { + // A port just let go of: nothing listens, the connection is refused. + let closed = std::net::TcpListener::bind("127.0.0.1:0").unwrap().local_addr().unwrap(); + let local = crate::client::DispatcherClient::new(format!("http://{closed}"), None); + let e = local.get_json("/install").await.unwrap_err().context("read the install"); + let detail = error_detail(&e); + assert_eq!(detail["daemonUnreachable"], true); + assert_eq!(detail["installUnreachable"], true); + let message = detail["message"].as_str().unwrap(); + assert!(message.starts_with("the weft daemon is not answering. Start it with `weft daemon start`"), "{message}"); + + let cause = reqwest::Client::new().get(format!("http://{closed}/")).send().await.unwrap_err(); + let remote = crate::client::InstallUnreachable { + url: "https://weft-prod.example.run.app".into(), + target: Some("prod".into()), + cause, + }; + let e = anyhow::Error::new(remote).context("POST https://weft-prod.example.run.app/projects/p/builds").context("ask the install to build"); + let detail = error_detail(&e); + assert_eq!(detail["daemonUnreachable"], false, "nothing on this machine would start it"); + assert_eq!(detail["installUnreachable"], true); + let message = detail["message"].as_str().unwrap(); + assert!( + message.contains("the install `prod` (https://weft-prod.example.run.app) did not answer: "), + "{message}" + ); + assert!(message.ends_with("; check the network and run the command again"), "{message}"); + assert!(!message.contains("weft daemon"), "{message}"); + } } diff --git a/crates/weft-compiler/src/enrich.rs b/crates/weft-compiler/src/enrich.rs index 59bbbe2c..cb527140 100644 --- a/crates/weft-compiler/src/enrich.rs +++ b/crates/weft-compiler/src/enrich.rs @@ -271,6 +271,7 @@ fn gate_inputs() -> [InputDefinition; 2] { // Both, and never switchable: `_should_flow` is the // language's gate, and validate types its constant itself. accepts: Accepts::both(), + required_when_wired: false, widget: None, default: None, label: None, @@ -293,6 +294,7 @@ fn gate_inputs() -> [InputDefinition; 2] { declared_type: None, }, accepts: Accepts::both(), + required_when_wired: false, widget: None, default: None, label: None, @@ -423,6 +425,7 @@ pub fn enrich_collecting( declared_type: None, }, accepts: spec.effective_accepts(meta.is_compiler_read(&spec.name)), + required_when_wired: spec.required_when_wired, // The DECLARED widget only; the type-derived default is // stamped after TypeVar resolution (see the final pass), // so a `T` input resolved to Image gets a file picker. @@ -530,6 +533,22 @@ pub fn enrich_collecting( } } + // An input required only once wired has its requiredness decided + // by its wiring, so a header line making it required always would + // quietly turn "unwired: the node runs without it" into a compile + // error. Declaring it optional (`name?: Type`) says nothing new and + // stays legal. + for wp in &weft_inputs { + if wp.required && catalog_inputs.iter().any(|cp| cp.name == wp.name && cp.required_when_wired) { + errors.push(EnrichError { span: node_span, file: node.source_file.clone(), message: format!( + "node '{}': input '{}' of node type {} is required only when wired, so its \ + wiring decides whether it is required; declare it `{}?` or drop it from \ + the node's inputs", + node.id, wp.name, node.node_type, wp.name, + )}); + } + } + let inputs = merge_ports( &catalog_inputs, &weft_inputs, diff --git a/crates/weft-compiler/src/validate.rs b/crates/weft-compiler/src/validate.rs index 272d5cf0..a9490222 100644 --- a/crates/weft-compiler/src/validate.rs +++ b/crates/weft-compiler/src/validate.rs @@ -3024,7 +3024,7 @@ fn check_warnings(project: &ProjectDefinition, d: &mut Vec) { if !gated && !wired.is_empty() && !node.outputs.is_empty() - && wired.iter().all(|p| !p.required) + && wired.iter().all(|p| !p.is_required(true)) && !node.features.optional_custom_inputs && node.features.one_of_required.is_empty() { diff --git a/crates/weft-compiler/tests/enrich_tests.rs b/crates/weft-compiler/tests/enrich_tests.rs index 8d2e3de2..7e027740 100644 --- a/crates/weft-compiler/tests/enrich_tests.rs +++ b/crates/weft-compiler/tests/enrich_tests.rs @@ -976,3 +976,24 @@ fn a_navigated_into_include_parses_under_the_ids_the_build_uses() { parsed.nodes.iter().map(|n| n.id.as_str()).collect::>() ); } + +/// An input required only once wired has its requiredness decided by its +/// wiring, so a header line declaring it required is refused, naming the +/// fix; declaring it optional says nothing new and stays legal. +#[test] +fn enrich_refuses_making_an_input_required_when_wired_always_required() { + let errors_for = |header: &str| { + let source = format!("\ns = StartInfra({header}) {{ node: \"svc\" }}\n"); + let mut project = compile(&source, uuid::Uuid::new_v4(), CompileFs::none()).expect("compile"); + weft_compiler::enrich::enrich_collecting(&mut project, &catalog(), weft_compiler::enrich::EnrichPolicy::Strict) + .into_iter() + .map(|e| e.message) + .collect::>() + }; + let refused = errors_for("instance: String"); + assert!( + refused.iter().any(|m| m.contains("required only when wired") && m.contains("instance?")), + "{refused:?}" + ); + assert!(errors_for("instance?: String").is_empty(), "optional is what it already is"); +} diff --git a/crates/weft-compiler/tests/validate_tests.rs b/crates/weft-compiler/tests/validate_tests.rs index 26f10d6c..7a31d4d6 100644 --- a/crates/weft-compiler/tests/validate_tests.rs +++ b/crates/weft-compiler/tests/validate_tests.rs @@ -385,6 +385,7 @@ t = Text declared_type: None, }, accepts: weft_core::node::Accepts::both(), + required_when_wired: false, widget: None, default: None, label: None, @@ -417,6 +418,7 @@ t = Text { value: "ok" } declared_type: None, }, accepts: weft_core::node::Accepts::wire_only(), + required_when_wired: false, widget: None, default: None, label: None, @@ -3819,22 +3821,43 @@ fn a_second_response_head_after_a_stream_is_refused() { /// `input_names` with a node: a node naming an infra node by text is held to /// the program's infra nodes and their kind at compile time; a wired -/// name is known only at run time and not checked. +/// name is known only at run time and not checked. With an `instance` it +/// names a `@per_instance` node, without one a shared node (the program's +/// own copy). #[test] fn an_infra_name_is_held_to_the_programs_infra_nodes() { let base = "bridge = BaileyBridge {\n @per_instance\n}\npg = PostgresDatabase { database: \"app\" }\n"; let named = |node: &str| messages(&format!("{base}{node}\n")); let refused = |found: &[String], name: &str| found.iter().any(|m| m.contains(&format!("names '{name}'"))); - assert!(!refused(&named("go = StartInstanceInfra { node: \"bridge\", instance: \"u1\" }"), "bridge")); - assert!(refused(&named("go = StartInstanceInfra { node: \"ghost\", instance: \"u1\" }"), "ghost")); - assert!(refused(&named("go = StopInstanceInfra { node: \"pg\", instance: \"u1\" }"), "pg"), "shared infra is no instance's"); + assert!(!refused(&named("go = StartInfra { node: \"bridge\", instance: \"u1\" }"), "bridge")); + assert!(refused(&named("go = StartInfra { node: \"ghost\", instance: \"u1\" }"), "ghost")); + assert!(refused(&named("go = StopInfra { node: \"pg\", instance: \"u1\" }"), "pg"), "shared infra is no instance's"); + assert!(!refused(&named("go = StopInfra { node: \"pg\" }"), "pg"), "no instance is the program's own copy"); + assert!(refused(&named("go = TerminateInfra { node: \"bridge\" }"), "bridge"), "a per-instance node needs an instance"); + assert!(refused(&named("go = StartInfra { node: \"ghost\" }"), "ghost")); + let given = named("pick = Text { value: \"u1\" }\ngo = StartInfra { node: \"bridge\" }\ngo.instance = pick.value"); + assert!(!refused(&given, "bridge"), "a wired instance is one given: {given:?}"); assert!(!refused(&named("ls = ListInstanceInfra { node: \"pg\" }"), "pg"), "listing takes a shared one"); let wipe = named("w = WipeInstance { instance: \"u1\", infra: [\"bridge\", \"ghost\"] }"); assert!(refused(&wipe, "ghost") && !refused(&wipe, "bridge"), "{wipe:?}"); - let wired = named("pick = Text { value: \"ghost\" }\ngo = StartInstanceInfra { instance: \"u1\" }\ngo.node = pick.value"); + let wired = named("pick = Text { value: \"ghost\" }\ngo = StartInfra { instance: \"u1\" }\ngo.node = pick.value"); assert!(!refused(&wired, "ghost"), "{wired:?}"); } +/// `input_names` with a trigger: `only` names per-instance triggers when +/// the node is given an `instance`, and shared ones (the program's own) +/// when it is not. +#[test] +fn a_trigger_name_follows_whose_triggers_the_node_acts_on() { + let base = "mine = Cron { cron: @instance_filled }\ndaily = Cron { cron: \"0 0 7 * * *\" }\n"; + let named = |node: &str| messages(&format!("{base}{node}\n")); + let refused = |found: &[String], name: &str| found.iter().any(|m| m.contains(&format!("names '{name}'"))); + assert!(!refused(&named("on = ActivateTriggers { instance: \"u1\", only: [\"mine\"] }"), "mine")); + assert!(refused(&named("on = ActivateTriggers { instance: \"u1\", only: [\"daily\"] }"), "daily"), "a shared trigger is no instance's"); + assert!(!refused(&named("off = DeactivateTriggers { only: [\"daily\"] }"), "daily"), "no instance is the program's own triggers"); + assert!(refused(&named("off = DeactivateTriggers { only: [\"mine\"] }"), "mine"), "a per-instance trigger needs an instance"); +} + /// `input_names` with a field: SetInstanceValues keys its `values` and /// lists its `clear` by `node.field`, and each must be a field this /// program writes `@instance_filled`. Renaming the step leaves the old @@ -4681,3 +4704,18 @@ send = BaileySend { bridge: "http://bridge:8090", to: "49151", message: "hi" } let hit = d.iter().find(|e| e.code.as_deref() == Some("input-accepts")); assert!(hit.is_some_and(|e| e.message.contains("`bridge` accepts: wire")), "a written URL must be refused: {d:?}"); } + +/// An input required only once wired is optional while unwired (no +/// `required-port-unmet`), and counts as required once wired (no +/// `no-required-skip`: a closed wire skips the node). +#[test] +fn an_input_required_when_wired_is_optional_unwired_and_required_wired() { + let unwired = parse_enrich("\nsvc = Text { value: \"x\" }\ns = StartInfra { node: \"svc\" }\n"); + let d = validate(&unwired, &catalog()); + assert!(!codes(&d).contains(&"required-port-unmet"), "{d:?}"); + + let wired = parse_enrich("\nid = Text { value: \"ada\" }\ns = StartInfra { node: \"svc\" }\ns.instance = id.value\n"); + let d = validate(&wired, &catalog()); + assert!(!codes(&d).contains(&"required-port-unmet"), "{d:?}"); + assert!(!codes(&d).contains(&"no-required-skip"), "a closed wire on `instance` skips the node: {d:?}"); +} diff --git a/crates/weft-core/src/access/spec.rs b/crates/weft-core/src/access/spec.rs index 13c162bd..cdea4bfc 100644 --- a/crates/weft-core/src/access/spec.rs +++ b/crates/weft-core/src/access/spec.rs @@ -297,6 +297,14 @@ pub struct Permission { /// Starts ticked on the picker. #[serde(default)] pub default: bool, + /// Asked for on every consent, whatever is ticked, and never shown to + /// tick: what the service's test call needs to learn who the account + /// is (Google's `userinfo.email`) when no tickable permission is sure + /// to cover it. Keep it to permissions the provider grants without + /// review and an app allows without setup, since every app and every + /// connection asks for it. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub always: bool, /// This capability creates or reads things INSIDE the credential's /// own account (minted voices, configured agents, phone numbers), /// so a runtime-supplied credential can never serve it: the result @@ -1576,6 +1584,20 @@ impl AccessSpec { p.id )); } + if p.always && !self.needs_browser_consent() { + return Err(format!( + "permission '{}' is `always` (asked for on every consent), but this service \ + connects without a browser consent, so nothing would ever ask for it", + p.id + )); + } + if p.always && (p.default || p.own_only) { + return Err(format!( + "permission '{}' is `always` (asked for on every consent, never ticked), so it \ + cannot also be `default` or `own_only`", + p.id + )); + } } } crate::access::events::validate_topics(&self.service, &self.events)?; @@ -1664,6 +1686,9 @@ impl AccessSpec { // The page is connect-time UI; a stored snapshot has no use for // it (and validation would rightly refuse static + paste). variant.own_page = None; + // A pasted credential is asked for nothing: what a consent asks + // for every time does not apply to it. + variant.permissions.retain(|p| !p.always); Some(variant) } @@ -1708,6 +1733,28 @@ impl AccessSpec { .collect() } + /// The catalogue entries a person ticks: everything but what is asked + /// for always, and the own-account-only capabilities (set up inside + /// the provider's account, never asked for at connect). + // SYNC: AccessSpec::tickable_permissions <-> packages/weft-connect/src/core/recipe.ts tickablePermissions + pub fn tickable_permissions(&self) -> Vec<&Permission> { + self.permissions.iter().filter(|p| !p.always && !p.own_only).collect() + } + + /// What a consent asks for when `ticked` is ticked: the permissions + /// asked for always ([`Permission::always`]), then the ticked ones, + /// each once. + // SYNC: AccessSpec::requested_permissions <-> packages/weft-connect/src/core/recipe.ts requestedPermissions + pub fn requested_permissions(&self, ticked: &[String]) -> Vec { + let mut out: Vec = self.permissions.iter().filter(|p| p.always).map(|p| p.id.clone()).collect(); + for id in ticked { + if !out.contains(id) { + out.push(id.clone()); + } + } + out + } + /// Is `id` in the permission catalogue? pub fn declares_permission(&self, id: &str) -> bool { self.permissions.iter().any(|p| p.id == id) @@ -1744,7 +1791,7 @@ impl AccessSpec { } } - /// The guide's steps with `{permissions}` replaced by the ticked + /// The guide's steps with `{permissions}` replaced by the requested /// permissions' labels (comma-joined), generated per pick rather /// than written as a static blob. Empty when no guide is declared. // SYNC: guide_steps <-> packages/weft-connect/src/core/recipe.ts guideSteps @@ -1752,17 +1799,18 @@ impl AccessSpec { let Some(guide) = self.own_page.as_ref().and_then(|p| p.guide.as_ref()) else { return Vec::new(); }; + let requested = self.requested_permissions(ticked); let labels: Vec<&str> = self .permissions .iter() - .filter(|p| ticked.iter().any(|t| t == &p.id)) + .filter(|p| requested.contains(&p.id)) .map(|p| p.label.as_str()) .collect(); let joined = labels.join(", "); guide.steps.iter().map(|s| s.replace("{permissions}", &joined)).collect() } - /// The guide's link with `{permissions}` replaced by the ticked + /// The guide's link with `{permissions}` replaced by the requested /// permission IDS, urlencoded and comma-joined: a link goes to a /// provider console, which wants machine ids where the steps' /// prose wants labels. The ONE substitution, so no surface renders @@ -1771,7 +1819,7 @@ impl AccessSpec { // SYNC: guide_link <-> packages/weft-connect/src/core/recipe.ts guideLink pub fn guide_link(&self, ticked: &[String]) -> Option { let link = self.own_page.as_ref()?.guide.as_ref()?.link.as_ref()?; - Some(link.replace("{permissions}", &percent_encode(&ticked.join(",")))) + Some(link.replace("{permissions}", &percent_encode(&self.requested_permissions(ticked).join(",")))) } } @@ -2316,6 +2364,45 @@ mod tests { assert!(spec.validate().unwrap_err().contains("no fields")); } + /// A consent asks for the permissions asked for always, then the + /// ticked ones, each once; the guide names them all, since the app + /// has to allow them all. + #[test] + fn a_consent_asks_for_the_always_permissions_and_the_ticked_ones() { + let mut spec = slack_spec(); + spec.permissions[1].always = true; + spec.permissions[1].default = false; + let always = spec.permissions[1].id.clone(); + let first = spec.permissions[0].id.clone(); + assert_eq!(spec.requested_permissions(std::slice::from_ref(&first)), vec![always.clone(), first.clone()]); + assert_eq!(spec.requested_permissions(&[always.clone(), first.clone()]), vec![always.clone(), first]); + assert_eq!(spec.requested_permissions(&[]), vec![always]); + assert!(spec.validate().is_ok()); + } + + /// Only a browser consent asks for anything, so `always` is refused on + /// a service that connects any other way, and a consent service's + /// paste variant (a credential pasted, asked for nothing) drops it. + #[test] + fn always_belongs_to_a_browser_consent() { + let mut keyed: AccessSpec = serde_json::from_value(json!({ + "service": "keyed", + "acquisition": { "kind": "static", "fields": [{ "name": "key", "secret": true }] }, + "permissions": [{ "id": "who", "label": "Who", "description": "Know the account.", "always": true }] + })) + .unwrap(); + assert!(keyed.validate().unwrap_err().contains("without a browser consent")); + keyed.permissions[0].always = false; + assert!(keyed.validate().is_ok()); + + let mut consent = slack_spec(); + consent.permissions[1].always = true; + consent.permissions[1].default = false; + consent.own_page = Some(serde_json::from_value(json!({ "paste": { "fields": [{ "name": "token", "secret": true }] } })).unwrap()); + let pasted = consent.paste_variant().expect("a paste section"); + assert!(pasted.permissions.iter().all(|p| !p.always), "a paste asks for nothing"); + } + #[test] fn validate_rejects_bad_shapes() { let mut spec = slack_spec(); @@ -2328,11 +2415,17 @@ mod tests { label: "Again".into(), description: "Duplicate.".into(), default: false, + always: false, own_only: false, guide: None, }); assert!(dup.validate().unwrap_err().contains("duplicate permission")); + let mut ticked_always = slack_spec(); + ticked_always.permissions[0].always = true; + assert!(ticked_always.permissions[0].default, "slack's first permission starts ticked"); + assert!(ticked_always.validate().unwrap_err().contains("`always`")); + let mut undesc = slack_spec(); undesc.permissions[0].description = String::new(); assert!(undesc.validate().unwrap_err().contains("description")); diff --git a/crates/weft-core/src/activation.rs b/crates/weft-core/src/activation.rs index 24a8bc0f..829910e0 100644 --- a/crates/weft-core/src/activation.rs +++ b/crates/weft-core/src/activation.rs @@ -130,13 +130,15 @@ impl ActivationScope { match (&owner, per_instance) { (Owner::Shared, true) => { return Err(format!( - "trigger '{trigger}' exists once per instance; name which with --instance " + "trigger '{trigger}' exists once per instance; name the instance (`--instance ` on a command, \ + the `instance` input on a node)" )); } (Owner::Instance(instance), false) => { return Err(format!( "trigger '{trigger}' is shared by every instance, so it cannot be activated \ - for instance '{instance}'; leave --instance out" + for instance '{instance}'; name no instance (leave out `--instance` on a command, \ + leave the `instance` input unwired on a node)" )); } _ => keys.push(ActivationKey::new(trigger.clone(), owner.clone())), @@ -396,11 +398,11 @@ mod tests { #[test] fn a_named_trigger_must_fit_the_owner() { let err = ActivationScope { triggers: vec!["receive".into()], instance: None }.resolve(&project()).unwrap_err(); - assert!(err.contains("--instance"), "{err}"); + assert!(err.contains("name the instance"), "{err}"); let err = ActivationScope { triggers: vec!["cron".into()], instance: Some(InstanceId::new("m").unwrap()) } .resolve(&project()) .unwrap_err(); - assert!(err.contains("shared by every instance"), "{err}"); + assert!(err.contains("name no instance"), "{err}"); let err = ActivationScope { triggers: vec!["step".into()], instance: None }.resolve(&project()).unwrap_err(); assert!(err.contains("not a trigger"), "{err}"); let one = ActivationScope { triggers: vec!["cron".into()], instance: None }.resolve(&project()).unwrap(); diff --git a/crates/weft-core/src/builds.rs b/crates/weft-core/src/builds.rs index 093579f0..db29125d 100644 --- a/crates/weft-core/src/builds.rs +++ b/crates/weft-core/src/builds.rs @@ -9,7 +9,8 @@ use serde::{Deserialize, Serialize}; /// What a build produced: the compiled program, its three hashes and the /// implementations its worker carries, and the infra image every infra /// place resolves to. The install registers all of it in one transaction -/// and answers it; downstream verbs name the build by its hashes. +/// and answers it (a 200) once every image the version needs is in the +/// registry; downstream verbs name the build by its hashes. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct BuiltProgram { pub definition: crate::ProjectDefinition, @@ -27,10 +28,6 @@ pub struct BuiltProgram { /// place's row is (`one.db`), which is how the supervisor reads it. #[serde(rename = "infraImages")] pub infra_images: BTreeMap>, - /// The image refs this build had to build (absent from the registry - /// when it began); empty when every image was already there. - #[serde(rename = "builtImages")] - pub built_images: Vec, /// The infra places whose images differ from the ones the build /// before registered: a new copy of one starts on the new image, while /// a copy already running keeps its own until it is upgraded (`weft @@ -96,6 +93,81 @@ impl NodeSet { } } +/// What `POST /projects/{id}/builds` answers, with a 202, when images of +/// the version were missing: the version build now waiting on them, and +/// the image builds it waits on, each one started by this request or +/// already running for the same content. The request answers as soon as +/// they run rather than when they end, so nothing is held open for the +/// minutes a build takes. The install registers the version itself once +/// every image is built; the caller follows it +/// (`GET /projects/{id}/builds/{build}`, [`VersionBuildState`]) until it +/// is registered or failed. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct BuildsUnderway { + /// This version build, as `GET /projects/{id}/builds/{build}` names it. + pub build: uuid::Uuid, + pub images: Vec, +} + +/// On the 409 `POST /projects/{id}/builds` answers when another ask of +/// the project is starting its image builds right now: that ask lets go +/// within moments, so a caller sending its ask again (one whose first +/// ask's answer was lost on the way) may wait and send it again, and its +/// ask then joins the version build the first one started. +pub const BUILD_BUSY_HEADER: &str = "x-weft-build-busy"; + +/// One image build a version waits on. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ImageBuild { + /// The image ref it pushes. + pub image: String, + /// The name weft minted for it. Known from the moment the build is + /// decided, before the builder gives it an id of its own. + pub name: String, +} + +/// Where a version build that waited on image builds stands. +// SYNC: VersionBuildStatus <-> crates/weft-dispatcher/src/build/waiting.rs (version_build.state) +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum VersionBuildStatus { + /// Some image it needs is still building. + Waiting, + /// Every image was built and the version is the project's running one. + Registered, + /// An image's build failed, or the version could not be registered; + /// `reason` says which. + Failed, + /// The project's build was cancelled. + Cancelled, + /// A newer build of the project was asked for before this one was + /// registered, so this one never will be. + Superseded, +} + +/// The answer to `GET /projects/{id}/builds/{build}`: where a version +/// build stands, and each image build it waits on. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct VersionBuildState { + pub state: VersionBuildStatus, + pub images: Vec, + /// What registered, once it did. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub program: Option, + /// Why it failed, was cancelled or superseded. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub reason: Option, +} + +/// One image build of a [`VersionBuildState`]: which, and how it stands. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ImageBuildState { + #[serde(flatten)] + pub build: ImageBuild, + #[serde(flatten)] + pub state: crate::projects::BuildStateResponse, +} + /// `POST /projects/{id}/builds`: build this version of the project. /// What the CLI sends and the install reads. #[derive(Debug, Clone, Serialize, Deserialize)] @@ -131,11 +203,10 @@ mod tests { infra_hash: "i".into(), implementations: [("Llm".to_string(), "h".to_string())].into(), infra_images: [("one.db".to_string(), [("db".to_string(), "r/db:1".to_string())].into())].into(), - built_images: vec!["r/db:1".into()], replaced_infra_images: vec!["one.db".into()], }; let wire = serde_json::to_value(&built).unwrap(); - for key in ["binaryHash", "definitionHash", "infraHash", "infraImages", "builtImages", "replacedInfraImages"] { + for key in ["binaryHash", "definitionHash", "infraHash", "infraImages", "replacedInfraImages"] { assert!(wire.get(key).is_some(), "{key} missing from {wire}"); } let back: BuiltProgram = serde_json::from_value(wire.clone()).unwrap(); diff --git a/crates/weft-core/src/context.rs b/crates/weft-core/src/context.rs index c9c73a25..40705c36 100644 --- a/crates/weft-core/src/context.rs +++ b/crates/weft-core/src/context.rs @@ -101,6 +101,7 @@ pub fn catchable_message(error: &WeftError) -> Option { | WeftError::Type(_) | WeftError::Suspended { .. } | WeftError::Suspension(_) + | WeftError::WaitSkipped(_) | WeftError::Cancelled => None, } } diff --git a/crates/weft-core/src/error.rs b/crates/weft-core/src/error.rs index e9c850ec..04b2cc3d 100644 --- a/crates/weft-core/src/error.rs +++ b/crates/weft-core/src/error.rs @@ -31,6 +31,14 @@ pub enum WeftError { #[error("{0}")] WaitGaveUp(String), + /// A person skipped what the waiting `ctx` call waited on (a form's + /// skip): the step does not run on, it ends skipped with every output + /// closed, the way a step whose input closed does. Returned by the + /// waiting call, so a body passing errors on with `?` needs no code + /// of its own for it. + #[error("{0}")] + WaitSkipped(String), + #[error("cancelled")] Cancelled, diff --git a/crates/weft-core/src/exec/ready.rs b/crates/weft-core/src/exec/ready.rs index 50ccf5c0..666ad1a5 100644 --- a/crates/weft-core/src/exec/ready.rs +++ b/crates/weft-core/src/exec/ready.rs @@ -272,12 +272,7 @@ pub fn find_ready_among<'a>( for ((execution_id, frames), group_pulses) in groups { let out_of_scope = dispatchable.is_some_and(|s| !s.contains(&Located::at(&node.id, &frames))); let wired = wired_inputs(project, program_idx, &node.id, &frames); - let required: HashSet<&str> = node - .inputs - .iter() - .filter(|p| p.required && program_idx.includes_port(node, &frames, &p.name)) - .map(|p| p.name.as_str()) - .collect(); + let required = required_inputs(node, &wired, &frames, program_idx); let literal_filled = literal_filled_ports(node, &wired, &frames, program_idx); if let Some(group) = ready_group_at( node, &group_pulses, execution_id, &frames, &required, &wired, &literal_filled, out_of_scope, project, program_idx, @@ -397,7 +392,7 @@ pub fn kicked_group( // gates still decide scope_permission before this kick dispatches. None if kick.firing => check_flow_permission(node, &[]), None => check_should_skip(node, &effective_refs, - &node.inputs.iter().filter(|p| p.required && program_idx.includes_port(node, frames, &p.name)).map(|p| p.name.as_str()).collect(), + &required_inputs(node, &wired, frames, program_idx), &wired, &literal_filled_ports(node, &wired, frames, program_idx)), }; ReadyGroup { @@ -492,7 +487,7 @@ pub fn firing_input( // (see `check_input`). A mismatch on a required port aggregates // into `type_errors` (the node fails loudly); a mismatch on an // optional port nulls the port and the node proceeds. - let type_errors = check_bag(node, &mut obj); + let type_errors = check_bag(node, wired, &mut obj); let mut closed_ports: Vec = wired .iter() @@ -533,13 +528,13 @@ pub fn firing_input( /// not data: `_should_flow` takes any value and only a `false` says /// no (`check_flow_permission` is its one rule), so it is never held /// to a type here. -fn check_bag(node: &NodeDefinition, obj: &mut InputBag) -> Vec { +fn check_bag(node: &NodeDefinition, wired: &HashSet<&str>, obj: &mut InputBag) -> Vec { let mut errors = Vec::new(); for port in node.inputs.iter().filter(|p| !crate::exec::skip::is_gate_port(&p.name)) { let Some(value) = obj.get(&port.name) else { continue; }; - match check_input(port, value) { + match check_input(port, value, wired.contains(port.name.as_str())) { InputCheck::Ok => {} InputCheck::Fail(err) => { tracing::error!(target: "weft::exec::ready", node = %node.id, "{err}"); @@ -608,6 +603,23 @@ pub fn fill_input_from_literals( } } +/// The inputs whose closure skips `node` at this point (rule 1 of +/// `check_should_skip`): the required ones, and those required only once +/// wired (`requiredWhenWired`) that a wire feeds here. Only ports the +/// run reads count. +pub fn required_inputs<'a>( + node: &'a NodeDefinition, + wired: &HashSet<&str>, + frames: &LoopFrames, + program_idx: &ProgramIndex, +) -> HashSet<&'a str> { + node.inputs + .iter() + .filter(|p| p.is_required(wired.contains(p.name.as_str())) && program_idx.includes_port(node, frames, &p.name)) + .map(|p| p.name.as_str()) + .collect() +} + /// The unwired ports a written constant fills at `frames`: what keeps a /// node with no live wire alive in the skip rules. An `@instance_filled` /// literal counts, since a run is only born once each such field it @@ -685,13 +697,13 @@ pub fn build_kicked_input( // poll interval of zero, a fractional one that would truncate), and // skipping the check here meant those rules bound nothing on the // one path that uses them. - let errors = check_bag(node, &mut obj); + let errors = check_bag(node, &HashSet::new(), &mut obj); (obj, errors) } /// Check one incoming value against its input port type. THE single /// place input type enforcement lives. -fn check_input(port: &crate::project::InputDefinition, value: &Value) -> InputCheck { +fn check_input(port: &crate::project::InputDefinition, value: &Value, wired: bool) -> InputCheck { // A generator port's pulses carry ITEMS: each value is checked // against the ELEMENT type, never against `Generator[T]` itself // (the whole-port handle only exists in the consumer's bag, built @@ -727,8 +739,9 @@ fn check_input(port: &crate::project::InputDefinition, value: &Value) -> InputCh } // The type does not fit. A null is data only where the type admits // it (accepted above); on an optional port a null means "nothing - // arrived" and the bag fills the default. - if value.is_null() && !port.required { + // arrived" and the bag fills the default. A port required once wired + // is not optional when a wire delivered the null. + if value.is_null() && !port.is_required(wired) { return InputCheck::Ok; } // A wrong-typed value is upstream sending something this port @@ -819,6 +832,68 @@ mod tests { assert_eq!(out.dropped.len(), 1); } + /// An input required only once wired: wired and closed, the node + /// skips as for a required input; unwired, or wired with a value, + /// it runs. A wired null fails it, as on a required input, instead + /// of reading as "nothing arrived". + #[test] + fn an_input_required_when_wired_skips_only_when_its_wire_closes() { + let project = |wire_instance: bool| -> ProjectDefinition { + let mut edges = vec![json!({ "id": "e1", "source": "src", "target": "n", + "sourceHandle": "data", "targetHandle": "data" })]; + if wire_instance { + edges.push(json!({ "id": "e2", "source": "src", "target": "n", + "sourceHandle": "id", "targetHandle": "instance" })); + } + serde_json::from_value(json!({ + "id": "00000000-0000-0000-0000-000000000000", + "name": "p", "description": null, + "nodes": [ + { "id": "src", "nodeType": "S", "label": null, "config": null, "position": { "x": 0.0, "y": 0.0 }, + "inputs": [], "outputs": [ + { "name": "id", "portType": "String", "required": true }, + { "name": "data", "portType": "String", "required": true }], + "features": {}, "scope": [], "groupBoundary": null, "requiresInfra": false, "images": [] }, + { "id": "n", "nodeType": "N", "label": null, "config": null, "position": { "x": 0.0, "y": 0.0 }, + "inputs": [ + { "name": "instance", "portType": "String", "required": false, "requiredWhenWired": true }, + { "name": "data", "portType": "String", "required": false }], + "outputs": [], + "features": {}, "scope": [], "groupBoundary": null, "requiresInfra": false, "images": [] } + ], + "edges": edges, "groups": [], + "createdAt": "1970-01-01T00:00:00Z", "updatedAt": "1970-01-01T00:00:00Z" + })) + .unwrap() + }; + let execution_id = uuid::Uuid::nil(); + let ready = |project: &ProjectDefinition, instance: Option| { + let mut table = crate::pulse::PulseTable::default(); + let bucket = table.entry("n".into()).or_default(); + bucket.push(data(execution_id, vec![], "n", "data", json!("d"))); + bucket.extend(instance); + let index = ProgramIndex::build(project); + let mut found = super::find_ready_nodes(project, &table, &index, None); + assert_eq!(found.len(), 1, "the node's group is formed"); + found.remove(0).1 + }; + let wired = project(true); + + let closed = ready(&wired, Some(Pulse::closure(uuid::Uuid::new_v4(), execution_id, vec![], "n", "instance"))); + assert_eq!(closed.skip, Some(crate::exec::skip::SkipReason::RequiredInputClosed { port: "instance".into(), failure: None })); + + let given = ready(&wired, Some(data(execution_id, vec![], "n", "instance", json!("ada")))); + assert_eq!(given.skip, None, "a value on the wire runs the node"); + assert!(given.received.type_errors.is_empty()); + + let null = ready(&wired, Some(data(execution_id, vec![], "n", "instance", json!(null)))); + assert_eq!(null.skip, None); + assert_eq!(null.received.type_errors.len(), 1, "a wired null is not 'nothing arrived': {:?}", null.received.type_errors); + + let unwired = ready(&project(false), None); + assert_eq!(unwired.skip, None, "unwired, the node runs without it"); + } + /// A firing sees only the pulses at its own frame stack: the group /// is formed by exact `(execution_id, frames)`, so a shallower pulse is /// never in its view. @@ -880,8 +955,8 @@ mod tests { // gate once had its own infer-based check beside // `accepts_runtime_value` and failed every Named input. let p = port("Profile={ name: String, age: Number, nickname?: String }", true); - assert_eq!(check_input(&p, &json!({"name": "Ada", "age": 36})), InputCheck::Ok); - match check_input(&p, &json!({"name": "Ada"})) { + assert_eq!(check_input(&p, &json!({"name": "Ada", "age": 36}), false), InputCheck::Ok); + match check_input(&p, &json!({"name": "Ada"}), false) { InputCheck::Fail(msg) => assert!(msg.contains("Profile"), "names the type: {msg}"), other => panic!("missing required field must fail, got {other:?}"), } @@ -889,15 +964,15 @@ mod tests { #[test] fn matching_value_is_ok_regardless_of_required() { - assert_eq!(check_input(&port("String", true), &json!("ok")), InputCheck::Ok); - assert_eq!(check_input(&port("String", false), &json!("ok")), InputCheck::Ok); + assert_eq!(check_input(&port("String", true), &json!("ok"), false), InputCheck::Ok); + assert_eq!(check_input(&port("String", false), &json!("ok"), false), InputCheck::Ok); } #[test] fn null_is_data_only_where_the_type_admits_it() { - assert_eq!(check_input(&port("String | Null", true), &json!(null)), InputCheck::Ok); - assert_eq!(check_input(&port("String", false), &json!(null)), InputCheck::Ok, "optional: absent"); - match check_input(&port("String", true), &json!(null)) { + assert_eq!(check_input(&port("String | Null", true), &json!(null), false), InputCheck::Ok); + assert_eq!(check_input(&port("String", false), &json!(null), false), InputCheck::Ok, "optional: absent"); + match check_input(&port("String", true), &json!(null), false) { InputCheck::Fail(msg) => assert!(msg.contains("expected String, got Null"), "{msg}"), other => panic!("a required String has nothing to run with on null, got {other:?}"), } @@ -921,9 +996,9 @@ mod tests { "name": "count", "portType": "Number", "required": true, "widget": { "kind": "number", "min": 1, "max": 8, "step": 1 } })).unwrap(); - assert_eq!(check_input(&count, &json!(3)), InputCheck::Ok); + assert_eq!(check_input(&count, &json!(3), false), InputCheck::Ok); for bad in [json!(0), json!(9), json!(2.5)] { - match check_input(&count, &bad) { + match check_input(&count, &bad, false) { InputCheck::Fail(msg) => assert!(msg.contains("'count'"), "{msg}"), other => panic!("{bad} must be refused by the widget, got {other:?}"), } @@ -934,8 +1009,8 @@ mod tests { "name": "mode", "portType": "String", "required": false, "widget": { "kind": "select", "options": ["added", "removed", "both"] } })).unwrap(); - assert_eq!(check_input(&mode, &json!("both")), InputCheck::Ok); - match check_input(&mode, &json!("sideways")) { + assert_eq!(check_input(&mode, &json!("both"), false), InputCheck::Ok); + match check_input(&mode, &json!("sideways"), false) { InputCheck::Fail(msg) => assert!(msg.contains("'mode'") && msg.contains("'added'"), "{msg}"), other => panic!("a value outside the options must be refused, got {other:?}"), } @@ -950,23 +1025,23 @@ mod tests { "name": "n", "portType": "Number", "required": true, "widget": { "kind": "number", "min": 1, "step": 2 } })).unwrap(); - assert_eq!(check_input(&every_two, &json!(3)), InputCheck::Ok); - assert_eq!(check_input(&every_two, &json!(4)), InputCheck::Ok); - assert!(matches!(check_input(&every_two, &json!(2.5)), InputCheck::Fail(_))); + assert_eq!(check_input(&every_two, &json!(3), false), InputCheck::Ok); + assert_eq!(check_input(&every_two, &json!(4), false), InputCheck::Ok); + assert!(matches!(check_input(&every_two, &json!(2.5), false), InputCheck::Fail(_))); let tenths: InputDefinition = serde_json::from_value(json!({ "name": "temperature", "portType": "Number", "required": false, "widget": { "kind": "number", "min": 0, "max": 2, "step": 0.1 } })).unwrap(); for fine in [json!(0.85), json!(1.0), json!(0.07)] { - assert_eq!(check_input(&tenths, &fine), InputCheck::Ok, "{fine}"); + assert_eq!(check_input(&tenths, &fine, false), InputCheck::Ok, "{fine}"); } - assert!(matches!(check_input(&tenths, &json!(2.5)), InputCheck::Fail(_)), "the range still binds"); + assert!(matches!(check_input(&tenths, &json!(2.5), false), InputCheck::Fail(_)), "the range still binds"); } #[test] fn mismatch_on_required_fails() { let p = port("String", true); - match check_input(&p, &json!(42)) { + match check_input(&p, &json!(42), false) { InputCheck::Fail(msg) => assert!(msg.contains("expected")), other => panic!("expected Fail, got {other:?}"), } @@ -977,11 +1052,11 @@ mod tests { /// bug on any port, and the firing fails. #[test] fn mismatch_on_optional_fails_too() { - match check_input(&port("String", false), &json!(42)) { + match check_input(&port("String", false), &json!(42), false) { InputCheck::Fail(msg) => assert!(msg.contains("type mismatch"), "{msg}"), other => panic!("expected Fail, got {other:?}"), } - assert_eq!(check_input(&port("String", false), &json!(null)), InputCheck::Ok, "a null on an optional port is nothing arrived"); + assert_eq!(check_input(&port("String", false), &json!(null), false), InputCheck::Ok, "a null on an optional port is nothing arrived"); } /// A required nullable port used to swallow a wrong-typed value as @@ -989,12 +1064,12 @@ mod tests { /// "arrived wrong". Only an OPTIONAL port drops a value. #[test] fn a_wrong_type_on_a_required_nullable_port_fails() { - match check_input(&port("String | Null", true), &json!(42)) { + match check_input(&port("String | Null", true), &json!(42), false) { InputCheck::Fail(msg) => assert!(msg.contains("type mismatch"), "{msg}"), other => panic!("expected Fail, got {other:?}"), } // A null itself is still data there. - assert_eq!(check_input(&port("String | Null", true), &json!(null)), InputCheck::Ok); + assert_eq!(check_input(&port("String | Null", true), &json!(null), false), InputCheck::Ok); } /// A connection picked in the source is a stored handle until the @@ -1008,7 +1083,7 @@ mod tests { "widget": { "kind": "access", "service": "slack" } })).unwrap(); let handle = json!({ "id": "11111111-1111-1111-1111-111111111111", "identity": "someone" }); - assert_eq!(check_input(&account, &handle), InputCheck::Ok); + assert_eq!(check_input(&account, &handle, false), InputCheck::Ok); // A resource pick is stored the same way, on a port typed for // the id the node ends up reading. let sheet: InputDefinition = serde_json::from_value(json!({ @@ -1016,12 +1091,12 @@ mod tests { "widget": { "kind": "remote_select", "access": "account", "sources": [] } })).unwrap(); assert_eq!( - check_input(&sheet, &json!({ "id": "1AbC", "label": "Budget" })), + check_input(&sheet, &json!({ "id": "1AbC", "label": "Budget" }), false), InputCheck::Ok ); // A pasted raw id is the port's own type and still checked. - assert_eq!(check_input(&sheet, &json!("1AbC")), InputCheck::Ok); - assert!(matches!(check_input(&sheet, &json!(7)), InputCheck::Fail(_))); + assert_eq!(check_input(&sheet, &json!("1AbC"), false), InputCheck::Ok); + assert!(matches!(check_input(&sheet, &json!(7), false), InputCheck::Fail(_))); let node = kicked_node("SlackAccess", vec![account], json!({ "account": handle.clone() })); let (input, refusals) = build_kicked_input(&node, None); assert!(refusals.is_empty(), "{refusals:?}"); diff --git a/crates/weft-core/src/exec/skip.rs b/crates/weft-core/src/exec/skip.rs index 1077e9d6..d82577f0 100644 --- a/crates/weft-core/src/exec/skip.rs +++ b/crates/weft-core/src/exec/skip.rs @@ -84,6 +84,9 @@ pub enum SkipReason { /// its `_should_flow` said no, or a loop's list never came. Every /// node inside a gated scope carries this, however deep. ScopeSkipped { scope: String }, + /// A person skipped what the node was waiting for (a form's skip): + /// it ended there, with nothing to send on. + WaitSkipped, } impl SkipReason { @@ -98,7 +101,7 @@ impl SkipReason { /// failure), so it carries none. pub fn inherited_failure(&self) -> Option<&Failure> { match self { - Self::DidNotFlow | Self::DidFlow | Self::ScopeSkipped { .. } => None, + Self::DidNotFlow | Self::DidFlow | Self::ScopeSkipped { .. } | Self::WaitSkipped => None, Self::WatchedNodeFailed { failure } => Some(failure), Self::FlowClosed { failure } | Self::RequiredInputClosed { failure, .. } @@ -142,6 +145,7 @@ impl std::fmt::Display for SkipReason { Self::ScopeSkipped { scope } => { write!(f, "the scope '{scope}' it lives in did not run") } + Self::WaitSkipped => write!(f, "a person skipped what it was waiting for"), } } } @@ -262,6 +266,8 @@ pub fn check_should_skip( } // Rule 1: any wired required port that arrived as a closure -> skip. + // `required` is built by `ready::required_inputs`, which counts a + // `requiredWhenWired` input as required exactly where it is wired. // (`literal_filled` never overlaps `wired`: wires are authoritative // and config only fills unwired ports, so no config check here.) // Generator ports are exempt: their closure is the empty stream, diff --git a/crates/weft-core/src/live_event.rs b/crates/weft-core/src/live_event.rs index 0444faad..2a93340d 100644 --- a/crates/weft-core/src/live_event.rs +++ b/crates/weft-core/src/live_event.rs @@ -124,7 +124,16 @@ pub enum DispatcherEvent { at_unix: u64, }, NodeSuspended { execution_id: ExecutionId, node: String, frames: LoopFrames, token: String, #[serde(default, skip_serializing_if = "Option::is_none")] inherited_from: Option, project_id: uuid::Uuid, at_unix: u64 }, - NodeResumed { execution_id: ExecutionId, node: String, frames: LoopFrames, token: Option, value: Option, #[serde(default, skip_serializing_if = "Option::is_none")] inherited_from: Option, project_id: uuid::Uuid, at_unix: u64 }, + /// `token`: the wait the firing resumed from, `None` for a resume no + /// wait drove (a boundary fired again). How that wait ended is its + /// [`Self::WaitEnded`]. + NodeResumed { execution_id: ExecutionId, node: String, frames: LoopFrames, token: Option, #[serde(default, skip_serializing_if = "Option::is_none")] inherited_from: Option, project_id: uuid::Uuid, at_unix: u64 }, + /// A wait of the firing at `node` and `frames` ended, the first ending + /// on record: its answer, a person skipping it, or the run giving it + /// up. Its own event because a wait the run holds in its worker never + /// suspends or resumes the firing, so `NodeSuspended` and `NodeResumed` + /// see only the waits that paused the run. + WaitEnded { execution_id: ExecutionId, node: String, frames: LoopFrames, token: String, ended: crate::primitive::AwaitEnd, #[serde(default, skip_serializing_if = "Option::is_none")] inherited_from: Option, project_id: uuid::Uuid, at_unix: u64 }, NodeCancelled { execution_id: ExecutionId, node: String, frames: LoopFrames, reason: String, #[serde(default, skip_serializing_if = "Option::is_none")] inherited_from: Option, project_id: uuid::Uuid, at_unix: u64 }, NodeCompleted { execution_id: ExecutionId, node: String, frames: LoopFrames, output: serde_json::Value, #[serde(default, skip_serializing_if = "Option::is_none")] inherited_from: Option, project_id: uuid::Uuid, at_unix: u64 }, NodeFailed { execution_id: ExecutionId, node: String, frames: LoopFrames, error: String, #[serde(default, skip_serializing_if = "Option::is_none")] inherited_from: Option, project_id: uuid::Uuid, at_unix: u64 }, @@ -376,6 +385,7 @@ impl DispatcherEvent { | Self::NodeStarted { project_id, .. } | Self::NodeSuspended { project_id, .. } | Self::NodeResumed { project_id, .. } + | Self::WaitEnded { project_id, .. } | Self::NodeCancelled { project_id, .. } | Self::NodeCompleted { project_id, .. } | Self::NodeFailed { project_id, .. } @@ -418,6 +428,7 @@ impl DispatcherEvent { Self::NodeStarted { frames, .. } | Self::NodeSuspended { frames, .. } | Self::NodeResumed { frames, .. } + | Self::WaitEnded { frames, .. } | Self::NodeCancelled { frames, .. } | Self::NodeCompleted { frames, .. } | Self::NodeFailed { frames, .. } @@ -477,6 +488,7 @@ impl DispatcherEvent { | Self::NodeStarted { execution_id, .. } | Self::NodeSuspended { execution_id, .. } | Self::NodeResumed { execution_id, .. } + | Self::WaitEnded { execution_id, .. } | Self::NodeCancelled { execution_id, .. } | Self::NodeCompleted { execution_id, .. } | Self::NodeFailed { execution_id, .. } diff --git a/crates/weft-core/src/node.rs b/crates/weft-core/src/node.rs index 135edfe9..fd1f874b 100644 --- a/crates/weft-core/src/node.rs +++ b/crates/weft-core/src/node.rs @@ -1033,6 +1033,20 @@ impl NodeMetadata { input.name )); } + if input.required_when_wired { + let problem = if input.required { + Some("is also `required`; pick one (`required` skips on a closed value wired or not)") + } else if input.input_type.as_generator().is_some() { + Some("is a stream, and a stream input is always `required`") + } else if !input.effective_accepts(self.is_compiler_read(&input.name)).wire { + Some("takes no wire, so it could never be wired") + } else { + None + }; + if let Some(problem) = problem { + return Err(format!("input '{}': `requiredWhenWired` {problem}", input.name)); + } + } // A widget implies the SHAPE of the value it edits. On an // input whose type cannot hold that shape, the widget's own // contract (a number widget's min/max, a checkbox's bool) @@ -2769,6 +2783,16 @@ pub struct InputSpec { pub input_type: WeftType, #[serde(default)] pub required: bool, + /// Optional while nothing is wired here, required once something is: + /// an unwired input lets the node run without it (no compile error), + /// and a wired one that arrives closed skips the node exactly as a + /// required input does. For an input whose absence has a meaning of + /// its own (an `instance` left unwired acts on the program's own + /// copies) that a wire which never delivered must not fall back to. + /// Never together with `required`; the metadata check refuses it. + // SYNC: InputSpec.required_when_wired <-> packages/weft-graph/src/protocol.ts InputSpec.requiredWhenWired + #[serde(default, rename = "requiredWhenWired", skip_serializing_if = "std::ops::Not::not")] + pub required_when_wired: bool, /// Which drivers this input takes, when the author narrows it. /// Absent means both (a constant in the source, or a wire from /// another node); `["wire"]` refuses every written value, `["literal"]` @@ -4438,9 +4462,37 @@ mod input_semantics_tests { description: None, requires_scopes: None, requires_values: None, + required_when_wired: false, } } + /// `requiredWhenWired` is a third requiredness beside `required` and + /// optional, so it never sits with `required`, and it is refused where + /// it could never mean anything: a stream (always required) or an + /// input that takes no wire. + #[test] + fn required_when_wired_is_refused_where_it_cannot_apply() { + let string = || WeftType::primitive(WeftPrimitive::String); + let ok = metadata_with(vec![InputSpec { required_when_wired: true, ..input("instance", string()) }]); + assert!(ok.validate_semantics().is_ok()); + + let both = metadata_with(vec![InputSpec { required: true, required_when_wired: true, ..input("instance", string()) }]); + assert!(both.validate_semantics().unwrap_err().contains("also `required`")); + + let stream = metadata_with(vec![InputSpec { + required_when_wired: true, + ..input("rows", WeftType::Generator(Box::new(string()))) + }]); + assert!(stream.validate_semantics().unwrap_err().contains("stream")); + + let literal_only = metadata_with(vec![InputSpec { + required_when_wired: true, + accepts: Some(Accepts::literal_only()), + ..input("instance", string()) + }]); + assert!(literal_only.validate_semantics().unwrap_err().contains("takes no wire")); + } + /// Every input takes both drivers unless something narrows it: the /// author's list, a live-handle type, or the compiler-read fixed rule. #[test] diff --git a/crates/weft-core/src/node_test.rs b/crates/weft-core/src/node_test.rs index c6c0535b..ae4494a7 100644 --- a/crates/weft-core/src/node_test.rs +++ b/crates/weft-core/src/node_test.rs @@ -405,6 +405,7 @@ pub fn fixture_spec(name: &str, label: &str, description: &str) -> InputSpec { description: Some(description.to_string()), requires_scopes: None, requires_values: None, + required_when_wired: false, } } @@ -1152,6 +1153,14 @@ impl FakeRig { self.state.signals.lock().unwrap().push_back(AwaitEnd::GaveUp { error }); } + /// Make the node's next `ctx.await_signal` end the way a person + /// skipping it does (the `/skip` door, `WeftError::WaitSkipped`), + /// queued in order with [`Self::signal`]: in production the step then + /// ends skipped with every output closed. + pub fn signal_skipped(&self) { + self.state.signals.lock().unwrap().push_back(AwaitEnd::Skipped); + } + /// Make every run on this rig a run for this instance, what /// `ctx.instance()` answers, the way an instance's firing, instance token or /// `Weft-Instance` header would in production. A blank or malformed id @@ -2266,6 +2275,7 @@ impl ContextHandle for TestHandle { match ended { AwaitEnd::Answered { value } => Ok(value), AwaitEnd::GaveUp { error } => Err(WeftError::WaitGaveUp(error)), + AwaitEnd::Skipped => Err(crate::primitive::wait_skipped()), } } diff --git a/crates/weft-core/src/primitive.rs b/crates/weft-core/src/primitive.rs index 1b516381..16753440 100644 --- a/crates/weft-core/src/primitive.rs +++ b/crates/weft-core/src/primitive.rs @@ -298,12 +298,11 @@ pub struct ExecutionSnapshot { /// the engine has consumed the kick (the node started at that /// location; further kicks at the same location are a no-op). pub kicked: HashMap, - /// Fires that arrived for live suspensions but haven't been - /// consumed by a worker's node completion yet. The worker - /// seeds these into its link on startup so every waiting node - /// finds its value when re-dispatched. Survives worker restarts - /// because it's derived from journal events, not slot queues. - pub pending_deliveries: HashMap, + /// Wait endings on record before their wait's registration (an + /// answer the dispatcher wrote while nobody drove the run): the + /// registration takes its ending from here, so the two rows fold the + /// same in either order. + pub pending_deliveries: HashMap, /// Per-(node, frames) ordered sequence of past `await_signal` /// calls. Each entry has the call_index (0-based ordinal of /// the call within the body), the token, and either the @@ -428,6 +427,7 @@ pub enum AwaitedEntryKind { /// How a past `ctx.await_signal` ended. The first ending on record /// stands: an answer that comes after a wait was given up is ignored. +// SYNC: AwaitEnd <-> extension-vscode/src/execFollower.ts AwaitEnd #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] #[serde(tag = "type", rename_all = "snake_case")] pub enum AwaitEnd { @@ -436,6 +436,34 @@ pub enum AwaitEnd { /// The run held it in its worker and gave it up /// (`SuspensionGaveUp`): the call failed with `error`. GaveUp { error: String }, + /// A person skipped it (`SuspensionSkipped`): the call ends its step + /// skipped ([`crate::error::WeftError::WaitSkipped`]). + Skipped, +} + +/// What answers a wait: a value, or a person skipping it (the `/skip` +/// door). One type from the door to the waiting call, so a skip is never +/// spelled as a value a real answer could also be. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +pub enum WaitAnswer { + Given { value: Value }, + Skipped, +} + +impl WaitAnswer { + /// How the wait it answers ends. + pub fn end(&self) -> AwaitEnd { + match self { + Self::Given { value } => AwaitEnd::Answered { value: value.clone() }, + Self::Skipped => AwaitEnd::Skipped, + } + } +} + +/// The error a waiting call returns when a person skipped it. +pub fn wait_skipped() -> crate::error::WeftError { + crate::error::WeftError::WaitSkipped("a person skipped what this step was waiting for".into()) } /// What a body's `ctx.await_signal` finds in its journal at its call @@ -474,6 +502,7 @@ pub fn replay_await( AwaitedEntryKind::Await { token, ended } => match ended { Some(AwaitEnd::Answered { value }) => Ok(Some(ReplayedAwait::Resolved(value))), Some(AwaitEnd::GaveUp { error }) => Err(crate::error::WeftError::WaitGaveUp(error)), + Some(AwaitEnd::Skipped) => Err(wait_skipped()), None => Ok(Some(ReplayedAwait::Pending { token })), }, AwaitedEntryKind::Run { name, .. } => Err(crate::error::WeftError::NodeExecution(format!( diff --git a/crates/weft-core/src/project.rs b/crates/weft-core/src/project.rs index f81ec4b8..e8cf3048 100644 --- a/crates/weft-core/src/project.rs +++ b/crates/weft-core/src/project.rs @@ -637,6 +637,12 @@ pub struct InputDefinition { // SYNC: InputDefinition.accepts <-> packages/weft-graph/src/protocol.ts InputDefinition.accepts, packages/weft-connect/src/core/wire.ts InputDefinition.accepts #[serde(default = "Accepts::both")] pub accepts: Accepts, + /// Optional while unwired, required once wired (mirrored from the + /// metadata's `requiredWhenWired`): a wire into it that arrives + /// closed skips the node. [`Self::is_required`] is the one reading. + // SYNC: InputDefinition.required_when_wired <-> packages/weft-connect/src/core/wire.ts InputDefinition.requiredWhenWired + #[serde(default, rename = "requiredWhenWired", skip_serializing_if = "std::ops::Not::not")] + pub required_when_wired: bool, /// The input's effective editor widget (declared, else derived from /// the RESOLVED instance type after TypeVar substitution). Always /// present after enrich. @@ -698,6 +704,13 @@ impl PortDefinition { } impl InputDefinition { + /// Whether a closed value on this input skips the node, given + /// whether a wire feeds it where the node fires: always for a + /// required input, and for a `requiredWhenWired` one only when wired. + pub fn is_required(&self, wired: bool) -> bool { + self.required || (wired && self.required_when_wired) + } + /// An input for a pure WIRE port (a boundary passthrough side, a /// source-declared custom port): drivers from the type, no editor /// surface beyond what enrich later stamps. @@ -705,6 +718,7 @@ impl InputDefinition { Self { accepts: Accepts::for_type(&port.port_type), port, + required_when_wired: false, widget: None, default: None, label: None, @@ -1780,6 +1794,7 @@ mod project_wire_tests { declared_type: None, }, accepts: Accepts::wire_only(), + required_when_wired: false, widget: None, default: None, label: None, diff --git a/crates/weft-core/src/project/selection.rs b/crates/weft-core/src/project/selection.rs index 109c48bf..c44af16e 100644 --- a/crates/weft-core/src/project/selection.rs +++ b/crates/weft-core/src/project/selection.rs @@ -262,7 +262,8 @@ pub struct ShutGate { /// needed by, or `None` when nothing that needs it lies on its path. /// /// A node answers for itself: its input is needed when it is required -/// with no default, or when it belongs to a `@require_one_of` set whose +/// (or required once wired, which a port reached along a wire is) with +/// no default, or when it belongs to a `@require_one_of` set whose /// every other member gets nothing (`fed(place, port)` says whether an /// input of the node at `place` gets a value some other way). A group, a /// call site or a body boundary requires nothing of its own (its only @@ -294,7 +295,9 @@ pub fn required_consumer( let Some(node) = project.node(&place.id) else { continue }; let start = start.or_else(|| is_start(&place).then(|| (place.clone(), port.clone()))); let input = node.inputs.iter().find(|p| p.name == port); - if input.is_some_and(|input| input.required && input.default.is_none()) { + // The walk only ever reaches a port along a wire, so an input + // required once wired counts as required here. + if input.is_some_and(|input| input.is_required(true) && input.default.is_none()) { return Some(Need { needer: (place, port), start, one_of: None }); } let last_of = node.features.one_of_required.iter() diff --git a/crates/weft-core/src/projects.rs b/crates/weft-core/src/projects.rs index 4a3ae500..acc3dbd5 100644 --- a/crates/weft-core/src/projects.rs +++ b/crates/weft-core/src/projects.rs @@ -302,8 +302,8 @@ pub struct BuildInFlight { pub log_url: Option, } -/// Where one image build of a project is: going on, or how it ended -/// (`GET /projects/{id}/builds/{build}`, `build` the builder's id). +/// Where one image build is: going on, or how it ended +/// ([`crate::builds::ImageBuildState`]). // SYNC: BuildState <-> crates/weft-dispatcher/src/build/ledger.rs (image_build.status) #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] @@ -314,11 +314,23 @@ pub enum BuildState { Cancelled, } -/// The answer to `GET /projects/{id}/builds/{build}`: `None` when no such -/// build of the project is on record. -#[derive(Debug, Clone, Serialize, Deserialize)] +/// Everything a caller following one image build is told about it. +/// `state` is `None` when that build is no longer on record (a cleanup +/// forgot its image, or another build of the same image replaced it). +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] pub struct BuildStateResponse { pub state: Option, + /// Why it failed, once it did. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub reason: Option, + /// The builder's own id for it, once the builder made it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub build: Option, + #[serde(default, rename = "startedAtUnix", skip_serializing_if = "Option::is_none")] + pub started_at_unix: Option, + /// Where its log is read, when the builder keeps one at an address. + #[serde(default, rename = "logUrl", skip_serializing_if = "Option::is_none")] + pub log_url: Option, } /// One trigger's runs in the last minute or two (`ProjectStatusResponse::runs`). diff --git a/crates/weft-core/src/run_spec.rs b/crates/weft-core/src/run_spec.rs index 6c853c2a..150abe0b 100644 --- a/crates/weft-core/src/run_spec.rs +++ b/crates/weft-core/src/run_spec.rs @@ -10,6 +10,7 @@ use serde::{Deserialize, Serialize}; use serde_json::Value; use crate::frames::{Located, LoopFrames}; +use crate::primitive::AwaitEnd; use crate::project::selection::{source_place, RunSelection, SelectionBounds}; use crate::project::ProjectDefinition; use crate::pulse::Failure; @@ -191,21 +192,91 @@ impl<'de> Deserialize<'de> for UniqueValue { } } -/// A person's answer to a human step, as recorded at freeze time: +/// How one wait of a run ended, as recorded at freeze time: the +/// person's answer, their skip, or the run giving the wait up. /// `question` is what the node showed the person, so whoever answers on -/// a new run reads old question, old answer, new question. The runtime -/// never replays answers; Tangle plays the person. +/// a new run reads old question, old ending, new question. The runtime +/// never replays answers; Tangle plays the person. The file spells a +/// given answer `"payload": `, a skip `"skipped": true`, and a +/// wait given up `"gave_up": ""`, exactly one of the three +/// ([`AnswerFile`]). +// SYNC: Answer <-> packages/weft-graph/src/run-spec.ts Answer #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] -#[serde(deny_unknown_fields)] +#[serde(try_from = "AnswerFile", into = "AnswerFile")] pub struct Answer { pub node: String, - #[serde(default)] pub frames: LoopFrames, - pub payload: Value, - #[serde(default, skip_serializing_if = "Option::is_none")] + /// How the wait ended: end it the same way on a new run. + pub ended: AwaitEnd, pub question: Option, } +/// [`Answer`] as the file holds it, both ways. `payload` is the field +/// every frozen example has had since the first; `skipped` came with +/// skips and `gave_up` with waits given up, and exactly one of the +/// three is present. `deny_unknown_fields` and the defaults live here, +/// because with `try_from` serde never reads attributes on [`Answer`] +/// itself. +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct AnswerFile { + node: String, + #[serde(default)] + frames: LoopFrames, + /// `Some(Null)` is a given null; `None` is no payload at all. + #[serde(default, deserialize_with = "present", skip_serializing_if = "Option::is_none")] + payload: Option, + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + skipped: bool, + /// The error the waiting step failed with when the run gave the wait up. + #[serde(default, skip_serializing_if = "Option::is_none")] + gave_up: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + question: Option, +} + +/// A field that is there, `null` included. +fn present<'de, D: serde::Deserializer<'de>>(deserializer: D) -> Result, D::Error> { + Value::deserialize(deserializer).map(Some) +} + +impl TryFrom for Answer { + type Error = String; + + // SYNC: answer refusals <-> packages/weft-graph/src/run-spec.ts parseRunSpec answers + fn try_from(f: AnswerFile) -> Result { + let ended = match (f.payload, f.skipped, f.gave_up) { + (Some(value), false, None) => AwaitEnd::Answered { value }, + (None, true, None) => AwaitEnd::Skipped, + (None, false, Some(error)) => AwaitEnd::GaveUp { error }, + (None, false, None) => return Err(format!( + "the answer to '{}' says nothing about how its wait ended; give the value the person answered (\"payload\"), \"skipped\": true if they skipped it, or \"gave_up\" with the error if the run gave it up", + f.node + )), + (payload, skipped, gave_up) => { + let said: Vec<&str> = [(payload.is_some(), "a payload"), (skipped, "\"skipped\": true"), (gave_up.is_some(), "\"gave_up\"")] + .into_iter().filter(|(present, _)| *present).map(|(_, name)| name).collect(); + return Err(format!( + "the answer to '{}' says its wait ended more than one way ({}); a wait ends once, so keep only one", + f.node, said.join(" and ") + )); + } + }; + Ok(Self { node: f.node, frames: f.frames, ended, question: f.question }) + } +} + +impl From for AnswerFile { + fn from(a: Answer) -> Self { + let (payload, skipped, gave_up) = match a.ended { + AwaitEnd::Answered { value } => (Some(value), false, None), + AwaitEnd::Skipped => (None, true, None), + AwaitEnd::GaveUp { error } => (None, false, Some(error)), + }; + Self { node: a.node, frames: a.frames, payload, skipped, gave_up, question: a.question } + } +} + /// Where a frozen example's `expected` came from. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] @@ -1116,6 +1187,51 @@ mod tests { use super::*; use serde_json::json; + /// A given answer is written `payload` (a null one included), a skip + /// `"skipped": true` and a wait given up `"gave_up": ""`, each + /// with nothing else; each reads back as itself. + #[test] + fn a_wait_ending_is_a_payload_a_skip_or_a_give_up_on_disk() { + let cases = [ + (AwaitEnd::Answered { value: json!({ "answer": "yes" }) }, json!({ "node": "review", "frames": [], "payload": { "answer": "yes" } })), + (AwaitEnd::Answered { value: Value::Null }, json!({ "node": "review", "frames": [], "payload": null })), + (AwaitEnd::Skipped, json!({ "node": "review", "frames": [], "skipped": true })), + (AwaitEnd::GaveUp { error: "node 'review' gave up its wait: quiet".into() }, json!({ "node": "review", "frames": [], "gave_up": "node 'review' gave up its wait: quiet" })), + ]; + for (ended, file) in cases { + let answer = Answer { node: "review".into(), frames: vec![], ended, question: None }; + assert_eq!(serde_json::to_value(&answer).unwrap(), file); + assert_eq!(serde_json::from_value::(file).unwrap(), answer); + } + } + + /// A file saying two endings at once, or none, is refused by name; + /// unknown fields still are. + #[test] + fn a_wait_ending_is_exactly_one_of_payload_skipped_or_gave_up() { + for (two, named) in [ + (json!({ "node": "review", "payload": null, "skipped": true }), "a payload and \"skipped\": true"), + (json!({ "node": "review", "payload": 1, "gave_up": "quiet" }), "a payload and \"gave_up\""), + (json!({ "node": "review", "skipped": true, "gave_up": "quiet" }), "\"skipped\": true and \"gave_up\""), + ] { + let error = serde_json::from_value::(two).unwrap_err().to_string(); + assert!(error.contains("ended more than one way") && error.contains(named), "{error}"); + } + for neither in [json!({ "node": "review" }), json!({ "node": "review", "skipped": false })] { + let error = serde_json::from_value::(neither).unwrap_err(); + assert!(error.to_string().contains("says nothing about how its wait ended"), "{error}"); + } + assert!(serde_json::from_value::(json!({ "node": "review", "payload": 1, "stray": 1 })).is_err(), "unknown fields still refused"); + } + + /// An answer frozen before skips and give-ups existed (`node`, + /// `frames`, `payload`, `question`) reads as the given answer it was. + #[test] + fn an_answer_frozen_with_only_a_payload_still_reads() { + let old: Answer = serde_json::from_str(r#"{"node":"review","frames":[],"payload":"yes","question":"Continue?"}"#).unwrap(); + assert_eq!(old, Answer { node: "review".into(), frames: vec![], ended: AwaitEnd::Answered { value: json!("yes") }, question: Some(json!("Continue?")) }); + } + #[test] fn fire_bake_requires_full_identity_and_an_actual_capture() { let program = crate::project::hash::ProgramIdentity { @@ -1405,7 +1521,7 @@ mod tests { fn flat_wire_shape_retains_frozen_evidence_and_rejects_old_fields() { let spec = RunSpec { from: BTreeMap::from([("b".into(), BTreeMap::new())]), before: vec!["c".into()], caller: vec![json!({"text": "hello"})], - answers: vec![Answer { node: "b".into(), frames: vec![], payload: json!("yes"), question: Some(json!("Continue?")) }], + answers: vec![Answer { node: "b".into(), frames: vec![], ended: AwaitEnd::Answered { value: json!("yes") }, question: Some(json!("Continue?")) }], ..RunSpec::whole("x") }; assert_eq!(serde_json::from_value::(serde_json::to_value(&spec).unwrap()).unwrap(), spec); for field in ["scope", "kicks", "provided", "emitted", "input", "bogus"] { diff --git a/crates/weft-dispatcher/src/api/infra.rs b/crates/weft-dispatcher/src/api/infra.rs index ffeb1c9d..d7ab77f0 100644 --- a/crates/weft-dispatcher/src/api/infra.rs +++ b/crates/weft-dispatcher/src/api/infra.rs @@ -154,13 +154,19 @@ pub(crate) fn resolve_infra_nodes( (true, None) => { return Err(( StatusCode::BAD_REQUEST, - format!("infra node '{node}' exists once per instance; name whose copy with --instance "), + format!( + "infra node '{node}' exists once per instance; name the instance whose copy to act on \ + (`--instance ` on a command, the `instance` input on a node)" + ), )); } (false, Some(m)) => { return Err(( StatusCode::BAD_REQUEST, - format!("infra node '{node}' is shared by every instance, so there is no copy of it for '{m}'; leave --instance out"), + format!( + "infra node '{node}' is shared by every instance, so there is no copy of it for '{m}'; name no \ + instance (leave out `--instance` on a command, leave the `instance` input unwired on a node)" + ), )); } _ => { diff --git a/crates/weft-dispatcher/src/api/mod.rs b/crates/weft-dispatcher/src/api/mod.rs index 9fa86cd3..83bca6b2 100644 --- a/crates/weft-dispatcher/src/api/mod.rs +++ b/crates/weft-dispatcher/src/api/mod.rs @@ -254,7 +254,10 @@ fn core_routes(cors: CorsLayer, state: DispatcherState) -> Router Router Router Router { with_hops(outside_caller_routes(state.clone()), hops).with_state(state) } +/// Whether `e` failed because the database is out of reach for now +/// (`weft_task_store::db::unreachable`), anywhere in its chain. +pub(crate) fn database_unreachable(e: &anyhow::Error) -> bool { + e.chain().filter_map(|cause| cause.downcast_ref::()).any(weft_task_store::db::unreachable) +} + +/// The answer to a read that failed doing `what`: a 503 the caller asks +/// again on when the database is out of reach ([`database_unreachable`]), +/// a 500 naming the failure otherwise, which asking again would not fix. +pub(crate) fn unavailable_or_internal(what: &str, e: anyhow::Error) -> (axum::http::StatusCode, String) { + if database_unreachable(&e) { + tracing::warn!(target: "weft_dispatcher::api", error = %format!("{e:#}"), "could not {what}: the database is out of reach"); + return (axum::http::StatusCode::SERVICE_UNAVAILABLE, format!("the install could not {what} right now; try again shortly")); + } + tracing::error!(target: "weft_dispatcher::api", error = %format!("{e:#}"), "could not {what}"); + (axum::http::StatusCode::INTERNAL_SERVER_ERROR, format!("could not {what}: {e:#}")) +} + +#[cfg(test)] +mod unavailable_tests { + use axum::http::StatusCode; + + #[test] + fn only_an_unreachable_database_asks_the_caller_again() { + let pool_gone = anyhow::Error::from(sqlx::Error::PoolTimedOut).context("read the build"); + assert_eq!(super::unavailable_or_internal("read the build", pool_gone).0, StatusCode::SERVICE_UNAVAILABLE); + let undecodable = anyhow::anyhow!("a version build state the ledger does not know: 'odd'"); + assert_eq!(super::unavailable_or_internal("read the build", undecodable).0, StatusCode::INTERNAL_SERVER_ERROR); + let bad_row = anyhow::Error::from(sqlx::Error::RowNotFound); + assert_eq!(super::unavailable_or_internal("read the build", bad_row).0, StatusCode::INTERNAL_SERVER_ERROR); + } +} + #[cfg(test)] mod token_door_tests { #[test] diff --git a/crates/weft-dispatcher/src/api/project.rs b/crates/weft-dispatcher/src/api/project.rs index 84e67c6e..57a90fdf 100644 --- a/crates/weft-dispatcher/src/api/project.rs +++ b/crates/weft-dispatcher/src/api/project.rs @@ -275,9 +275,10 @@ pub(crate) async fn declare_project( .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("empty program: {e}")))?; let summary = state .projects - .register_with_hashes(empty, name, "", tenant.as_str(), None, None, None, None, None, None) + .register_with_hashes(empty, name, "", tenant.as_str(), None, None, None, None, None, None, None) .await - .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("declare the project: {e}")))?; + .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("declare the project: {e}")))? + .ok_or((StatusCode::INTERNAL_SERVER_ERROR, "declaring the project registered nothing".to_string()))?; state .events .publish(DispatcherEvent::ProjectRegistered { @@ -289,12 +290,23 @@ pub(crate) async fn declare_project( } /// `POST /projects/{id}/builds`: build this version of the project inside -/// the install (`crate::build`) and register what it produced: the -/// compiled program, its three hashes, the implementations its worker +/// the install (`crate::build`). +/// +/// When every image the version needs is in the registry, register what +/// it produced and answer it (200, [`weft_core::builds::BuiltProgram`]): +/// the compiled program, its three hashes, the implementations its worker /// carries, the infra images every place runs, and the version's files as /// the program's source. The project's running pointers move to it in the /// same transaction, so the next verb acts on exactly this build. /// +/// When images are missing, start their builds (or join the ones already +/// running for the same content), write the version down as waiting on +/// them, and answer at once (202, [`weft_core::builds::BuildsUnderway`]), +/// never after the minutes the builds take: a request held open that long +/// with nothing on the wire is dropped by an idle network path. The +/// install registers the version itself once they all ended +/// (`crate::build::waiting`); the caller follows it (`version_build`). +/// /// This is the ONE way a project's program and images come to exist on an /// install: nothing the client computed is believed. pub async fn build( @@ -302,69 +314,57 @@ pub async fn build( caller: CallerTenant, Path(id): Path, Json(req): Json, -) -> Result, (StatusCode, String)> { - declare_project(&state, id, &req.name, &caller.0).await?; - crate::api::versions::validate_manifest(&req.manifest)?; +) -> Result { + use axum::response::IntoResponse; + declare_project(&state, id, &req.name, &caller.0).await.map_err(IntoResponse::into_response)?; + crate::api::versions::validate_manifest(&req.manifest).map_err(IntoResponse::into_response)?; + // Taken as the ask arrives, before the minutes of compiling: a + // version registers only over an older ask, whichever finishes first + // (`crate::build::waiting`). + let ask = state + .projects + .next_build_ask(id) + .await + .map_err(|e| crate::api::unavailable_or_internal("take the build's place in the project's order", e).into_response())?; // Claims every image the build plans from its first look at the - // registry until the registration commits, so no prune deletes an - // image this build found or built before anything references it + // registry until the version registers, so no prune deletes an image + // this build found or built before anything references it // (`crate::build::prune::ImageHold`). let hold = crate::build::prune::ImageHold::new(&state.pg_pool, id); - let crate::build::Build { program: mut built, images } = - crate::transition::build_version_gated(&state, id, &caller.0, &req, &hold).await?; - hold.confirm().await.map_err(|e| (StatusCode::CONFLICT, format!("{e:#}")))?; - // Read before registering: which infra places this build moves onto - // another image, for the caller to say so. - let before = state - .projects - .running_infra_image_tags(id) - .await - .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("read the registered infra images: {e:#}")))? - .unwrap_or_default(); - built.replaced_infra_images = crate::build::replaced_infra_images(&before, &built.infra_images); - let infra_images: crate::project_store::InfraImageTags = built - .infra_images - .iter() - .map(|(place, images)| (place.clone(), images.iter().map(|(k, v)| (k.clone(), v.clone())).collect())) - .collect(); - let summary = state - .projects - .register_with_hashes( - built.definition.clone(), - &req.name, - "", - caller.0.as_str(), - Some(&built.binary_hash), - Some(&built.definition_hash), - Some(&built.infra_hash), - Some(&infra_images), - Some(&built.implementations), - Some(&req.manifest), + let (version, hold) = match crate::transition::build_version_gated(&state, id, &caller.0, &req, ask, hold).await? { + crate::build::VersionBuild::Waiting(underway) => { + return Ok((StatusCode::ACCEPTED, Json(underway)).into_response()); + } + crate::build::VersionBuild::Ready { version, hold } => (version, hold), + }; + if let Err(e) = hold.confirm().await { + let released = hold.release().await; + let settled = crate::transition::settle(&state, Some(id)).await; + let also: Vec = [released.err(), settled.err()].into_iter().flatten().map(|e| format!("{e:#}")).collect(); + let mut message = format!("{e:#}"); + if !also.is_empty() { + message.push_str(&format!("\n(and letting go of the build failed too: {})", also.join("; "))); + } + return Err((StatusCode::CONFLICT, message).into_response()); + } + let registrar = crate::build::waiting::Registrar::Request { hold }; + match crate::build::waiting::register(&state, id, caller.0.as_str(), &version, registrar).await { + Ok(Some(built)) => Ok(Json(built).into_response()), + Ok(None) => Err(( + StatusCode::CONFLICT, + "a newer build of this project registered while this one was compiling, so this one was not registered; \ + build again to run this version" + .to_string(), ) - .await - .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("register the build: {e}")))?; - // Still under the hold: what the project now runs is how recent each - // of its images is when a prune picks what to reclaim. - crate::build::ledger::note_running(&state.pg_pool, id, &images, crate::lease::now_unix()) - .await - .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("{e:#}")))?; - // Let go only now that the registration above has committed: a prune - // deciding about one of these images under its lock then sees either - // this claim or the committed `running_binary_hash` / infra tags - // (`crate::build::ledger::begin_reclaim`), never neither. - hold.release().await.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("{e:#}")))?; - // The images this project's earlier builds left, bounded now rather - // than whenever somebody cleans: its current one and the one before - // stay (`crate::build::prune::AfterBuildPrunes`). - state.builder.prunes.request(&state, Some(id)); - state - .events - .publish(DispatcherEvent::ProjectRegistered { - project_id: summary.id, - name: weft_core::truncate_user_string(&summary.name, 4096), - }) - .await; - Ok(Json(built)) + .into_response()), + Err(e) => { + let mut message = format!("register the build: {e:#}"); + if let Err(settled) = crate::transition::settle(&state, Some(id)).await { + message.push_str(&format!("\n(and settling the project failed too: {settled:#})")); + } + Err((StatusCode::INTERNAL_SERVER_ERROR, message).into_response()) + } + } } /// Answers through `StatusError` so "no project you may see under this @@ -464,6 +464,10 @@ pub async fn remove( crate::build::ledger::forget_project(&state.pg_pool, id) .await .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("{e:#}")))?; + // Its version builds go too, a waiting one's claim with it. + crate::build::waiting::forget_project(&state.pg_pool, id) + .await + .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("{e:#}")))?; state.builder.prunes.request(&state, None); // Reclaim the project's instance-specific stored data through the ONE hook. // The default frees the project's `project/`-scoped runtime files; a reclaimer @@ -2484,19 +2488,25 @@ async fn prepare_trigger_setup( } } -/// Where one of the project's image builds is (`build`, the builder's id, -/// as the status names it): what a person watching the build is told when -/// an image stops building. -pub async fn build_state( +/// `GET /projects/{id}/builds/{build}`: where the version build a 202 from +/// the build request named stands (`build`), with each image build it +/// waits on and, once registered, what registered, or why it ended +/// otherwise. What the caller following a build reads until it ends. +pub async fn version_build( State(state): State, caller: CallerTenant, - Path((id, build)): Path<(uuid::Uuid, String)>, -) -> Result, (StatusCode, String)> { + Path((id, build)): Path<(uuid::Uuid, uuid::Uuid)>, +) -> Result, (StatusCode, String)> { authorize_project(&state, &caller.0, id).await?; - let state = crate::build::ledger::state_of(&state.pg_pool, id, &build) + // The database out of reach is the install unable to serve this right + // now, never the build's state: a 503, which the caller following the + // build reads as "look again". Anything else (a row that no longer + // reads) is a 500 asking again would not fix. + crate::build::waiting::state(&state.pg_pool, id, build) .await - .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("{e:#}")))?; - Ok(Json(weft_core::projects::BuildStateResponse { state })) + .map_err(|e| crate::api::unavailable_or_internal("read where the build is", e))? + .map(Json) + .ok_or((StatusCode::NOT_FOUND, format!("no build {build} of this project"))) } pub async fn activate( @@ -3690,13 +3700,15 @@ pub async fn cancel_running( Ok(StatusCode::NO_CONTENT) } -/// `POST /projects/{id}/cancel-build`. Cancel the in-flight build -/// transition: CAS `transition` building → cancelling_build, the -/// durable, cross-process cancel signal the driving process's build gate polls -/// between looks at its build processes (it stops them on seeing it). -/// Cancel reconciles, never asserts: the response is 202 and the -/// displayed state is whatever the backend reports next (the build -/// may still complete if it beat the cancel). +/// `POST /projects/{id}/cancel-build`. Cancel the in-flight build: +/// `transition` → cancelling_build, then every image build the project +/// started is stopped and its waiting version ends cancelled +/// (`VersionBuilder::cancel_project`), and the transition lands at rest +/// once nothing holds it (`crate::transition::settle`). A build request +/// still starting its builds is refused its next claim, and stops what it +/// started itself. Cancel reconciles, never asserts: the +/// response is 202 and the displayed state is whatever the backend +/// reports next (a build may still complete if it beat the cancel). /// /// 412 when no build is in flight (stale tab; the client refetches /// `/status` and reconciles). @@ -3718,6 +3730,14 @@ pub async fn cancel_build( )); } crate::transition::publish_transition_changed(&state, id).await; + state + .builder + .cancel_project(id) + .await + .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("stop the project's builds: {e:#}")))?; + crate::transition::settle(&state, Some(id)) + .await + .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("settle the build transition: {e:#}")))?; Ok(StatusCode::ACCEPTED) } diff --git a/crates/weft-dispatcher/src/api/signal.rs b/crates/weft-dispatcher/src/api/signal.rs index 95d3f256..2b0b3783 100644 --- a/crates/weft-dispatcher/src/api/signal.rs +++ b/crates/weft-dispatcher/src/api/signal.rs @@ -98,25 +98,33 @@ pub async fn fire_signal( fire_signal_inner(&state, &token, &caller, payload).await } -/// `POST /signal/{token}/skip`. Resume the suspended firing with a -/// null payload. Sibling firings of the same execution keep going; the -/// skipped firing wakes, downstream null-propagation decides what -/// happens (most nodes auto-skip on null inputs). +/// `POST /signal/{token}/skip`. A person declines what a waiting run +/// asked: the waiting step ends skipped, every output of it closed, and +/// what reads them skips in turn. Sibling firings of the same execution +/// keep going. Only a waiting run's question can be skipped; an entry +/// has nothing to decline. The answer is weft's own +/// (`WaitAnswer::Skipped`), never a value the signal's kind processes, +/// so no kind can mistake it for an answer. /// /// Auth: signal token alone (knowing it = permission to skip). /// Same auth model as fire: a consumer that can answer the form /// can also refuse to answer it. -/// -/// Implementation: thin wrapper around fire_signal with body=null. -/// No special engine path; the engine sees a normal -/// SuspensionResolved with value=null and unwinds via existing -/// null-propagation rules. pub async fn skip_signal( State(state): State, - caller: crate::api::CallerAddress, Path(token): Path, ) -> axum::response::Response { - fire_signal_inner(&state, &token, &caller, Value::Null).await + use axum::response::IntoResponse; + let routing = match lookup_signal_routing(&state, &token).await { + Ok(r) => r, + Err(e) => return e.into_response(), + }; + if routing.surface_kind == "internal" { + return (StatusCode::NOT_FOUND, "internal signal kind has no public surface").into_response(); + } + if !routing.is_resume { + return (StatusCode::CONFLICT, "only a question a waiting run asked can be skipped; this signal starts runs").into_response(); + } + answer_run(&state, &token, &routing, weft_core::primitive::WaitAnswer::Skipped).await.into_response() } async fn fire_signal_inner( @@ -241,7 +249,7 @@ pub(crate) async fn take_event( .await .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("listener dispatch: {e:#}")))?; match outcome.target { - ProcessTarget::Resume { .. } => Ok(answer_run(state, token, routing, outcome.value).await?), + ProcessTarget::Resume { .. } => Ok(answer_run(state, token, routing, weft_core::primitive::WaitAnswer::Given { value: outcome.value }).await?), ProcessTarget::Entry => { let fire = weft_core::door_fire::DoorFire { token: token.to_string(), @@ -286,11 +294,11 @@ pub(crate) async fn answer_run( state: &DispatcherState, token: &str, routing: &FireGateInfo, - value: Value, + answer: weft_core::primitive::WaitAnswer, ) -> Result { match routing.standing().arrival(crate::lease::now_unix()) { weft_core::arrival::Arrival::Live => { - let answered = state.journal.answer(token, &value).await.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("answer: {e:#}")))?; + let answered = state.journal.answer(token, &answer).await.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("answer: {e:#}")))?; match answered { crate::journal::Answered::Reached { consumed } => { // The row is gone now, so the listener only learns of @@ -311,7 +319,7 @@ pub(crate) async fn answer_run( // there is no run to hand it to now, or later. weft_core::arrival::Arrival::Refused => Err((StatusCode::GONE, "This no longer takes answers.".into())), weft_core::arrival::Arrival::Wait => { - let waiting = weft_task_store::parked_fires::waiting(uuid::Uuid::new_v4(), value, None, 0, None); + let waiting = weft_task_store::parked_fires::waiting_answer(uuid::Uuid::new_v4(), answer); // The shared park names its refusal; never swallow one under a // 200. match park(&state.pg_pool, token, &waiting, None).await.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("park: {e}")))? { diff --git a/crates/weft-dispatcher/src/app.rs b/crates/weft-dispatcher/src/app.rs index a1be072e..8794449b 100644 --- a/crates/weft-dispatcher/src/app.rs +++ b/crates/weft-dispatcher/src/app.rs @@ -155,7 +155,7 @@ pub const DISPATCHER_CHANNELS: &[&str] = &[ crate::held::WORKER_SETTINGS_CHANNEL, crate::held::INFRA_STATUS_CHANNEL, crate::held::ACCESS_CHANNEL, - crate::build::follow::BUILD_STARTED_CHANNEL, + crate::build::follow::BUILD_CLAIMED_CHANNEL, ]; /// What the dispatcher is built from: the install's config and what the @@ -222,7 +222,6 @@ pub async fn build_state(settings: DispatcherSettings<'_>, defaults: Defaults) - images, pool: pool.clone(), compile_lanes: config.build.compile_lanes, - poll_every: weft_core::time_scale::scaled(std::time::Duration::from_secs(2)), prunes: Default::default(), blobs: crate::build::blob_cache::BlobCache::in_temp_dir(), }); diff --git a/crates/weft-dispatcher/src/authenticator.rs b/crates/weft-dispatcher/src/authenticator.rs index 665fa86f..292df5c1 100644 --- a/crates/weft-dispatcher/src/authenticator.rs +++ b/crates/weft-dispatcher/src/authenticator.rs @@ -224,8 +224,10 @@ pub const NO_SUCH_PROJECT: &str = "this dispatcher holds no project under that i /// Authorize a caller against a project: the project must exist AND belong to /// the caller's tenant. Returns the same `NOT_FOUND` for "no such project" and /// "exists but belongs to another tenant" so a caller cannot probe which -/// project ids exist in other tenants (no existence leak); `INTERNAL_SERVER_ERROR` -/// only on a real store failure. +/// project ids exist in other tenants (no existence leak); `SERVICE_UNAVAILABLE` +/// only when the database is out of reach right now, which a caller may +/// try again, and `INTERNAL_SERVER_ERROR` for any other failure +/// (`crate::api::unavailable_or_internal`). /// /// This is the single gate every user-facing, project-scoped handler calls /// before acting. It builds on `ProjectStore::tenant_for` (the project to tenant @@ -241,18 +243,7 @@ pub async fn authorize_project( Ok(Some(owner)) if owner == caller.as_str() => Ok(()), // Missing OR cross-tenant: indistinguishable to the caller. Ok(_) => Err((StatusCode::NOT_FOUND, NO_SUCH_PROJECT.to_string())), - Err(e) => { - tracing::warn!( - target: "weft_dispatcher::auth", - project_id = %id, - error = %e, - "tenant_for failed during authorization" - ); - Err(( - StatusCode::INTERNAL_SERVER_ERROR, - "authorization failed".to_string(), - )) - } + Err(e) => Err(crate::api::unavailable_or_internal("read who owns this project", e.context(format!("project {id}")))), } } @@ -285,16 +276,10 @@ pub async fn authorize_execution( Ok(Some(o)) => o, Ok(None) => return Err((StatusCode::NOT_FOUND, "not found".to_string())), Err(e) => { - tracing::warn!( - target: "weft_dispatcher::auth", - execution_id = %execution_id, - error = %e, - "execution_owner failed during authorization" - ); - return Err(( - StatusCode::INTERNAL_SERVER_ERROR, - "authorization failed".to_string(), - )); + return Err(crate::api::unavailable_or_internal( + "read who owns this execution", + e.context(format!("execution {execution_id}")), + )) } }; if owner.tenant != caller.as_str() { diff --git a/crates/weft-dispatcher/src/build/follow.rs b/crates/weft-dispatcher/src/build/follow.rs index 1565d14d..6c945328 100644 --- a/crates/weft-dispatcher/src/build/follow.rs +++ b/crates/weft-dispatcher/src/build/follow.rs @@ -1,12 +1,16 @@ //! Seeing a build through: asking the builder how each running build is -//! doing and recording how it ended. +//! doing and recording how it ended, then registering each waiting +//! version whose builds all ended (`super::waiting`). //! -//! Anyone who looks moves a build forward ([`advance`]): the verb waiting -//! on it, and otherwise the dispatcher's build loop ([`drain_loop`]), -//! whichever dispatcher wakes. A build starting wakes the loop -//! ([`BUILD_STARTED_CHANNEL`]), which then looks every [`look_every`] -//! until no build runs, and sleeps after. No process holds a build, so -//! none has to stay up for it to finish. +//! The dispatcher's build loop ([`drain_loop`]) does it, whichever +//! dispatcher wakes. A build claimed, or a version written down as +//! waiting, wakes the loop ([`BUILD_CLAIMED_CHANNEL`]), which then looks +//! every [`look_every`] while a build runs or a version waits, and sleeps +//! after. It watches a build from its claim, before it is made on the +//! builder, so a start whose dispatcher went away is ended as soon as its +//! project's build heartbeat goes stale (`ledger::end_lost_starts`). No +//! process holds a build, and the request that started one answered as +//! soon as it ran, so none has to stay up for it to finish. use anyhow::Result; use sqlx::PgPool; @@ -22,38 +26,48 @@ fn look_every() -> std::time::Duration { weft_core::time_scale::scaled(std::time::Duration::from_secs(5)) } -/// The channel a build announces on once it runs on its builder -/// ([`ledger::started`]); the payload is its image. -pub const BUILD_STARTED_CHANNEL: &str = "weft_build_started"; +/// The channel a build announces on the moment its row says it runs +/// ([`ledger::claim`]), and a version the moment it waits +/// (`super::waiting::ask`); the payload is the image, or the version's id. +pub const BUILD_CLAIMED_CHANNEL: &str = "weft_build_claimed"; -pub(crate) static WAKE_ON: &[WakeOn] = &[WakeOn::any(BUILD_STARTED_CHANNEL)]; +pub(crate) static WAKE_ON: &[WakeOn] = &[WakeOn::any(BUILD_CLAIMED_CHANNEL)]; -/// Ask the builder about the build running for `image_ref` and record its -/// end when it ended. Errs only when the ledger cannot be read or -/// written: a builder that does not answer is looked at again, and given -/// up on (the build ends failed) once it has not answered for -/// [`ledger::UNANSWERED_GIVE_UP_SECS`]. -pub async fn advance(pool: &PgPool, images: &dyn ImageBuilder, image_ref: &str) -> Result<()> { - let Some(build) = ledger::running_for(pool, image_ref).await? else { return Ok(()) }; - advance_build(pool, images, &build).await +/// One look at every running build: the starts nobody drives any more +/// ended (`ledger::end_lost_starts`), then each asked about and its end +/// recorded when it ended. Answers whether any build was running. One +/// build's failure, or a failed sweep, never skips the others; the +/// failures are answered together after. +pub async fn look(pool: &PgPool, images: &dyn ImageBuilder) -> Result { + let now = crate::lease::now_unix(); + let mut failed = Vec::new(); + let stale_before = now - crate::transition::heartbeat_stale_secs(); + let swept = match pool.acquire().await { + Ok(mut conn) => ledger::end_lost_starts(&mut conn, ledger::LostStarts::Undriven { stale_before }, now).await, + Err(e) => Err(e.into()), + }; + if let Err(e) = swept { + failed.push(format!("ending the starts nobody drives: {e:#}")); + } + let builds = ledger::running(pool, None).await?; + for build in &builds { + if let Err(e) = advance_build(pool, images, build).await { + failed.push(format!("{}: {e:#}", build.image_ref)); + } + } + anyhow::ensure!(failed.is_empty(), "could not move these builds forward: {}", failed.join("; ")); + Ok(!builds.is_empty()) } +/// Ask the builder about `build` and record its end when it ended. Errs +/// only when the ledger cannot be read or written: a builder that does not +/// answer is looked at again, and given up on (the build ends failed) once +/// it has not answered for [`ledger::UNANSWERED_GIVE_UP_SECS`]. async fn advance_build(pool: &PgPool, images: &dyn ImageBuilder, build: &ledger::RunningBuild) -> Result<()> { let now = crate::lease::now_unix(); - let Some(builder_id) = &build.builder_id else { - // Not made on the builder yet: nothing to ask. A start that never - // recorded the build's id there went with its starter. - if now > build.held_until { - let outcome = ledger::Outcome::Failed(format!( - "the build {} was never made on the builder: its start did not finish within {} seconds (the \ - dispatcher starting it went away). Build again", - build.build_name, - ledger::START_HOLD_SECS - )); - ledger::finish(pool, &build.image_ref, &build.build_name, outcome, now).await?; - } - return Ok(()); - }; + // Not made on the builder yet: nothing to ask. Its starter still + // drives it, or `ledger::end_lost_starts` ended it. + let Some(builder_id) = &build.builder_id else { return Ok(()) }; let handle = BuildHandle::named(builder_id.clone()); let status = match images.poll(&handle).await { Ok(status) => { @@ -96,22 +110,31 @@ async fn advance_build(pool: &PgPool, images: &dyn ImageBuilder, build: &ledger: Ok(()) } -/// The dispatcher's build loop: every running build moved forward, then a -/// look again soon while any still runs. One dispatcher at a time -/// (`crate::reaper`'s install-wide lock), so the builder is asked once per -/// look, not once per replica. +/// The dispatcher's build loop: every running build moved forward, then +/// every waiting version whose builds ended registered or ended +/// (`super::waiting::advance`), then every project whose build transition +/// nothing holds any more let out of it (`crate::transition::settle`), then +/// a look again soon while a build runs or a version waits. One dispatcher +/// at a time (`crate::reaper`'s install-wide lock), so the builder is asked +/// once per look, not once per replica. pub fn drain_loop(state: &DispatcherState) -> DrainLoop { crate::reaper::woken(state, WAKE_ON, "image_builds", |state| async move { - let builds = ledger::running(&state.pg_pool, None).await?; - // One build's failure never skips the others; the failures are - // answered together after. - let mut failed = Vec::new(); - for build in &builds { - if let Err(e) = advance_build(&state.pg_pool, state.builder.images.as_ref(), build).await { - failed.push(format!("{}: {e:#}", build.image_ref)); + let ran = look(&state.pg_pool, state.builder.images.as_ref()).await; + // Advanced whatever the look answered: a build whose end was + // recorded before another failed still lets its version register. + let advanced = super::waiting::advance(&state.pg_pool, state.projects.as_ref()).await; + if let Ok(advanced) = &advanced { + for summary in &advanced.registered { + super::waiting::announce(&state, summary).await; } } - anyhow::ensure!(failed.is_empty(), "could not move these builds forward: {}", failed.join("; ")); - Ok(if builds.is_empty() { DrainStep::Done } else { DrainStep::RetryIn(look_every()) }) + crate::transition::settle(&state, None).await?; + let advanced = advanced?; + anyhow::ensure!( + advanced.failed.is_empty(), + "could not move these versions forward: {}", + advanced.failed.join("; ") + ); + Ok(if ran? || advanced.waiting { DrainStep::RetryIn(look_every()) } else { DrainStep::Done }) }) } diff --git a/crates/weft-dispatcher/src/build/ledger.rs b/crates/weft-dispatcher/src/build/ledger.rs index eef91d3c..2f6f3803 100644 --- a/crates/weft-dispatcher/src/build/ledger.rs +++ b/crates/weft-dispatcher/src/build/ledger.rs @@ -4,26 +4,25 @@ //! The build runs on the platform's builder (a `docker build` on a local //! install, a Cloud Build on GCP). The row is how everybody finds it: a //! second verb needing the same image joins the running build instead of -//! starting another, and every verb waiting on it reads its end here. +//! starting another, and the version build waiting on it reads its end +//! here (`super::waiting`, `image_states`). //! //! The row is the truth about a build, and anyone may move it forward: //! the dispatcher's build loop (`super::follow`) asks the builder how //! each running build is doing and records how it ended, on whichever //! dispatcher wakes next. No process has to stay up for a build to be //! seen through, which a dispatcher that scales to zero could not. -//! -//! A waiter follows the row, not one name: whatever ends there is the -//! answer, since every build of one ref pushes the same content. use anyhow::{Context, Result}; use sqlx::PgPool; -/// How long a build just claimed may take to be made on the builder -/// (`started` records its id there). Until it is, nobody can ask the -/// builder about it; a start still not recorded after this is taken for -/// one whose starter went away, and the build ends failed. Generous on -/// purpose: a start stages the whole build context first. -pub const START_HOLD_SECS: i64 = 1800; +/// What a claim writes into `image_build.held_until`, for a dispatcher of +/// an older release still up beside this one (a rollout): it ends a start +/// not made on the builder by then. Nothing here reads the column any +/// more: a start whose starter went away is noticed from its project's +/// build heartbeat instead (`end_lost_starts`), within a minute. The column +/// is dropped in a release after this one, once no older reader is left. +const OLDER_READERS_START_HOLD_SECS: i64 = 1800; /// How long the builder may keep failing to answer about a build before /// it is given up on, ending failed with the last answer. An internal wait @@ -36,7 +35,7 @@ pub const CLAIM_LEASE_SECS: i64 = 60; pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { name: "image_build", - tables: &["image_build", "image_use", "image_claim"], + tables: &["image_build", "image_use", "image_claim", "version_build"], ddl: &[ r#"CREATE TABLE IF NOT EXISTS image_build ( -- The content-addressed ref the build pushes: one build per @@ -57,7 +56,8 @@ pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { -- SYNC: the statuses <-> weft_core::projects::BuildState status TEXT NOT NULL CHECK (status IN ('running', 'succeeded', 'failed', 'cancelled')), reason TEXT, - -- Until when its start may still record `builder_id`. + -- Read only by dispatchers of an older release (see + -- OLDER_READERS_START_HOLD_SECS); dropped in a later release. held_until BIGINT NOT NULL, started_at BIGINT NOT NULL, finished_at BIGINT, @@ -83,20 +83,23 @@ pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { // One row per image a build in progress relies on before its // project's running version names it (`claim_images`): every image // it found in the registry or is building. A prune spares every - // ref with a live claim. A claim lives as long as the build request - // holding it renews it (`super::prune::ImageHold`), so one whose - // dispatcher is gone stops protecting anything. + // ref with a live claim (`claim_live`). A claim lives as long as + // the build request holding it renews it (`super::prune::ImageHold`), + // and, once the request answered while its builds ran, as long as + // the version build it became waits (`super::waiting`: the claim's + // id is that version's). One nobody keeps stops protecting anything. r#"CREATE TABLE IF NOT EXISTS image_claim ( image_ref TEXT NOT NULL, claim_id UUID NOT NULL, -- The project whose build request holds the claim: what its - -- status lists while the build runs, whoever started it. NULL - -- only on a claim taken before claims named it, which lapses - -- within a minute. + -- status lists while the build runs, whoever started it. Every + -- claim names one (`claim_images`). project_id UUID, holder_until BIGINT NOT NULL, PRIMARY KEY (image_ref, claim_id) )"#, + super::waiting::VERSION_BUILD_TABLE, + super::waiting::ONE_WAITING_PER_PROJECT, ], seed: &[], }; @@ -108,7 +111,8 @@ pub enum Claim { /// none for an image that compiles nothing). The row is already /// written, so a sibling sees the build before it starts. Start { name: String, lane: Option }, - /// A build is running for it, minted as `name`: wait on the row. + /// A build is running for it, minted as `name`: the project waits on + /// that one. /// `ours` when this same project started it, so a cancel of this /// project stops it (a build another project started is left to that /// project, and this one only stops waiting). @@ -133,6 +137,10 @@ impl Claim { /// Decide, for one image, whether to start a build or join the running /// one, and record the decision, in one transaction serialized per image. +/// Refused ([`super::BuildCancelled`]) once the project's build was +/// cancelled, so an image claimed after a cancel never starts: the claim +/// either commits before the cancel, which then finds the build and +/// stops it, or sees it. /// /// A row that says the build succeeded is believed only while `images` /// still holds the image: a prune forgets the row when it deletes one, @@ -171,6 +179,7 @@ pub async fn claim( }; let mut tx = pool.begin().await.context("begin the build claim")?; lock_image(&mut tx, image_ref).await?; + refuse_once_cancelled(&mut tx, project_id).await?; let row: Option<(String, uuid::Uuid, String)> = sqlx::query_as("SELECT status, project_id, build_name FROM image_build WHERE image_ref = $1") .bind(image_ref) @@ -204,6 +213,11 @@ pub async fn claim( None => None, }; let name = build_name(); + // Announced in the same statement (delivered when it commits), so the + // build loop watches the build from its first moment: one whose + // starter goes away before it is made on the builder is ended once + // its project's heartbeat goes stale (`end_lost_starts`), even on an + // install that was asleep. sqlx::query( "INSERT INTO image_build (image_ref, project_id, tenant_id, build_name, builder_id, lane, status, reason, \ held_until, started_at, finished_at, log_url, failing_since) \ @@ -211,15 +225,17 @@ pub async fn claim( ON CONFLICT (image_ref) DO UPDATE SET project_id = EXCLUDED.project_id, tenant_id = EXCLUDED.tenant_id, \ build_name = EXCLUDED.build_name, builder_id = NULL, lane = EXCLUDED.lane, status = 'running', \ reason = NULL, held_until = EXCLUDED.held_until, started_at = EXCLUDED.started_at, \ - finished_at = NULL, log_url = NULL, failing_since = NULL", + finished_at = NULL, log_url = NULL, failing_since = NULL \ + RETURNING pg_notify($8, image_ref)", ) .bind(image_ref) .bind(project_id) .bind(tenant) .bind(&name) .bind(lane.map(|l| l as i32)) - .bind(now + START_HOLD_SECS) + .bind(now + OLDER_READERS_START_HOLD_SECS) .bind(now) + .bind(super::follow::BUILD_CLAIMED_CHANNEL) .execute(&mut *tx) .await .context("record the build")?; @@ -227,24 +243,38 @@ pub async fn claim( Ok(Claim::Start { name, lane }) } +/// Refuse, inside `tx`, once `project`'s build was cancelled (its +/// transition is `cancelling_build`). The project's row is read under a +/// share lock, so a cancel landing meanwhile waits for `tx` to commit and +/// then finds what it wrote. +pub(crate) async fn refuse_once_cancelled(tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, project: uuid::Uuid) -> Result<()> { + let transition: Option = sqlx::query_scalar("SELECT transition FROM project WHERE id = $1 FOR SHARE") + .bind(project) + .fetch_optional(&mut **tx) + .await + .context("read whether the project's build was cancelled")?; + if transition.as_deref() == Some(weft_core::projects::ProjectTransition::CancellingBuild.as_str()) { + return Err(super::BuildCancelled.into()); + } + Ok(()) +} + /// The build `name` was made on the builder as `handle` (whose id the /// builder may have chosen): the row records that id, so anyone reading /// it asks the builder about the right build. `false` when the row is no /// longer this running build's (it was cancelled, or given up on, while /// it was being made): nobody will ask about it, so the caller frees it. pub async fn started(pool: &PgPool, image_ref: &str, name: &str, handle: &weft_platform_traits::BuildHandle) -> Result { - // Announced in the same statement, so the build loop watches it from - // the moment it is recorded. + // The build loop already watches it, woken by its claim. let recorded = sqlx::query( "UPDATE image_build SET builder_id = $3, log_url = $4 \ WHERE image_ref = $1 AND build_name = $2 AND status = 'running' AND builder_id IS NULL \ - RETURNING pg_notify($5, image_ref)", + RETURNING 1", ) .bind(image_ref) .bind(name) .bind(&handle.external_build_id) .bind(&handle.log_url) - .bind(super::follow::BUILD_STARTED_CHANNEL) .fetch_optional(pool) .await .context("record the build's id on its builder")?; @@ -361,35 +391,6 @@ pub async fn cancel(pool: &PgPool, image_ref: &str, name: &str, now: i64) -> Res .context("record the build as cancelled") } -/// What the row of `image_ref` says now. -#[derive(Debug, Clone, PartialEq, Eq)] -pub enum Seen { - Running, - /// Its build ended, recorded by whoever saw it end (`finish`). - Ended(Outcome), -} - -/// How the build of `image_ref` stands. A build ends once however many -/// verbs wait on it, and whoever saw it end recorded that before freeing -/// its process, so every waiter reads the end here. -pub async fn look(pool: &PgPool, image_ref: &str) -> Result { - let row: Option<(String, Option)> = - sqlx::query_as("SELECT status, reason FROM image_build WHERE image_ref = $1") - .bind(image_ref) - .fetch_optional(pool) - .await - .context("read how the build stands")?; - let Some((status, reason)) = row else { - return Ok(Seen::Ended(Outcome::Failed(format!("the build of {image_ref} is no longer recorded")))); - }; - Ok(match status.as_str() { - "running" => Seen::Running, - "succeeded" => Seen::Ended(Outcome::Succeeded), - "cancelled" => Seen::Ended(Outcome::Cancelled), - _ => Seen::Ended(Outcome::Failed(reason.unwrap_or_else(|| "the build failed".into()))), - }) -} - /// One build the ledger says runs. #[derive(Debug, Clone, PartialEq, Eq, sqlx::FromRow)] pub struct RunningBuild { @@ -399,48 +400,140 @@ pub struct RunningBuild { pub build_name: String, /// Its id on the builder, once its start recorded it (`started`). pub builder_id: Option, - /// Until when its start may still record that id (`START_HOLD_SECS`). - pub held_until: i64, pub started_at: i64, pub log_url: Option, } -const RUNNING_COLUMNS: &str = "image_ref, project_id, build_name, builder_id, held_until, started_at, log_url"; - -/// The build running for `image_ref`, if one does. -pub async fn running_for(pool: &PgPool, image_ref: &str) -> Result> { - sqlx::query_as(&format!("SELECT {RUNNING_COLUMNS} FROM image_build WHERE image_ref = $1 AND status = 'running'")) - .bind(image_ref) - .fetch_optional(pool) +const RUNNING_COLUMNS: &str = "image_ref, project_id, build_name, builder_id, started_at, log_url"; + +/// Which starts [`end_lost_starts`] ends: those still waiting to be made +/// on the builder whose starter went away. +#[derive(Debug, Clone, Copy)] +pub enum LostStarts { + /// Every such start whose project no request drives any more + /// (`crate::project_store::build_driven`, against this cutoff): the + /// build loop's sweep. The starter's heartbeat beats from before its + /// first claim until its last start is recorded, on whichever + /// dispatcher it runs, so a live start is never ended. + Undriven { stale_before: i64 }, + /// Every such start of this project: a request just took its build + /// transition over from one whose heartbeat went stale + /// (`crate::project_store::ProjectStoreOps::try_begin_building`), so + /// none of them is driven. + Of(uuid::Uuid), +} + +/// End, failed, every build still waiting to be made on the builder whose +/// starter went away between the claim and recording the builder's id: +/// nothing will ever make it. Answers how many it ended. +pub async fn end_lost_starts(conn: &mut sqlx::PgConnection, lost: LostStarts, now: i64) -> Result { + let (project, stale_before) = match lost { + LostStarts::Undriven { stale_before } => (None, stale_before), + LostStarts::Of(project) => (Some(project), 0), + }; + let ended = sqlx::query(&format!( + "UPDATE image_build b SET status = 'failed', finished_at = $1, \ + reason = 'the build ' || b.build_name || ' was never made on the builder: the dispatcher starting it went \ + away before recording it there. Build again' \ + WHERE b.status = 'running' AND b.builder_id IS NULL \ + AND CASE WHEN $2::UUID IS NULL \ + THEN NOT EXISTS (SELECT 1 FROM project p WHERE p.id = b.project_id AND {}) \ + ELSE b.project_id = $2 END", + crate::project_store::build_driven("p", "$3") + )) + .bind(now) + .bind(project) + .bind(stale_before) + .execute(conn) .await - .context("read the running build") + .context("end the starts whose starter went away")?; + Ok(ended.rows_affected()) +} + +/// The SQL condition "the claim row `c` still protects its image": its +/// request renews it (`holder_until` not past `now`), or the version build +/// its id names still waits (`super::waiting`), whatever its lease says. +/// One definition for every reader of a claim: a prune deciding about an +/// image, the sweep of lapsed claims, and which builds a project waits on. +pub(crate) fn claim_live(c: &str, now: &str) -> String { + format!( + "({c}.holder_until >= {now} OR EXISTS (SELECT 1 FROM version_build v WHERE v.id = {c}.claim_id AND v.state = '{}'))", + super::waiting::WAITING + ) } -/// Where the build the builder knows as `builder_id` is, when it is one -/// `project` started or waits on (the same builds [`running`] lists for it). -pub async fn state_of(pool: &PgPool, project: uuid::Uuid, builder_id: &str) -> Result> { - let status: Option = sqlx::query_scalar( - "SELECT b.status FROM image_build b WHERE b.builder_id = $2 AND (b.project_id = $1 \ - OR EXISTS (SELECT 1 FROM image_claim c WHERE c.image_ref = b.image_ref AND c.project_id = $1))", +/// The SQL condition "the project bound at `project` waits on the build +/// row `b`": it started the build, or a live claim of its names the image +/// (a build another project started of the same content). What the +/// project's status lists as its builds. +pub(crate) fn waited_on_by(project: &str, now: &str) -> String { + format!( + "(b.project_id = {project} OR EXISTS (SELECT 1 FROM image_claim c WHERE c.image_ref = b.image_ref \ + AND c.project_id = {project} AND {}))", + claim_live("c", now) ) - .bind(project) - .bind(builder_id) - .fetch_optional(pool) +} + +/// Where the build of each of `images` stands: the one the ledger holds for +/// its image now, under the name it was minted with, which is a newer +/// build than the one named when the image's build was started again +/// since. An image with no row (forgotten) has no state. +pub async fn image_states( + pool: &PgPool, + images: &[weft_core::builds::ImageBuild], +) -> Result> { + #[derive(sqlx::FromRow)] + struct Row { + image_ref: String, + build_name: String, + status: String, + reason: Option, + builder_id: Option, + started_at: i64, + log_url: Option, + } + let refs: Vec<&str> = images.iter().map(|image| image.image.as_str()).collect(); + let rows: Vec = sqlx::query_as( + "SELECT image_ref, build_name, status, reason, builder_id, started_at, log_url FROM image_build \ + WHERE image_ref = ANY($1)", + ) + .bind(&refs) + .fetch_all(pool) .await - .context("read a build's state")?; - status.map(|s| serde_json::from_value(serde_json::Value::String(s)).context("a build state the ledger does not know")).transpose() + .context("read where the version's image builds are")?; + let mut out = Vec::with_capacity(images.len()); + for image in images { + let state = match rows.iter().find(|row| row.image_ref == image.image) { + None => weft_core::builds::ImageBuildState { build: image.clone(), state: Default::default() }, + Some(row) => weft_core::builds::ImageBuildState { + build: weft_core::builds::ImageBuild { image: image.image.clone(), name: row.build_name.clone() }, + state: weft_core::projects::BuildStateResponse { + state: Some( + serde_json::from_value(serde_json::Value::String(row.status.clone())) + .context("a build state the ledger does not know")?, + ), + reason: row.reason.clone(), + build: row.builder_id.clone(), + started_at_unix: Some(row.started_at), + log_url: row.log_url.clone(), + }, + }, + }; + out.push(state); + } + Ok(out) } /// Every build running, oldest first; when `project` is given, the ones it -/// waits on: those it started, and those its build request claims (a -/// build another project started of the same content). +/// waits on ([`waited_on_by`]). pub async fn running(pool: &PgPool, project: Option) -> Result> { sqlx::query_as(&format!( - "SELECT {RUNNING_COLUMNS} FROM image_build b WHERE status = 'running' AND ($1::UUID IS NULL OR b.project_id = $1 \ - OR EXISTS (SELECT 1 FROM image_claim c WHERE c.image_ref = b.image_ref AND c.project_id = $1)) \ - ORDER BY started_at, image_ref" + "SELECT {RUNNING_COLUMNS} FROM image_build b WHERE status = 'running' AND ($1::UUID IS NULL OR {}) \ + ORDER BY started_at, image_ref", + waited_on_by("$1", "$2") )) .bind(project) + .bind(crate::lease::now_unix()) .fetch_all(pool) .await .context("list the running builds") @@ -539,8 +632,9 @@ pub enum Reclaim { /// its deletion: a build that claims it afterwards finds it gone and /// builds it again. /// -/// Both checks are needed because a build lets go of its claim only after -/// its registration commits (`crate::api::project::build`): a reader +/// Both checks are needed because a build lets go of its claim only once +/// its registration commits, or in the same transaction +/// (`super::waiting::register_version`): a reader /// holding the lock sees either the live claim or the committed reference, /// never neither. The prune's own keep-set was read before its loop and /// can predate that registration, so it cannot stand in for the reference @@ -550,10 +644,11 @@ pub async fn begin_reclaim(pool: &PgPool, image_ref: &str, now: i64) -> Result= $2) \ + let claimed: bool = sqlx::query_scalar(&format!( + "SELECT EXISTS (SELECT 1 FROM image_claim c WHERE c.image_ref = $1 AND {}) \ OR EXISTS (SELECT 1 FROM image_build WHERE image_ref = $1 AND status = 'running')", - ) + claim_live("c", "$2") + )) .bind(image_ref) .bind(now) .fetch_one(&mut *tx) @@ -618,10 +713,11 @@ pub async fn drop_claim(pool: &PgPool, claim: uuid::Uuid) -> Result<()> { Ok(()) } -/// Clear the claims whose holder stopped renewing them (its dispatcher is -/// gone): they protect nothing any more. +/// Clear the claims no longer live ([`claim_live`]: their holder stopped +/// renewing them, its dispatcher gone, and no waiting version keeps them): +/// they protect nothing any more. pub async fn drop_lapsed_claims(pool: &PgPool, now: i64) -> Result<()> { - sqlx::query("DELETE FROM image_claim WHERE holder_until < $1") + sqlx::query(&format!("DELETE FROM image_claim c WHERE NOT {}", claim_live("c", "$1"))) .bind(now) .execute(pool) .await diff --git a/crates/weft-dispatcher/src/build/mod.rs b/crates/weft-dispatcher/src/build/mod.rs index 1f016f47..65c32d14 100644 --- a/crates/weft-dispatcher/src/build/mod.rs +++ b/crates/weft-dispatcher/src/build/mod.rs @@ -6,11 +6,14 @@ //! version's files are fetched back and checked against their hashes, the //! project is compiled with the catalog this install ships, every image it //! needs is planned with content-addressed refs in the platform's image -//! store, the stale ones are built by the platform's builder -//! (`weft_platform_traits::ImageBuilder`), and the caller registers the -//! result. Nothing the client -//! computed is believed: not a hash, not an image, not a stored file's -//! metadata. +//! store, and the stale ones are started on the platform's builder +//! (`weft_platform_traits::ImageBuilder`). The request answers as soon as +//! they run: a build takes minutes, and a request held open that long with +//! nothing on the wire is one any network path between may drop. The +//! version is written down as waiting on them (`waiting`), and the build +//! loop (`follow`) sees each build through and registers the version once +//! they all ended; the caller only follows it. Nothing the client computed +//! is believed: not a hash, not an image, not a stored file's metadata. //! //! What the compile cannot do on its own is read an `@asset` that lives on //! the author's disk outside the project, or a URL the author fetched: the @@ -24,13 +27,14 @@ pub mod follow; pub mod ledger; pub mod prune; pub mod source; +pub mod waiting; use std::collections::BTreeMap; use std::sync::Arc; use anyhow::{anyhow, bail, Context, Result}; use async_trait::async_trait; -use weft_platform_traits::{BuildHandle, BuildRequest, ImageBuilder}; +use weft_platform_traits::{BuildHandle, BuildRequest, ImageBuilder, Staging}; /// The control point the version builder (`crate::build`) calls around REAL build work, so the /// dispatcher's `building` transition only engages when something actually @@ -38,22 +42,44 @@ use weft_platform_traits::{BuildHandle, BuildRequest, ImageBuilder}; /// an up-to-date project never serialize against each other). /// /// The mechanism behind the gate is weft's (the project row's `transition` -/// marker + heartbeat + the stuck-transition reaper); the KNOWLEDGE of "a real +/// marker, held by the request while it starts the builds and by the builds +/// themselves after, see `crate::transition`); the KNOWLEDGE of "a real /// build is starting" is the builder's. This trait is the seam between them. #[async_trait] pub trait BuildGate: Send + Sync { /// Called once, just before the first actual image build is submitted. /// Errs when the project cannot enter the `building` transition right now /// (another verb is already building, or the lifecycle is mid-flip); the - /// builder aborts with that error and the verb surfaces it. + /// builder aborts with that error and the verb surfaces it. From here + /// until [`Self::let_go`] the start is on record as alive. async fn begin(&self) -> anyhow::Result<()>; - /// Whether the user requested cancellation of this build. Polled while - /// the builds run; on `true` every build this project started is - /// stopped (`ImageBuilder::release`, after the ledger records the - /// cancel), a build another project started is left running, and the - /// verb errs with a cancellation message. - async fn cancel_requested(&self) -> anyhow::Result; + /// Called once the starts [`Self::begin`] opened are all answered and + /// the version waiting on them is written down, whether or not anybody + /// still waits on the request: the project rests from here on its + /// waiting version. A no-op when `begin` was never called, or refused. + async fn let_go(&self); +} + +/// Why a build request failed when the person cancelled the project's +/// build while it was starting its images: an image claimed after the +/// cancel is refused (`ledger::claim`), and so is a version written down +/// after it (`waiting::ask`). +#[derive(Debug)] +pub struct BuildCancelled; + +impl std::fmt::Display for BuildCancelled { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("the build was cancelled") + } +} + +impl std::error::Error for BuildCancelled {} + +/// Whether `e` is a cancel's refusal ([`BuildCancelled`]), wherever in its +/// chain. +pub fn cancelled(e: &anyhow::Error) -> bool { + e.downcast_ref::().is_some() } /// Content-addressed image refs, named in the platform's image store. @@ -86,16 +112,38 @@ pub fn replaced_infra_images( .collect() } -/// What a build produced: the answer the caller registers and hands the -/// client (its `replaced_infra_images` filled at registration, from -/// [`replaced_infra_images`]), and the images the version runs. +/// What a build request comes to: the version ready to register, or the +/// version build now waiting on its images. +#[derive(Debug)] +pub enum VersionBuild { + /// Every image the version needs is in the registry: register it + /// (`waiting::register`) under `hold`, which claims them all. + Ready { version: Box, hold: prune::ImageHold }, + /// These builds run for the version (started by this request, or + /// already running for the same content), and the version waits on + /// them: the install registers it once they ended. + Waiting(weft_core::builds::BuildsUnderway), +} + +/// A version as a build compiled it: what registration writes, and what a +/// version waiting on its builds keeps until then. #[derive(Debug, Clone)] -pub struct Build { +pub struct Version { + /// The project's name, as the request named it. + pub name: String, + /// The version's files, registered as the program's source. + pub manifest: weft_core::project::hash::Manifest, + /// The answer registration completes (its `replaced_infra_images`, + /// from [`replaced_infra_images`]) and hands the client. pub program: weft_core::builds::BuiltProgram, /// Every image ref this version runs, built now or found already /// there. Registration records them as the project's running version /// (`ledger::note_running`); the client has no use for it. pub images: Vec, + /// Its request's place in the project's order of asks + /// (`ProjectStoreOps::next_build_ask`): it registers only over an + /// older one (`waiting::settle`). + pub ask: i64, } /// The project's files and stored-file metadata, as the build reads them. @@ -122,8 +170,6 @@ pub struct VersionBuilder { /// How many worker builds compile side by side, each in a compile /// cache of its own (`weft_compiler::worker_image::COMPILE_LANE_ARG`). pub compile_lanes: u32, - /// How often an in-flight build is looked at. - pub poll_every: std::time::Duration, /// The reclaims registered builds asked for (`prune::AfterBuildPrunes`). pub prunes: Arc, /// The version files this replica fetched before (`blob_cache`). @@ -132,24 +178,31 @@ pub struct VersionBuilder { impl VersionBuilder { /// Build `request` for `project_id`: fetch and check its files, compile, - /// plan, build every stale image. `gate` is entered before the first - /// real build and polled for cancellation while one runs; a version - /// whose images all exist never touches it. `hold` claims every - /// planned image against a prune until the caller registers them. + /// plan, and start a build of every stale image. `gate` is entered + /// before the first real build; a version whose images all exist never + /// touches it. `hold` claims every planned image against a prune until + /// the version registers. `ask` is the request's place in the + /// project's order of asks, taken as it arrived. + #[allow(clippy::too_many_arguments)] pub async fn build( &self, storage: &dyn ProjectStorage, project_id: uuid::Uuid, tenant: &str, request: &weft_core::builds::VersionBuildRequest, - gate: &dyn BuildGate, - hold: &prune::ImageHold, - ) -> Result { + ask: i64, + gate: Arc, + hold: prune::ImageHold, + ) -> Result { let workdir = tempfile::Builder::new() .prefix("weft-version-") .tempdir() .context("create the build's working directory")?; let root = workdir.path().to_path_buf(); + // The image contexts are staged under it, and a build started from + // one may read it after this request answered: every start keeps a + // clone, and the directory goes with the last (`Staging`). + let staging = Staging::new(workdir); source::materialize(storage, &self.blobs, tenant, &request.manifest, &root).await?; let assets = request.assets.clone(); @@ -176,14 +229,13 @@ impl VersionBuilder { .await .context("the planning task panicked")??; - let built_images = self.ensure_images(&plan.images, project_id, tenant, gate, hold).await?; let images: Vec = plan.images.iter().map(|image| image.image_ref.clone()).collect::>().into_iter().collect(); let infra_images = infra_places(&definition, &plan.images)?; - drop(workdir); - Ok(Build { + let version = Version { + name: request.name.clone(), + manifest: request.manifest.clone(), program: weft_core::builds::BuiltProgram { - built_images, definition, binary_hash: plan.binary_hash, definition_hash: plan.definition_hash, @@ -193,30 +245,40 @@ impl VersionBuilder { replaced_infra_images: Vec::new(), }, images, - }) + ask, + }; + self.start_images(version, &plan.images, &staging, project_id, tenant, gate, hold).await } - /// Make every planned image exist in the registry, building the ones - /// that do not side by side, and answer the refs it built. Every image - /// is claimed in `hold` before the registry is asked about any, so a - /// prune cannot take one this build found. The gate is entered once, - /// before the first build, whether this process starts it or joins one - /// already running. + /// Start a build of every planned image the registry lacks, side by + /// side, and write `version` down as waiting on the builds it needs: + /// each one this request started, or the one already running for the + /// same ref (another project with the same content). Ready when every + /// image is there, including one a build of another request finished + /// meanwhile. Every image is claimed in `hold` before the registry is + /// asked about any, so a prune cannot take one this build found. The + /// gate is entered once, before the first build. Each build started + /// keeps a clone of `staging`, what keeps the contexts on disk. /// - /// Each image is waited on by a task of its own. A build never depends - /// on this verb to be seen through: its row is moved forward by - /// whoever looks (`follow::advance`), this verb while it waits and the - /// build loop otherwise, so a sibling image failing, or the request - /// that asked being dropped, leaves nothing stuck. The verb waits for - /// every image and reports every failure, the first one first. - pub async fn ensure_images( + /// Nothing here waits for a build to end: the build loop sees each one + /// through and registers the version (`follow`, `waiting`). A start + /// that fails stops the builds this request may stop, so the project is + /// left with nothing half started, and the request fails. All of that, + /// from the gate entered to the gate let go, runs on a task the request + /// waits on without owning: a caller that goes away drops the request, + /// never the starts, the version waiting on them, or what they owe once + /// they end. + #[allow(clippy::too_many_arguments)] + pub async fn start_images( &self, + version: Version, images: &[weft_compiler::build_plan::PlannedImage], + staging: &Staging, project_id: uuid::Uuid, tenant: &str, - gate: &dyn BuildGate, - hold: &prune::ImageHold, - ) -> Result> { + gate: Arc, + hold: prune::ImageHold, + ) -> Result { let refs: Vec = images.iter().map(|image| image.image_ref.clone()).collect(); hold.claim(&refs).await?; let mut stale = Vec::new(); @@ -230,42 +292,99 @@ impl VersionBuilder { } } if stale.is_empty() { - return Ok(Vec::new()); + return Ok(VersionBuild::Ready { version: Box::new(version), hold }); } + let (this, staging, tenant) = (self.clone(), staging.clone(), tenant.to_string()); + let (answer, answered) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + let started = this.start_stale(stale, &staging, project_id, &tenant, gate.as_ref(), version, hold).await; + gate.let_go().await; + // Nobody waits any more (the caller went away): what went wrong + // is said here, or nowhere. + if let Err(Err(e)) = answer.send(started) { + tracing::error!( + target: "weft_dispatcher::build", + %project_id, + error = %format!("{e:#}"), + "starting the project's images failed after its request went away" + ); + } + }); + answered.await.map_err(|_| anyhow!("the start of the project's images ended without an answer"))? + } + + /// [`Self::start_images`]' starts and the version written down as + /// waiting on them, on the task the request waits on. + #[allow(clippy::too_many_arguments)] + async fn start_stale( + &self, + stale: Vec, + staging: &Staging, + project_id: uuid::Uuid, + tenant: &str, + gate: &dyn BuildGate, + version: Version, + hold: prune::ImageHold, + ) -> Result { gate.begin().await?; - let (cancel, cancelled) = tokio::sync::watch::channel(false); - let waiters: Vec<(String, tokio::task::JoinHandle>)> = stale + // Each image starts on a task of its own, so one start's panic is + // that image's failure. Each carries the staging until its start is + // recorded. + let tasks: Vec<_> = stale .into_iter() .map(|image| { - let builder = self.clone(); - let (tenant, cancelled) = (tenant.to_string(), cancelled.clone()); - let image_ref = image.image_ref.clone(); - (image_ref, tokio::spawn(async move { builder.build_one(&image, project_id, &tenant, cancelled).await })) + let (this, staging, tenant) = (self.clone(), staging.clone(), tenant.to_string()); + tokio::spawn(async move { this.start_one(&image, &staging, project_id, &tenant).await }) }) .collect(); - let (refs, handles): (Vec, Vec<_>) = waiters.into_iter().unzip(); - let mut all = std::pin::pin!(futures::future::join_all(handles)); - let joined = loop { - tokio::select! { - joined = &mut all => break joined, - _ = tokio::time::sleep(self.poll_every) => { - if !*cancel.borrow() && gate.cancel_requested().await? { - cancel.send_replace(true); + let starts = futures::future::join_all(tasks).await.into_iter().map(|joined| match joined { + Ok(started) => started, + Err(e) => Err(anyhow!("an image's start ended without an answer: {e}")), + }); + let mut underway = Vec::new(); + // The builds this project may stop: the ones it started, and one + // it joined that it had started before. + let mut stoppable = Vec::new(); + let mut failures = Vec::new(); + for start in starts { + match start { + Ok(Some((build, ours))) => { + if ours { + stoppable.push(build.clone()); } + underway.push(build); } + Ok(None) => {} + Err(e) => failures.push(e), } - }; - let built = refs.clone(); - let mut failures = Vec::new(); - for (image_ref, joined) in refs.into_iter().zip(joined) { - match joined { - Ok(Ok(())) => {} - Ok(Err(e)) => failures.push(e), - Err(e) => failures.push(anyhow!("the wait on the build of {image_ref} panicked: {e}")), + } + if failures.is_empty() { + if underway.is_empty() { + return Ok(VersionBuild::Ready { version: Box::new(version), hold }); } + // A claim that lapsed may have let a prune take an image the + // version found, so it is never passed to a waiting version. + let written = match hold.confirm().await { + Ok(()) => waiting::ask(&self.pool, hold, project_id, tenant, &version, &underway, crate::lease::now_unix()) + .await + .context("write the version down as waiting on its builds"), + Err(e) => Err(e), + }; + match written { + Ok(build) => return Ok(VersionBuild::Waiting(weft_core::builds::BuildsUnderway { build, images: underway })), + Err(e) => failures.push(e), + } + } + // A failed stop never hides why a start failed: every error goes + // into the answer. + if let Err(e) = self.stop(&stoppable).await { + failures.push(e.context("stop the builds this request started")); + } + if failures.iter().any(cancelled) { + return Err(BuildCancelled.into()); } let mut failures = failures.into_iter(); - let Some(first) = failures.next() else { return Ok(built) }; + let first = failures.next().expect("only a start that failed gets here"); let others: Vec = failures.map(|e| format!("{e:#}")).collect(); if others.is_empty() { return Err(first); @@ -273,53 +392,57 @@ impl VersionBuilder { bail!("{first:#}\n\n{} other image(s) failed too:\n{}", others.len(), others.join("\n\n")) } - /// See one image to a pushed image: start its build, or join the one - /// already running for the same ref (another project with the same - /// content), then wait for its row to end, moving it forward while - /// waiting. - /// - /// A cancel of this project stops a build it may stop (`stoppable`): - /// the end is recorded, then the build is freed. A build it may not - /// stop goes on; the verb only stops waiting on it. - async fn build_one( + /// Start one image's build, or join the one already running for the + /// same ref; `None` when a build of it finished meanwhile. With the + /// build, whether a cancel of this project may stop it + /// (`ledger::Claim::stoppable`). + async fn start_one( &self, image: &weft_compiler::build_plan::PlannedImage, + staging: &Staging, project_id: uuid::Uuid, tenant: &str, - cancelled: tokio::sync::watch::Receiver, - ) -> Result<()> { + ) -> Result> { let now = crate::lease::now_unix(); let lanes = (image.kind == weft_compiler::build_plan::ImageKind::Worker).then_some(self.compile_lanes); let claim = ledger::claim(&self.pool, self.images.as_ref(), &image.image_ref, project_id, tenant, lanes, now).await?; let stoppable = claim.stoppable(); - // The build this verb waits on: a cancel stops that one, never a - // later build of the same ref the row may run by then. let name = match claim { ledger::Claim::Start { name, lane } => { - self.start(image, project_id, tenant, &name, lane).await?; + self.start(image, staging, project_id, tenant, &name, lane).await?; name } ledger::Claim::Join { name, .. } => name, - ledger::Claim::Built => return Ok(()), + ledger::Claim::Built => return Ok(None), }; - loop { - if *cancelled.borrow() { - if !stoppable { - bail!("the build of {} was cancelled", image.image_ref); - } - // Recorded before anything is freed, like any end. A build - // not made on the builder yet is freed by its starter, which - // finds the row no longer its own. - if let Some(Some(builder_id)) = ledger::cancel(&self.pool, &image.image_ref, &name, crate::lease::now_unix()).await? { - self.images.release(&BuildHandle::named(builder_id)).await; - } - } - follow::advance(&self.pool, self.images.as_ref(), &image.image_ref).await?; - if let ledger::Seen::Ended(outcome) = ledger::look(&self.pool, &image.image_ref).await? { - return ended(&image.image_ref, outcome); + Ok(Some((weft_core::builds::ImageBuild { image: image.image_ref.clone(), name }, stoppable))) + } + + /// Stop `builds`: each one's end recorded as cancelled, then freed on + /// the builder. A build not made on the builder yet is freed by its + /// starter, which finds the row no longer its own (`ledger::started`); + /// one that ended already is left as it ended. + async fn stop(&self, builds: &[weft_core::builds::ImageBuild]) -> Result<()> { + for build in builds { + if let Some(Some(builder_id)) = ledger::cancel(&self.pool, &build.image, &build.name, crate::lease::now_unix()).await? { + self.images.release(&BuildHandle::named(builder_id)).await; } - tokio::time::sleep(self.poll_every).await; } + Ok(()) + } + + /// Cancel what `project` builds: every build it started is stopped + /// ([`Self::stop`]), the ones other projects started go on for them, + /// and its waiting version ends `cancelled` (`waiting::cancel`). + pub async fn cancel_project(&self, project: uuid::Uuid) -> Result<()> { + let own: Vec = ledger::running(&self.pool, Some(project)) + .await? + .into_iter() + .filter(|build| build.project_id == project) + .map(|build| weft_core::builds::ImageBuild { image: build.image_ref, name: build.build_name }) + .collect(); + self.stop(&own).await?; + waiting::cancel(&self.pool, project).await } /// Start the build `name` of `image` (in `lane`, for a worker), whose @@ -329,6 +452,7 @@ impl VersionBuilder { async fn start( &self, image: &weft_compiler::build_plan::PlannedImage, + staging: &Staging, project_id: uuid::Uuid, tenant: &str, name: &str, @@ -343,6 +467,7 @@ impl VersionBuilder { project_id, tenant: tenant.to_string(), context_dir: image.context_dir.clone(), + staging: staging.clone(), image_ref: image.image_ref.clone(), build_args, }; @@ -397,15 +522,6 @@ fn with_secondary(primary: anyhow::Error, recorded: Result<()>) -> anyhow::Error } } -/// What the verb waiting on `image_ref` gets from how its build ended. -fn ended(image_ref: &str, outcome: ledger::Outcome) -> Result<()> { - match outcome { - ledger::Outcome::Succeeded => Ok(()), - ledger::Outcome::Failed(reason) => bail!("the build of {image_ref} failed:\n{reason}"), - ledger::Outcome::Cancelled => bail!("the build of {image_ref} was cancelled"), - } -} - /// Apply the author's `@asset` resolutions to the definition the build /// compiled. A stored-file resolution names a key; its value is rebuilt /// from what the store records for that key (never the metadata the client diff --git a/crates/weft-dispatcher/src/build/prune.rs b/crates/weft-dispatcher/src/build/prune.rs index e6c2933f..baef80d8 100644 --- a/crates/weft-dispatcher/src/build/prune.rs +++ b/crates/weft-dispatcher/src/build/prune.rs @@ -42,6 +42,12 @@ use weft_core::images::PruneReport; /// dispatcher that dies stops renewing /// and its claim stops protecting anything, with no connection held open /// and nothing to wait out. +/// +/// The request owns the claim until it says what becomes of it: let go +/// ([`Self::release`], once registered, or the request failed), or passed +/// to the version build it became ([`Self::pass_to_version`], answered +/// while its builds run), which keeps it alive while it waits. A hold +/// dropped without either (the request was abandoned) lets go on its own. pub struct ImageHold { pool: sqlx::PgPool, id: uuid::Uuid, @@ -50,6 +56,9 @@ pub struct ImageHold { /// The images the claim covers, to check none lapsed. held: std::sync::Mutex>, renewer: tokio::task::JoinHandle<()>, + /// Whether the claim is still this hold's to let go of: cleared once + /// released or passed to its version. + owned: bool, } impl ImageHold { @@ -65,7 +74,12 @@ impl ImageHold { } } }); - Self { pool: pool.clone(), id, project, held: Default::default(), renewer } + Self { pool: pool.clone(), id, project, held: Default::default(), renewer, owned: true } + } + + /// This claim's id: also the id of the version build it may become. + pub fn id(&self) -> uuid::Uuid { + self.id } /// Claim `images` before the build looks for any of them. @@ -90,18 +104,46 @@ impl ImageHold { Ok(()) } - /// Let go once the images are registered (or the build ended). - pub async fn release(self) -> Result<()> { + /// Let go once the images are registered, or the request failed. + pub async fn release(mut self) -> Result<()> { + self.owned = false; self.renewer.abort(); super::ledger::drop_claim(&self.pool, self.id).await } + + /// Leave the claim in place for the waiting version build written + /// under this hold's id (`super::waiting::ask`), which keeps it alive + /// until it ends (`ledger::claim_live`) and lets go of it then. + pub fn pass_to_version(mut self) { + self.owned = false; + } +} + +impl std::fmt::Debug for ImageHold { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("ImageHold").field("id", &self.id).field("project", &self.project).finish_non_exhaustive() + } } impl Drop for ImageHold { - /// A request abandoned midway: the claim stops being renewed, and - /// lapses on its own. + /// Stop renewing, and let go of a claim still owned (an abandoned + /// request), off the dropping thread. Without a runtime to do it on, + /// the claim lapses within its lease. fn drop(&mut self) { self.renewer.abort(); + if !self.owned { + return; + } + let Ok(runtime) = tokio::runtime::Handle::try_current() else { + tracing::warn!(target: "weft_dispatcher::build", claim = %self.id, "an abandoned build's claim on its images could not be let go of: no runtime"); + return; + }; + let (pool, id) = (self.pool.clone(), self.id); + runtime.spawn(async move { + if let Err(e) = super::ledger::drop_claim(&pool, id).await { + tracing::warn!(target: "weft_dispatcher::build", claim = %id, error = %format!("{e:#}"), "letting go of an abandoned build's claim on its images failed"); + } + }); } } diff --git a/crates/weft-dispatcher/src/build/waiting.rs b/crates/weft-dispatcher/src/build/waiting.rs new file mode 100644 index 00000000..29d1e565 --- /dev/null +++ b/crates/weft-dispatcher/src/build/waiting.rs @@ -0,0 +1,759 @@ +//! A version waiting on its image builds, and the one way any version +//! becomes its project's running one. +//! +//! A build request whose version needs images the registry lacks starts +//! their builds and answers at once (a 202). The version it compiled is +//! written down here, `waiting` ([`ask`]), and the build loop +//! (`super::follow`) registers it once every image it waits on is built, +//! or ends it `failed` with why ([`advance`]). Nobody has to ask again and +//! nothing has to stay up for it: the row is the whole of it, and the +//! caller follows it (`GET /projects/{id}/builds/{build}`, [`state`]). +//! +//! Every build request takes the next number in its project's order of +//! asks as it arrives, before it compiles, from the project's row +//! (`ProjectStoreOps::next_build_ask`), so every replica agrees on which +//! ask is newer whatever its clock says. A project has at most one waiting +//! version: a newer ask supersedes it, and the same version asked again +//! (a rerun after Ctrl+C) joins it. Every registration of a project, a +//! request's that found every image as much as the loop's, lands only +//! over an older ask than the one registered (`project.registered_ask`), +//! read under the project's version lock in the transaction that moves +//! the project's running pointers ([`settle`]), so an older version never +//! lands over a newer one, whichever finishes first. +//! +//! The row's id is the id of the request's claim on its images +//! (`super::prune::ImageHold`): the claim spares them from a prune for as +//! long as the version waits (`super::ledger::claim_live`), and goes in the +//! transaction that ends the row, whichever way it ends. + +use anyhow::{anyhow, Context, Result}; +use sqlx::PgPool; +use weft_core::builds::{BuiltProgram, ImageBuild, VersionBuildState, VersionBuildStatus}; + +use crate::project_store::{ProjectStoreOps, StoredProjectSummary}; +use crate::state::DispatcherState; + +/// The state a version waits in, as the column spells it. +pub(crate) const WAITING: &str = "waiting"; + +/// How long an ended version build stays readable: the caller following +/// it reads how it ended well within this. Older ones go on the build +/// loop's next pass ([`advance`]), and every one of a project as it is +/// removed ([`forget_project`]). +pub const ENDED_KEPT_SECS: i64 = 3600; + +/// Why a version superseded by a newer registration never registers. +const NEWER_REGISTERED: &str = "a newer build of this project registered first"; + +/// One row per version a build request answered while its images were +/// still building. +pub const VERSION_BUILD_TABLE: &str = r#"CREATE TABLE IF NOT EXISTS version_build ( + -- Also the claim_id of the image_claim rows sparing the + -- version's images while it waits. + id UUID PRIMARY KEY, + -- Its place in the project's order of asks (project.build_asks). + ask BIGINT NOT NULL, + project_id UUID NOT NULL, + tenant_id TEXT NOT NULL, + -- The project's name, as the request named it. + project_name TEXT NOT NULL, + -- The version's files, registered as the program's source. + manifest JSONB NOT NULL, + -- The BuiltProgram the request compiled (replacedInfraImages + -- empty: the registration fills it). + program JSONB NOT NULL, + -- Every image the version runs. + images TEXT[] NOT NULL, + -- The ImageBuilds it waits on, by image. + waits_on JSONB NOT NULL, + -- SYNC: the states <-> weft_core::builds::VersionBuildStatus + state TEXT NOT NULL CHECK (state IN ('waiting', 'registered', 'failed', 'cancelled', 'superseded')), + reason TEXT, + -- The BuiltProgram answered once registered. + registered JSONB, + asked_at BIGINT NOT NULL, + ended_at BIGINT + )"#; + +/// At most one waiting version per project. +pub const ONE_WAITING_PER_PROJECT: &str = + "CREATE UNIQUE INDEX IF NOT EXISTS version_build_one_waiting ON version_build (project_id) WHERE state = 'waiting'"; + +/// The SQL condition "the project bound at `project` has a waiting +/// version": what holds its build transition once no request drives it +/// (`crate::project_store::build_held`). +pub(crate) fn has_waiting_version(project: &str) -> String { + format!("EXISTS (SELECT 1 FROM version_build v WHERE v.project_id = {project} AND v.state = '{WAITING}')") +} + +fn status_name(status: VersionBuildStatus) -> &'static str { + match status { + VersionBuildStatus::Waiting => WAITING, + VersionBuildStatus::Registered => "registered", + VersionBuildStatus::Failed => "failed", + VersionBuildStatus::Cancelled => "cancelled", + VersionBuildStatus::Superseded => "superseded", + } +} + +fn status_of(name: &str) -> Result { + Ok(match name { + WAITING => VersionBuildStatus::Waiting, + "registered" => VersionBuildStatus::Registered, + "failed" => VersionBuildStatus::Failed, + "cancelled" => VersionBuildStatus::Cancelled, + "superseded" => VersionBuildStatus::Superseded, + other => return Err(anyhow!("a version build state the ledger does not know: '{other}'")), + }) +} + +/// Whether two compiled programs are the same version: the same worker, +/// runtime shape and infra. +pub fn same_version(a: &BuiltProgram, b: &BuiltProgram) -> bool { + a.binary_hash == b.binary_hash && a.definition_hash == b.definition_hash && a.infra_hash == b.infra_hash +} + +/// Serialize, for the transaction, every decision about `project`'s +/// versions: an ask writing one, a registration, a cancel. +async fn lock_versions(conn: &mut sqlx::PgConnection, project: uuid::Uuid) -> Result<()> { + sqlx::query("SELECT pg_advisory_xact_lock(hashtextextended('version_build:' || $1::TEXT, 0))") + .bind(project) + .execute(conn) + .await + .context("lock the project's versions")?; + Ok(()) +} + +/// End the version `id` as `status`, when it still waits, and let go of +/// its claim on its images in the same statement. `false` when it no +/// longer waited. +async fn end( + conn: &mut sqlx::PgConnection, + id: uuid::Uuid, + status: VersionBuildStatus, + reason: Option<&str>, + registered: Option<&BuiltProgram>, + now: i64, +) -> Result { + let ended: i64 = sqlx::query_scalar( + "WITH ended AS (UPDATE version_build SET state = $2, reason = $3, registered = $4, ended_at = $5 \ + WHERE id = $1 AND state = 'waiting' RETURNING id), \ + dropped AS (DELETE FROM image_claim WHERE claim_id IN (SELECT id FROM ended)) \ + SELECT count(*) FROM ended", + ) + .bind(id) + .bind(status_name(status)) + .bind(reason) + .bind(registered.map(serde_json::to_value).transpose()?) + .bind(now) + .fetch_one(conn) + .await + .context("end the version build")?; + Ok(ended == 1) +} + +/// The project's waiting version (its id, ask and program), read under its +/// version lock. +async fn waiting_of(conn: &mut sqlx::PgConnection, project: uuid::Uuid) -> Result> { + let row: Option<(uuid::Uuid, i64, serde_json::Value)> = + sqlx::query_as("SELECT id, ask, program FROM version_build WHERE project_id = $1 AND state = 'waiting'") + .bind(project) + .fetch_optional(conn) + .await + .context("read the project's waiting version")?; + row.map(|(id, asked_at, program)| { + Ok((id, asked_at, serde_json::from_value(program).context("a waiting version's program does not read")?)) + }) + .transpose() +} + +/// The ask of what `project` runs; 0 before its first registration, and +/// for a project removed meanwhile, which has nothing registered. +async fn registered_ask(conn: &mut sqlx::PgConnection, project: uuid::Uuid) -> Result { + sqlx::query_scalar("SELECT COALESCE((SELECT registered_ask FROM project WHERE id = $1), 0)") + .bind(project) + .fetch_one(conn) + .await + .context("read the ask the project runs") +} + +/// Write `version` down as waiting on `waits_on`, under `hold`'s id, and +/// answer that id. In one transaction: refused once the project's build +/// was cancelled; when the project's waiting version is this same version +/// it is joined instead (its id answered, its ask raised to this one, and +/// `hold`'s claim let go: the joined row's claim covers the same images); +/// otherwise the older of the two asks is superseded. A version older +/// than what the project runs, or than the version waiting, is written +/// down superseded already, for its caller to read why. The build loop is +/// woken as a waiting version commits, so one whose builds already ended +/// registers at once. +pub async fn ask( + pool: &PgPool, + hold: super::prune::ImageHold, + project: uuid::Uuid, + tenant: &str, + version: &super::Version, + waits_on: &[ImageBuild], + now: i64, +) -> Result { + let mut tx = pool.begin().await.context("begin writing the waiting version")?; + lock_versions(&mut tx, project).await?; + super::ledger::refuse_once_cancelled(&mut tx, project).await?; + let mut superseded_by = (registered_ask(&mut tx, project).await? >= version.ask).then_some(NEWER_REGISTERED); + if let Some((waiting, waiting_ask, program)) = waiting_of(&mut tx, project).await? { + if same_version(&program, &version.program) { + sqlx::query("UPDATE version_build SET ask = GREATEST(ask, $2) WHERE id = $1") + .bind(waiting) + .bind(version.ask) + .execute(&mut *tx) + .await + .context("join the waiting version")?; + tx.commit().await.context("commit joining the waiting version")?; + let_go_of(hold, project, "it joined the waiting version").await; + return Ok(waiting); + } + if waiting_ask > version.ask { + superseded_by = superseded_by.or(Some("a newer build of this project was asked for")); + } else if superseded_by.is_none() { + end(&mut tx, waiting, VersionBuildStatus::Superseded, Some("a newer build of this project was asked for"), None, now).await?; + } + } + let id = hold.id(); + let (state, ended_at) = match superseded_by { + Some(_) => (VersionBuildStatus::Superseded, Some(now)), + None => (VersionBuildStatus::Waiting, None), + }; + sqlx::query( + "INSERT INTO version_build (id, ask, project_id, tenant_id, project_name, manifest, program, images, waits_on, \ + state, reason, registered, asked_at, ended_at) \ + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, NULL, $12, $13) \ + RETURNING pg_notify($14, id::TEXT)", + ) + .bind(id) + .bind(version.ask) + .bind(project) + .bind(tenant) + .bind(&version.name) + .bind(serde_json::to_value(&version.manifest)?) + .bind(serde_json::to_value(&version.program)?) + .bind(&version.images) + .bind(serde_json::to_value(waits_on)?) + .bind(status_name(state)) + .bind(superseded_by) + .bind(now) + .bind(ended_at) + .bind(super::follow::BUILD_CLAIMED_CHANNEL) + .execute(&mut *tx) + .await + .context("write the waiting version")?; + tx.commit().await.context("commit the waiting version")?; + match superseded_by { + Some(_) => let_go_of(hold, project, "its version was superseded already").await, + None => hold.pass_to_version(), + } + Ok(id) +} + +/// Let go of a request's claim the version it asked for does not need +/// (`why`): a failure only leaves the claim to lapse within its lease. +async fn let_go_of(hold: super::prune::ImageHold, project: uuid::Uuid, why: &str) { + if let Err(e) = hold.release().await { + tracing::warn!( + target: "weft_dispatcher::build", + project_id = %project, + error = %format!("{e:#}"), + "letting go of a build request's claim on its images failed after {why}; the claim lapses within 60 seconds" + ); + } +} + +/// What a registration settles, read and written in the registration's +/// own transaction (`ProjectStoreOps::register_with_hashes`). Either way it +/// lands only over an older ask than the one the project runs. +pub enum Settles<'a> { + /// A request whose ask is `ask` found every image of its version: the + /// waiting version ends `registered` with `answer` when it is this same + /// version, `superseded` when it was asked before, and keeps waiting + /// when it was asked after (it registers over this one later). + Ask { ask: i64, answer: &'a BuiltProgram }, + /// The build loop registers the waiting `version` itself, as `answer`, + /// only while it still waits; it ends `superseded` when a newer ask + /// registered meanwhile. + Waiting { version: uuid::Uuid, answer: &'a BuiltProgram }, +} + +/// [`Settles`], inside the registration's transaction and under the +/// project's version lock. `false` when the registration must not land: +/// a newer ask registered already, the version the loop registers no +/// longer waits, or its project was removed, which ends it. +pub(crate) async fn settle(conn: &mut sqlx::PgConnection, project: uuid::Uuid, settles: &Settles<'_>, now: i64) -> Result { + lock_versions(conn, project).await?; + let running: Option = sqlx::query_scalar("SELECT registered_ask FROM project WHERE id = $1") + .bind(project) + .fetch_optional(&mut *conn) + .await + .context("read the ask the project runs")?; + let registers = match settles { + Settles::Waiting { version, answer } => { + let row: Option<(String, i64)> = + sqlx::query_as("SELECT state, ask FROM version_build WHERE id = $1 AND project_id = $2") + .bind(version) + .bind(project) + .fetch_optional(&mut *conn) + .await + .context("read the version to register")?; + let Some((state, ask)) = row else { + return Err(anyhow!("no version build {version} of project {project}")); + }; + if state != WAITING { + return Ok(false); + } + let Some(running) = running else { + end(conn, *version, VersionBuildStatus::Cancelled, Some("the project was removed"), None, now).await?; + return Ok(false); + }; + if ask <= running { + end(conn, *version, VersionBuildStatus::Superseded, Some(NEWER_REGISTERED), None, now).await?; + return Ok(false); + } + if !end(conn, *version, VersionBuildStatus::Registered, None, Some(answer), now).await? { + return Ok(false); + } + ask + } + Settles::Ask { ask, answer } => { + let running = running.ok_or_else(|| anyhow!("no project {project} to register a build of"))?; + if *ask <= running { + return Ok(false); + } + let mut registers = *ask; + if let Some((waiting, waiting_ask, program)) = waiting_of(conn, project).await? { + if same_version(&program, answer) { + end(conn, waiting, VersionBuildStatus::Registered, None, Some(answer), now).await?; + registers = registers.max(waiting_ask); + } else if waiting_ask < *ask { + end(conn, waiting, VersionBuildStatus::Superseded, Some(NEWER_REGISTERED), None, now).await?; + } + } + registers + } + }; + sqlx::query("UPDATE project SET registered_ask = $2 WHERE id = $1") + .bind(project) + .bind(registers) + .execute(&mut *conn) + .await + .context("record the ask the project runs")?; + Ok(true) +} + +/// Who registers a version. +pub enum Registrar { + /// The build request that found every image there and holds them in + /// `hold`, let go once the registration committed. + Request { hold: super::prune::ImageHold }, + /// The build loop, for the waiting version of this id (its claim goes + /// in the registration's own transaction). + Loop { version: uuid::Uuid }, +} + +/// Register `version` as `project`'s running one: the answer's +/// `replaced_infra_images` filled from what ran before, the program, its +/// hashes and its source written with the project's running pointers and +/// the waiting version's end in one transaction ([`Settles`]), then the +/// images recorded as the project's running version, for a prune picking +/// what to reclaim (`ledger::note_running`). With the project's summary, +/// what registered; `None` when it must not land ([`settle`]). +/// The database half of [`register`], which a test drives with a store. +pub async fn register_version( + projects: &dyn ProjectStoreOps, + pool: &PgPool, + project: uuid::Uuid, + tenant: &str, + version: &super::Version, + registrar: Registrar, +) -> Result> { + let (hold, settles_by) = match registrar { + Registrar::Request { hold } => (Some(hold), None), + Registrar::Loop { version: waiting } => (None, Some(waiting)), + }; + let registered = registered_in_store(projects, project, tenant, version, settles_by).await; + // Let go only now: a prune deciding about one of these images under + // its lock sees either this claim or the committed reference + // (`ledger::begin_reclaim`), never neither. + if let Some(hold) = hold { + if let Err(released) = hold.release().await { + match ®istered { + Ok(_) => tracing::error!( + target: "weft_dispatcher::build", + project_id = %project, + error = %format!("{released:#}"), + "letting go of the build's claim on its images failed after the version registered; the claim \ + lapses within 60 seconds" + ), + Err(e) => return Err(anyhow!("{e:#}\n(and letting go of the build's claim on its images failed too: {released:#})")), + } + } + } + let Some((summary, answer)) = registered? else { return Ok(None) }; + if let Err(e) = super::ledger::note_running(pool, project, &version.images, crate::lease::now_unix()).await { + tracing::error!( + target: "weft_dispatcher::build", + project_id = %project, + error = %format!("{e:#}"), + "recording the images the project runs failed after the version registered; a prune keeps one older build \ + of this project longer, until the next build registers" + ); + } + Ok(Some((summary, answer))) +} + +/// [`register_version`]'s write: `waiting` is the loop's waiting version, +/// `None` for a request. +async fn registered_in_store( + projects: &dyn ProjectStoreOps, + project: uuid::Uuid, + tenant: &str, + version: &super::Version, + waiting: Option, +) -> Result> { + let before = projects + .running_infra_image_tags(project) + .await + .context("read the registered infra images")? + .unwrap_or_default(); + let mut answer = version.program.clone(); + answer.replaced_infra_images = super::replaced_infra_images(&before, &answer.infra_images); + let infra_images: crate::project_store::InfraImageTags = answer + .infra_images + .iter() + .map(|(place, images)| (place.clone(), images.iter().map(|(k, v)| (k.clone(), v.clone())).collect())) + .collect(); + let settles = match waiting { + None => Settles::Ask { ask: version.ask, answer: &answer }, + Some(version) => Settles::Waiting { version, answer: &answer }, + }; + let summary = projects + .register_with_hashes( + answer.definition.clone(), + &version.name, + "", + tenant, + Some(&answer.binary_hash), + Some(&answer.definition_hash), + Some(&answer.infra_hash), + Some(&infra_images), + Some(&answer.implementations), + Some(&version.manifest), + Some(&settles), + ) + .await + .context("register the build")?; + Ok(summary.map(|summary| (summary, answer))) +} + +/// Register `version` ([`register_version`]), then say so: the project +/// lands at rest unless something else still holds its build transition +/// (a registered version must be runnable at once, not after the build +/// loop's next look), the images its earlier builds left are reclaimed +/// (`super::prune::AfterBuildPrunes`), and every editor hears of it. THE +/// way a version becomes its project's running one. +pub async fn register( + state: &DispatcherState, + project: uuid::Uuid, + tenant: &str, + version: &super::Version, + registrar: Registrar, +) -> Result> { + let Some((summary, answer)) = + register_version(state.projects.as_ref(), &state.pg_pool, project, tenant, version, registrar).await? + else { + return Ok(None); + }; + announce(state, &summary).await; + Ok(Some(answer)) +} + +/// What follows a registration that committed. Each step runs whatever +/// the others did, and a failure is logged with what it leaves, never +/// answered: the version IS the project's running one. +pub async fn announce(state: &DispatcherState, summary: &StoredProjectSummary) { + if let Err(e) = crate::transition::settle(state, Some(summary.id)).await { + tracing::error!( + target: "weft_dispatcher::build", + project_id = %summary.id, + error = %format!("{e:#}"), + "landing the project at rest failed after the version registered; the build loop or the stuck-transition \ + reaper settles it" + ); + } + state.builder.prunes.request(state, Some(summary.id)); + state + .events + .publish(crate::events::DispatcherEvent::ProjectRegistered { + project_id: summary.id, + name: weft_core::truncate_user_string(&summary.name, 4096), + }) + .await; +} + +/// How the image builds a version waits on stand, by image. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Verdict { + /// An image still builds (the build named, or a newer one of it). + Waits, + /// Every image was built. + Built, + /// Some image will not be built: why, each failed image's reason, the + /// first one the version names first. + Failed(String), +} + +/// One image's row in the ledger, as [`verdict`] reads it. +#[derive(Debug, Clone, sqlx::FromRow)] +pub struct ImageRow { + pub image_ref: String, + pub status: String, + pub reason: Option, +} + +/// Where a version waiting on `waits_on` stands, from the ledger's rows +/// of those images. An image with no row was forgotten (its build no +/// longer on record), which fails it. +pub fn verdict(waits_on: &[ImageBuild], rows: &[ImageRow]) -> Verdict { + let mut failed = Vec::new(); + for image in waits_on { + match rows.iter().find(|row| row.image_ref == image.image) { + Some(row) if row.status == "running" => return Verdict::Waits, + Some(row) if row.status == "succeeded" => {} + Some(row) if row.status == "cancelled" => failed.push(format!("{}: its build was cancelled", image.image)), + Some(row) => failed.push(format!( + "{}: {}", + image.image, + row.reason.as_deref().unwrap_or("its build failed without saying why") + )), + None => failed.push(format!("{}: the image's build is no longer on record; build again", image.image)), + } + } + if failed.is_empty() { + Verdict::Built + } else { + Verdict::Failed(failed.join("\n")) + } +} + +/// What one pass over the waiting versions did. +#[derive(Default)] +pub struct Advanced { + /// The versions it registered, to announce ([`announce`]). + pub registered: Vec, + /// Whether a version still waits on a build. + pub waiting: bool, + /// The versions it could not move forward, and why. + pub failed: Vec, +} + +/// Move every waiting version forward ([`verdict`]): register the ones +/// whose images were all built, end `failed` the ones an image will never +/// come for, and leave the rest waiting. A version that cannot be moved +/// forward for any reason but the database being out of reach (its +/// registration failed, what it stored no longer reads) ends `failed` +/// with why, rather than being tried again on every look for ever, which +/// would keep the install awake and its project building. One version's +/// failure never skips the others. The versions ended over +/// [`ENDED_KEPT_SECS`] ago go first. +pub async fn advance(pool: &PgPool, projects: &dyn ProjectStoreOps) -> Result { + let now = crate::lease::now_unix(); + sqlx::query("DELETE FROM version_build WHERE state <> 'waiting' AND ended_at < $1") + .bind(now - ENDED_KEPT_SECS) + .execute(pool) + .await + .context("forget the long-ended version builds")?; + let rows: Vec = sqlx::query_as( + "SELECT id, ask, project_id, tenant_id, project_name, manifest, program, images, waits_on FROM version_build \ + WHERE state = 'waiting' ORDER BY asked_at", + ) + .fetch_all(pool) + .await + .context("list the waiting versions")?; + let mut advanced = Advanced::default(); + for row in rows { + let id = row.id; + let moved = match advance_one(pool, projects, row).await { + Err(e) if !crate::api::database_unreachable(&e) => { + let reason = format!("{e:#}"); + let ended = match pool.acquire().await { + Ok(mut conn) => end(&mut conn, id, VersionBuildStatus::Failed, Some(&reason), None, crate::lease::now_unix()).await, + Err(e) => Err(e.into()), + }; + ended.map(|_| Moved::Ended).map_err(|ending| ending.context(reason)) + } + moved => moved, + }; + match moved { + Ok(Moved::Waits) => advanced.waiting = true, + Ok(Moved::Ended) => {} + Ok(Moved::Registered(summary)) => advanced.registered.push(summary), + Err(e) => { + advanced.waiting = true; + advanced.failed.push(format!("version build {id}: {e:#}")); + } + } + } + Ok(advanced) +} + +/// A waiting version as [`advance`] reads it. +#[derive(sqlx::FromRow)] +struct WaitingRow { + id: uuid::Uuid, + ask: i64, + project_id: uuid::Uuid, + tenant_id: String, + project_name: String, + manifest: serde_json::Value, + program: serde_json::Value, + images: Vec, + waits_on: serde_json::Value, +} + +/// What [`advance_one`] did with one waiting version. +enum Moved { + Waits, + /// Ended without registering (failed, or it no longer waited). + Ended, + Registered(StoredProjectSummary), +} + +/// Move one waiting version forward. Every failure is answered, for +/// [`advance`] to end the version with it. +async fn advance_one(pool: &PgPool, projects: &dyn ProjectStoreOps, row: WaitingRow) -> Result { + let waits_on: Vec = serde_json::from_value(row.waits_on).context("the images it waits on do not read")?; + let refs: Vec<&str> = waits_on.iter().map(|image| image.image.as_str()).collect(); + let images: Vec = sqlx::query_as("SELECT image_ref, status, reason FROM image_build WHERE image_ref = ANY($1)") + .bind(&refs) + .fetch_all(pool) + .await + .context("read the builds the version waits on")?; + match verdict(&waits_on, &images) { + Verdict::Waits => Ok(Moved::Waits), + Verdict::Failed(reason) => { + let mut conn = pool.acquire().await.context("a connection to end the version")?; + end(&mut conn, row.id, VersionBuildStatus::Failed, Some(&reason), None, crate::lease::now_unix()).await?; + Ok(Moved::Ended) + } + Verdict::Built => { + let version = super::Version { + name: row.project_name, + manifest: serde_json::from_value(row.manifest).context("its files do not read")?, + program: serde_json::from_value(row.program).context("its program does not read")?, + images: row.images, + ask: row.ask, + }; + let registrar = Registrar::Loop { version: row.id }; + Ok(match register_version(projects, pool, row.project_id, &row.tenant_id, &version, registrar) + .await + .context("every image was built, but registering the version failed")? + { + Some((summary, _)) => Moved::Registered(summary), + None => Moved::Ended, + }) + } + } +} + +/// Forget every version build of a removed project, letting go of a +/// waiting one's claim with it. +pub async fn forget_project(pool: &PgPool, project: uuid::Uuid) -> Result<()> { + sqlx::query( + "WITH gone AS (DELETE FROM version_build WHERE project_id = $1 RETURNING id) \ + DELETE FROM image_claim WHERE claim_id IN (SELECT id FROM gone)", + ) + .bind(project) + .execute(pool) + .await + .context("forget the removed project's version builds")?; + Ok(()) +} + +/// End `project`'s waiting version `cancelled` (the person cancelled its +/// build), letting go of its claim. +pub async fn cancel(pool: &PgPool, project: uuid::Uuid) -> Result<()> { + let mut tx = pool.begin().await.context("begin cancelling the waiting version")?; + lock_versions(&mut tx, project).await?; + if let Some((waiting, _, _)) = waiting_of(&mut tx, project).await? { + end(&mut tx, waiting, VersionBuildStatus::Cancelled, Some("the build was cancelled"), None, crate::lease::now_unix()).await?; + } + tx.commit().await.context("commit cancelling the waiting version")?; + Ok(()) +} + +/// Where `project`'s version build `id` stands, with each image build it +/// waits on; `None` when the project has no such version build. +pub async fn state(pool: &PgPool, project: uuid::Uuid, id: uuid::Uuid) -> Result> { + let row: Option<(String, Option, Option, serde_json::Value)> = + sqlx::query_as("SELECT state, reason, registered, waits_on FROM version_build WHERE id = $1 AND project_id = $2") + .bind(id) + .bind(project) + .fetch_optional(pool) + .await + .context("read the version build")?; + let Some((state, reason, registered, waits_on)) = row else { return Ok(None) }; + let waits_on: Vec = serde_json::from_value(waits_on).context("a version build's images do not read")?; + Ok(Some(VersionBuildState { + state: status_of(&state)?, + images: super::ledger::image_states(pool, &waits_on).await?, + program: registered.map(serde_json::from_value).transpose().context("a registered version's program does not read")?, + reason, + })) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn waits(images: &[&str]) -> Vec { + images.iter().map(|image| ImageBuild { image: image.to_string(), name: format!("{image}-build") }).collect() + } + + fn row(image: &str, status: &str, reason: Option<&str>) -> ImageRow { + ImageRow { image_ref: image.into(), status: status.into(), reason: reason.map(str::to_string) } + } + + #[test] + fn a_version_waits_while_any_image_builds() { + let rows = [row("a", "failed", Some("boom")), row("b", "running", None)]; + assert_eq!(verdict(&waits(&["a", "b"]), &rows), Verdict::Waits, "nothing is decided while one still builds"); + } + + #[test] + fn a_version_is_built_once_every_image_succeeded() { + let rows = [row("a", "succeeded", None), row("b", "succeeded", None)]; + assert_eq!(verdict(&waits(&["a", "b"]), &rows), Verdict::Built); + assert_eq!(verdict(&[], &[]), Verdict::Built); + } + + #[test] + fn a_failed_version_names_every_failed_image_first_first() { + let rows = [row("a", "succeeded", None), row("b", "failed", Some("cargo: error[E0425]")), row("c", "cancelled", None)]; + let Verdict::Failed(reason) = verdict(&waits(&["a", "b", "c", "d"]), &rows) else { panic!("failed") }; + assert_eq!( + reason, + "b: cargo: error[E0425]\nc: its build was cancelled\nd: the image's build is no longer on record; build again" + ); + } + + #[test] + fn every_state_round_trips_through_its_column_name() { + for status in [ + VersionBuildStatus::Waiting, + VersionBuildStatus::Registered, + VersionBuildStatus::Failed, + VersionBuildStatus::Cancelled, + VersionBuildStatus::Superseded, + ] { + assert_eq!(status_of(status_name(status)).unwrap(), status); + assert_eq!(serde_json::to_value(status).unwrap(), status_name(status), "the column spells it as the wire does"); + } + } +} diff --git a/crates/weft-dispatcher/src/journal/fake.rs b/crates/weft-dispatcher/src/journal/fake.rs index c5de6667..6d6f9a47 100644 --- a/crates/weft-dispatcher/src/journal/fake.rs +++ b/crates/weft-dispatcher/src/journal/fake.rs @@ -41,7 +41,7 @@ struct FakeState { next_tag_seq: i64, /// Answers handed to a run its worker drives, by run, for the worker /// to take (the Postgres `parked_fire` rows `hand_answer_in` writes). - handed: HashMap>, + handed: HashMap>, } #[derive(Default)] @@ -99,7 +99,7 @@ impl FakeJournal { } /// The answers handed to `execution_id`'s worker, in order. - pub fn handed(&self, execution_id: ExecutionId) -> Vec<(String, serde_json::Value)> { + pub fn handed(&self, execution_id: ExecutionId) -> Vec<(String, weft_core::primitive::WaitAnswer)> { self.inner.lock().unwrap().handed.get(&execution_id).cloned().unwrap_or_default() } @@ -388,7 +388,7 @@ impl Journal for FakeJournal { /// An answer reaches its run as Postgres's does: written into the /// record of a run nobody drives (queued to carry on), handed to the /// worker of a run one drives ([`Self::handed`]). - async fn answer(&self, token: &str, value: &serde_json::Value) -> anyhow::Result { + async fn answer(&self, token: &str, answer: &weft_core::primitive::WaitAnswer) -> anyhow::Result { use crate::journal::Answered; let mut g = self.inner.lock().unwrap(); let Some(consumed) = g.signals.get(token).filter(|s| s.is_resume).cloned() else { return Ok(Answered::Gone) }; @@ -396,11 +396,11 @@ impl Journal for FakeJournal { let Some(execution_id) = consumed.execution_id else { return Ok(Answered::Gone) }; let Some(row) = g.runs.get_mut(&execution_id).filter(|row| row.state != "ended") else { return Ok(Answered::RunEnded { consumed }) }; if row.owner.is_some() { - g.handed.entry(execution_id).or_default().push((token.to_string(), value.clone())); + g.handed.entry(execution_id).or_default().push((token.to_string(), answer.clone())); return Ok(Answered::Reached { consumed }); } row.state = "queued"; - g.events.push(ExecEvent::SuspensionResolved { execution_id, token: token.to_string(), value: value.clone(), at_unix: 0 }); + g.events.push(ExecEvent::wait_answered(execution_id, token.to_string(), answer.clone(), 0)); Ok(Answered::Reached { consumed }) } @@ -865,17 +865,17 @@ pub(crate) mod tests { resume.is_resume = true; resume.execution_id = Some(run); j.signal_insert(&resume).await.unwrap(); - let crate::journal::Answered::Reached { consumed } = j.answer("tok-r", &serde_json::json!("yes")).await.unwrap() else { + let crate::journal::Answered::Reached { consumed } = j.answer("tok-r", &weft_core::primitive::WaitAnswer::Given { value: serde_json::json!("yes") }).await.unwrap() else { panic!("the wait is answered") }; assert_eq!(consumed.token, "tok-r"); assert!(j.signal_get("tok-r").await.unwrap().is_none(), "single use"); assert_eq!(j.inner.lock().unwrap().runs[&run].state, "queued"); - assert!(matches!(j.answer("tok-r", &serde_json::Value::Null).await.unwrap(), crate::journal::Answered::Gone)); + assert!(matches!(j.answer("tok-r", &weft_core::primitive::WaitAnswer::Skipped).await.unwrap(), crate::journal::Answered::Gone)); // An entry row is never answered this way. j.signal_insert(®istration("tok-e")).await.unwrap(); - assert!(matches!(j.answer("tok-e", &serde_json::Value::Null).await.unwrap(), crate::journal::Answered::Gone)); + assert!(matches!(j.answer("tok-e", &weft_core::primitive::WaitAnswer::Skipped).await.unwrap(), crate::journal::Answered::Gone)); assert!(j.signal_get("tok-e").await.unwrap().is_some(), "entry rows stay"); } diff --git a/crates/weft-dispatcher/src/journal/mod.rs b/crates/weft-dispatcher/src/journal/mod.rs index e08d6b27..7d0ade1e 100644 --- a/crates/weft-dispatcher/src/journal/mod.rs +++ b/crates/weft-dispatcher/src/journal/mod.rs @@ -203,9 +203,9 @@ pub trait Journal: Send + Sync { program: Option<&weft_core::ProjectDefinition>, ) -> anyhow::Result; - /// Answer the wait `token` with `value` (`answer_in`, on a + /// Answer the wait `token` with `answer` (`answer_in`, on a /// transaction of its own). - async fn answer(&self, token: &str, value: &serde_json::Value) -> anyhow::Result; + async fn answer(&self, token: &str, answer: &weft_core::primitive::WaitAnswer) -> anyhow::Result; /// Persist a signal token (token-scoped enumeration credential). /// Record a freshly minted signal token. The api layer generates the token diff --git a/crates/weft-dispatcher/src/journal/postgres.rs b/crates/weft-dispatcher/src/journal/postgres.rs index 85cd0b81..04486ec1 100644 --- a/crates/weft-dispatcher/src/journal/postgres.rs +++ b/crates/weft-dispatcher/src/journal/postgres.rs @@ -117,7 +117,7 @@ impl PostgresJournal { } } -/// Answer the wait `token` with `value`, on the caller's transaction: +/// Answer the wait `token` with `answer`, on the caller's transaction: /// the wait's signal goes (a wait is answered once), and the answer /// reaches its run. A run nobody drives (parked, queued) gets it in its /// record, `SuspensionResolved` at `last_seq + 1`, and is queued for a @@ -129,7 +129,7 @@ impl PostgresJournal { pub(crate) async fn answer_in( tx: &mut sqlx::PgConnection, token: &str, - value: &serde_json::Value, + answer: &weft_core::primitive::WaitAnswer, from: AnswerFrom, ) -> anyhow::Result { use crate::journal::Answered; @@ -171,15 +171,10 @@ pub(crate) async fn answer_in( let consumed = row_to_signal(row)?; let Some(locked) = locked.filter(|locked| locked.state != "ended") else { return Ok(Answered::RunEnded { consumed }) }; if locked.owner.is_some() { - weft_task_store::parked_fires::hand_answer_in(&mut *tx, token, execution_id, value).await?; + weft_task_store::parked_fires::hand_answer_in(&mut *tx, token, execution_id, answer).await?; return Ok(Answered::Reached { consumed }); } - let resolved = ExecEvent::SuspensionResolved { - execution_id, - token: token.to_string(), - value: value.clone(), - at_unix: crate::lease::now_unix() as u64, - }; + let resolved = ExecEvent::wait_answered(execution_id, token.to_string(), answer.clone(), crate::lease::now_unix() as u64); match weft_journal::record::append_locked_in(&mut *tx, execution_id, &locked, std::slice::from_ref(&resolved), weft_journal::record::DISPATCHER, Then::Queued).await? { Appended::At(_) => Ok(Answered::Reached { consumed }), other => anyhow::bail!("run {execution_id} was locked with no owner, yet its answer was not written: {other:?}"), @@ -990,9 +985,9 @@ impl Journal for PostgresJournal { Ok(lost) } - async fn answer(&self, token: &str, value: &serde_json::Value) -> anyhow::Result { + async fn answer(&self, token: &str, answer: &weft_core::primitive::WaitAnswer) -> anyhow::Result { let mut tx = self.pool.begin().await?; - let answered = answer_in(&mut tx, token, value, AnswerFrom::Sender).await?; + let answered = answer_in(&mut tx, token, answer, AnswerFrom::Sender).await?; tx.commit().await?; weft_task_store::announce::committed(&self.pool); Ok(answered) diff --git a/crates/weft-dispatcher/src/parked_drain.rs b/crates/weft-dispatcher/src/parked_drain.rs index 722ca5f3..22e15c7c 100644 --- a/crates/weft-dispatcher/src/parked_drain.rs +++ b/crates/weft-dispatcher/src/parked_drain.rs @@ -212,7 +212,7 @@ async fn hand_answer( weft_task_store::parked_fires::remove_in(&mut tx, head).await?; None } - weft_core::arrival::Arrival::Live => match crate::journal::postgres::answer_in(&mut tx, &head.token, &head.waiting.payload, crate::journal::postgres::AnswerFrom::Queue).await? { + weft_core::arrival::Arrival::Live => match crate::journal::postgres::answer_in(&mut tx, &head.token, &head.waiting.answer(), crate::journal::postgres::AnswerFrom::Queue).await? { // The wait's signal went, and its queue with it. crate::journal::Answered::Reached { consumed } | crate::journal::Answered::RunEnded { consumed } => Some(consumed), crate::journal::Answered::Gone => { diff --git a/crates/weft-dispatcher/src/project_store.rs b/crates/weft-dispatcher/src/project_store.rs index 81cea021..68d9e5e2 100644 --- a/crates/weft-dispatcher/src/project_store.rs +++ b/crates/weft-dispatcher/src/project_store.rs @@ -59,6 +59,14 @@ pub trait ProjectStoreOps: Send + Sync { /// passes it here so the runnable stamp and the infra tags land together /// (never a runnable project with missing/half-written tags); plain /// registration (no build) passes `None`. + /// + /// `settles`: what a build's registration settles about the project's + /// waiting version (`crate::build::waiting::Settles`), read and + /// written under the project's version lock in this same transaction, + /// so a registration and the waiting version's end commit together and + /// an older version never lands over a newer one. `Ok(None)` only when + /// it names a waiting version that no longer waits: nothing registered. + #[allow(clippy::too_many_arguments)] async fn register_with_hashes( &self, project: ProjectDefinition, @@ -71,7 +79,8 @@ pub trait ProjectStoreOps: Send + Sync { infra_image_tags: Option<&InfraImageTags>, implementations: Option<&std::collections::BTreeMap>, source: Option<&weft_core::project::hash::Manifest>, - ) -> anyhow::Result; + settles: Option<&crate::build::waiting::Settles<'_>>, + ) -> anyhow::Result>; /// Sources registered with this exact running program. Refuse a concurrent build. async fn program_source(&self, id: uuid::Uuid, program: &weft_core::project::hash::ProgramIdentity) -> anyhow::Result; @@ -171,46 +180,66 @@ pub trait ProjectStoreOps: Send + Sync { /// project. async fn running_infra_image_tags(&self, id: uuid::Uuid) -> anyhow::Result>; + /// The next number in the order of the project's build asks, taken by + /// a build request as it arrives (`build_asks`): a version registers + /// only over an older ask (`crate::build::waiting::Settles`). Errs on + /// no such project. + async fn next_build_ask(&self, id: uuid::Uuid) -> anyhow::Result; + /// Read the project's verb-transition marker (the build axis, /// orthogonal to `status`). `Ok(None)` = no such project. async fn transition(&self, id: uuid::Uuid) -> anyhow::Result>; /// Single-flight entry into the `building` transition. Atomically - /// flips `transition` none → building IFF no other transition is - /// in flight AND the trigger lifecycle is not mid-flip - /// (activating / deactivating). Stamps the transition heartbeat. - /// `Ok(false)` = lost: another verb is building, or the lifecycle - /// is transitional; the caller rejects (409). - async fn try_begin_building(&self, id: uuid::Uuid) -> anyhow::Result; - - /// Request cancellation of the in-flight build: CAS `transition` - /// building → cancelling_build. The process driving the build polls - /// this (via `transition`) and interrupts the builder. `Ok(false)` - /// = no build in flight (already finished, or never started). + /// flips `transition` to building IFF no request is starting builds + /// for the project right now (the transition is none, or building held + /// only by its waiting version or by a request gone: its heartbeat is + /// older than `stale_before`) AND the trigger lifecycle is not mid-flip + /// (activating / deactivating). Stamps the transition heartbeat. Taking + /// over from a request gone ends, in the same transaction, the starts + /// it left waiting to be made on the builder + /// (`crate::build::ledger::LostStarts::Of`), so this request never + /// joins a start nothing will finish. A request entering while builds + /// run joins them (the ledger finds them running), which is how a + /// person who stopped following a build picks it up again. + /// `Ok(false)` = lost: another request is starting builds, a cancel is + /// under way, or the lifecycle is transitional; the caller rejects + /// (409). + async fn try_begin_building(&self, id: uuid::Uuid, stale_before: i64) -> anyhow::Result; + + /// Request cancellation of the in-flight build: `transition` building + /// (or cancelling_build already, so a cancel asked twice is the same + /// cancel) → cancelling_build. From then on an image claim and a + /// version written down as waiting are refused + /// (`crate::build::ledger::refuse_once_cancelled`), and the cancel stops + /// the builds already running and ends the waiting version itself. + /// `Ok(false)` = no build in flight (already finished, or never + /// started). async fn request_cancel_build(&self, id: uuid::Uuid) -> anyhow::Result; - /// Land the build transition back at rest: `transition` → none - /// from either building or cancelling_build. Idempotent (a no-op - /// when already none, e.g. the stuck-transition reaper got there - /// first). `Ok(true)` iff this call performed the flip. - async fn finish_building(&self, id: uuid::Uuid) -> anyhow::Result; - - /// Bump the transition heartbeat. Called on an interval by the - /// process DRIVING an in-process transitional state (an activation - /// window, a build) so the stuck-transition reaper only repairs - /// transitions whose driver actually died. + /// The request that entered the build transition is done with it + /// (answered, or failed): from here the transition rests on the + /// project's waiting version alone, so [`Self::settle_building`] may + /// clear it as soon as none waits. Stamps the heartbeat to zero, which + /// no live driver ever leaves it at. + async fn release_build_driver(&self, id: uuid::Uuid) -> anyhow::Result<()>; + + /// Land at rest (`transition` → none) every project in a build + /// transition that nothing holds any more ([`build_held`]: no request + /// drives it and no version of it waits). `project` narrows it to one. + /// Answers the projects it landed, for the caller to announce. Called + /// by the build loop, by a request letting go, by a registration, by a + /// cancel, and by the stuck-transition reaper, so one definition of + /// "building" holds everywhere. + async fn settle_building(&self, project: Option, stale_before: i64) -> anyhow::Result>; + + /// Bump the transition heartbeat of a build transition still driven + /// (never one released). Called on an interval by the process DRIVING + /// it (a build request starting its builds) so the + /// stuck-transition reaper only repairs transitions whose driver + /// actually died. async fn bump_transition_heartbeat(&self, id: uuid::Uuid) -> anyhow::Result<()>; - /// Projects stuck in a build transition whose heartbeat went stale - /// before `stale_before`: the driving process died mid-build. The - /// stuck-transition reaper lands each back at rest. (An activation - /// stuck the same way lives on its own row: - /// `ActivationStoreOps::list_stuck`.) - async fn list_stuck_transitions( - &self, - stale_before: i64, - ) -> anyhow::Result>; - /// The project's own worker levers (`Ok(None)` when the project does /// not exist). async fn worker_overrides(&self, id: uuid::Uuid) -> anyhow::Result>; @@ -245,11 +274,24 @@ pub fn project_transition_from_str(s: &str) -> anyhow::Result .ok_or_else(|| anyhow::anyhow!("unknown project.transition column value '{s}'")) } -/// One project stuck in a build transition (stale heartbeat). -#[derive(Debug, Clone)] -pub struct StuckTransition { - pub id: uuid::Uuid, - pub transition: ProjectTransition, +/// The SQL condition "a request drives the project row `p` through its +/// build transition right now": the row is in one and its heartbeat is no +/// older than `stale_before` (`crate::transition::heartbeat_stale_secs`). +/// The heartbeat beats while the request's starts run +/// (`crate::transition::ProjectBuildGate`), so a start-pending build of a +/// project not driven has lost its starter +/// (`crate::build::ledger::end_lost_starts`). One definition for every +/// reader, so a settle and the build loop never disagree about whether a +/// start is alive. +pub(crate) fn build_driven(p: &str, stale_before: &str) -> String { + format!("({p}.transition IN ('building', 'cancelling_build') AND {p}.transition_heartbeat_unix >= {stale_before})") +} + +/// The SQL condition "something holds the project row `p` in its build +/// transition": a request drives it ([`build_driven`]), or a version of it +/// waits on its builds (`crate::build::waiting`). What a settle reads. +pub(crate) fn build_held(p: &str, stale_before: &str) -> String { + format!("({} OR {})", build_driven(p, stale_before), crate::build::waiting::has_waiting_version(&format!("{p}.id"))) } /// Cloneable handle to whatever the dispatcher uses as project @@ -358,16 +400,18 @@ pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { health_protocols_json JSONB, -- Verb-transition marker, orthogonal to `status` (the -- BUILD axis): 'none' | 'building' | 'cancelling_build'. - -- Written only by its own single-flight CAS methods + -- Written only by its own single-flight methods -- (try_begin_building / request_cancel_build / - -- finish_building), never by lifecycle writes, so a - -- deactivate can't stomp an in-flight build marker. + -- settle_building), never by lifecycle writes, so a + -- deactivate can't stomp an in-flight build marker. Held + -- by the request starting the builds, then by the + -- project's waiting version until it ends. transition TEXT NOT NULL DEFAULT 'none', -- Heartbeat for the build transition - -- (transition='building'/'cancelling_build'): the replica - -- driving it bumps this on an interval; the stuck-transition - -- reaper repairs rows whose heartbeat went stale (the - -- driver died mid-transition). + -- (transition='building'/'cancelling_build'): the request + -- starting the builds bumps this on an interval and zeroes + -- it when it lets go; once it is stale, only a waiting + -- version holds the transition (settle_building). transition_heartbeat_unix BIGINT NOT NULL DEFAULT 0, -- The version tree's HEAD (`crate::versions`): the version -- the next checkpoint or run parents on, the run the next @@ -387,7 +431,16 @@ pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { -- `crate::front`): what `weft status` shows and where the -- listener hands the project's events. NULL while nothing -- of the project takes work. - api_address JSONB + api_address JSONB, + -- The order of the project's build asks: each build request + -- takes the next one as it arrives, before it compiles + -- (`next_build_ask`), from this row, so every replica + -- agrees on it. + build_asks BIGINT NOT NULL DEFAULT 0, + -- The ask whose version the project runs: a registration + -- lands only over an older ask + -- (`crate::build::waiting::settle`). + registered_ask BIGINT NOT NULL DEFAULT 0 )"#, "CREATE INDEX IF NOT EXISTS idx_project_tenant ON project(tenant_id)", "CREATE UNIQUE INDEX IF NOT EXISTS idx_project_api_port ON project(api_port) WHERE api_port IS NOT NULL", @@ -572,7 +625,8 @@ impl ProjectStoreOps for PostgresProjectStore { infra_image_tags: Option<&InfraImageTags>, implementations: Option<&std::collections::BTreeMap>, source: Option<&weft_core::project::hash::Manifest>, - ) -> anyhow::Result { + settles: Option<&crate::build::waiting::Settles<'_>>, + ) -> anyhow::Result> { let id = project.id; let name = name.to_string(); let description = description.to_string(); @@ -584,6 +638,12 @@ impl ProjectStoreOps for PostgresProjectStore { let project_json = serde_json::to_string(&project)?; let mut tx = self.pool.begin().await?; let now = crate::lease::now_unix(); + if let Some(settles) = settles { + if !crate::build::waiting::settle(&mut tx, id, settles, now).await? { + tx.commit().await?; + return Ok(None); + } + } // The conflict arm is GUARDED by `WHERE project.tenant_id = // EXCLUDED.tenant_id`: a re-register may only update a row that already // belongs to the same tenant. Without this guard the upsert would let @@ -668,7 +728,7 @@ impl ProjectStoreOps for PostgresProjectStore { .execute(&mut *tx).await?; } tx.commit().await?; - Ok(StoredProjectSummary { id, name, description }) + Ok(Some(StoredProjectSummary { id, name, description })) } async fn program_source(&self, id: uuid::Uuid, program: &weft_core::project::hash::ProgramIdentity) -> anyhow::Result { @@ -870,6 +930,14 @@ impl ProjectStoreOps for PostgresProjectStore { .transpose() } + async fn next_build_ask(&self, id: uuid::Uuid) -> anyhow::Result { + sqlx::query_scalar("UPDATE project SET build_asks = build_asks + 1 WHERE id = $1 RETURNING build_asks") + .bind(id) + .fetch_optional(&self.pool) + .await? + .ok_or_else(|| anyhow::anyhow!("no project {id} to take a build ask of")) + } + async fn transition(&self, id: uuid::Uuid) -> anyhow::Result> { let row: Option<(String,)> = sqlx::query_as("SELECT transition FROM project WHERE id = $1") @@ -879,26 +947,34 @@ impl ProjectStoreOps for PostgresProjectStore { row.map(|(t,)| project_transition_from_str(&t)).transpose() } - async fn try_begin_building(&self, id: uuid::Uuid) -> anyhow::Result { + async fn try_begin_building(&self, id: uuid::Uuid, stale_before: i64) -> anyhow::Result { + let now = crate::lease::now_unix(); + let mut tx = self.pool.begin().await?; let res = sqlx::query( "UPDATE project \ SET transition = 'building', transition_heartbeat_unix = $1 \ WHERE id = $2 \ - AND transition = 'none' \ + AND (transition = 'none' OR (transition = 'building' AND transition_heartbeat_unix < $3)) \ AND NOT EXISTS (SELECT 1 FROM trigger_activation a \ WHERE a.project_id = $2 AND a.status IN ('activating', 'deactivating'))", ) - .bind(crate::lease::now_unix()) + .bind(now) .bind(id) - .execute(&self.pool) + .bind(stale_before) + .execute(&mut *tx) .await?; - Ok(res.rows_affected() > 0) + if res.rows_affected() == 0 { + return Ok(false); + } + crate::build::ledger::end_lost_starts(&mut tx, crate::build::ledger::LostStarts::Of(id), now).await?; + tx.commit().await?; + Ok(true) } async fn request_cancel_build(&self, id: uuid::Uuid) -> anyhow::Result { let res = sqlx::query( "UPDATE project SET transition = 'cancelling_build' \ - WHERE id = $1 AND transition = 'building'", + WHERE id = $1 AND transition IN ('building', 'cancelling_build')", ) .bind(id) .execute(&self.pool) @@ -906,47 +982,48 @@ impl ProjectStoreOps for PostgresProjectStore { Ok(res.rows_affected() > 0) } - async fn finish_building(&self, id: uuid::Uuid) -> anyhow::Result { - let res = sqlx::query( - "UPDATE project SET transition = 'none' \ + async fn release_build_driver(&self, id: uuid::Uuid) -> anyhow::Result<()> { + sqlx::query( + "UPDATE project SET transition_heartbeat_unix = 0 \ WHERE id = $1 AND transition IN ('building', 'cancelling_build')", ) .bind(id) .execute(&self.pool) .await?; - Ok(res.rows_affected() > 0) - } - - async fn bump_transition_heartbeat(&self, id: uuid::Uuid) -> anyhow::Result<()> { - sqlx::query("UPDATE project SET transition_heartbeat_unix = $1 WHERE id = $2") - .bind(crate::lease::now_unix()) - .bind(id) - .execute(&self.pool) - .await?; Ok(()) } - async fn list_stuck_transitions( - &self, - stale_before: i64, - ) -> anyhow::Result> { + async fn settle_building(&self, project: Option, stale_before: i64) -> anyhow::Result> { // NOTE (compiler-invisible status set): this WHERE names the - // driver-backed transitional states by string. Adding a new - // driver-backed transitional state means adding it HERE too; - // the Rust exhaustiveness checker cannot flag this SQL. - let rows: Vec<(uuid::Uuid, String)> = sqlx::query_as( - "SELECT id, transition FROM project \ - WHERE transition IN ('building', 'cancelling_build') \ - AND transition_heartbeat_unix < $1", - ) + // build transition's states by string, like the CAS methods above. + Ok(sqlx::query_scalar(&format!( + "UPDATE project p SET transition = 'none' \ + WHERE p.transition IN ('building', 'cancelling_build') \ + AND NOT {} \ + AND ($2::UUID IS NULL OR p.id = $2) \ + RETURNING p.id", + build_held("p", "$1") + )) .bind(stale_before) + .bind(project) .fetch_all(&self.pool) + .await?) + } + + async fn bump_transition_heartbeat(&self, id: uuid::Uuid) -> anyhow::Result<()> { + // A released heartbeat (0, `release_build_driver`) stays released: + // the beat's task is stopped without waiting for it, so a bump + // already on its way may land after the release, and must not make + // the project driven again. + sqlx::query( + "UPDATE project SET transition_heartbeat_unix = $1 \ + WHERE id = $2 AND transition IN ('building', 'cancelling_build') AND transition_heartbeat_unix > 0", + ) + .bind(crate::lease::now_unix()) + .bind(id) + .execute(&self.pool) .await?; - rows.into_iter() - .map(|(id, transition)| { - Ok(StuckTransition { id, transition: project_transition_from_str(&transition)? }) - }) - .collect() + Ok(()) } /// Read-only access to the full ProjectDefinition. JSON decode @@ -1027,6 +1104,8 @@ pub struct FakeProjectStore { /// columns. Missing entry = (None, 0), matching the column /// defaults on a fresh row. transitions: RwLock>, + /// Mirror of `build_asks`. + build_asks: RwLock>, } #[cfg(any(test, feature = "test-helpers"))] @@ -1046,6 +1125,7 @@ impl FakeProjectStore { has_infra: RwLock::new(HashMap::new()), worker_overrides: RwLock::new(HashMap::new()), transitions: RwLock::new(HashMap::new()), + build_asks: RwLock::new(HashMap::new()), } } } @@ -1072,7 +1152,10 @@ impl ProjectStoreOps for FakeProjectStore { infra_image_tags: Option<&InfraImageTags>, implementations: Option<&std::collections::BTreeMap>, source: Option<&weft_core::project::hash::Manifest>, - ) -> anyhow::Result { + // The fake holds no version builds (they live in Postgres): a + // registration always lands. + _settles: Option<&crate::build::waiting::Settles<'_>>, + ) -> anyhow::Result> { let id = project.id; let name_owned = name.to_string(); let description_owned = description.to_string(); @@ -1137,11 +1220,11 @@ impl ProjectStoreOps for FakeProjectStore { None => { sources.remove(&id); } } } - Ok(StoredProjectSummary { + Ok(Some(StoredProjectSummary { id, name: name_owned, description: description_owned, - }) + })) } async fn running_program_identity(&self, id: uuid::Uuid) -> anyhow::Result> { @@ -1228,6 +1311,7 @@ impl ProjectStoreOps for FakeProjectStore { self.has_infra.write().await.remove(&id); self.worker_overrides.write().await.remove(&id); self.transitions.write().await.remove(&id); + self.build_asks.write().await.remove(&id); Ok(was_present) } @@ -1297,6 +1381,14 @@ impl ProjectStoreOps for FakeProjectStore { Ok(Some(self.infra_image_tags.read().await.get(&id).cloned().unwrap_or_default())) } + async fn next_build_ask(&self, id: uuid::Uuid) -> anyhow::Result { + anyhow::ensure!(self.inner.read().await.contains_key(&id), "no project {id} to take a build ask of"); + let mut asks = self.build_asks.write().await; + let ask = asks.entry(id).or_insert(0); + *ask += 1; + Ok(*ask) + } + async fn transition(&self, id: uuid::Uuid) -> anyhow::Result> { if !self.inner.read().await.contains_key(&id) { return Ok(None); @@ -1311,7 +1403,7 @@ impl ProjectStoreOps for FakeProjectStore { )) } - async fn try_begin_building(&self, id: uuid::Uuid) -> anyhow::Result { + async fn try_begin_building(&self, id: uuid::Uuid, stale_before: i64) -> anyhow::Result { // The half of the guard that reads activations lives in // Postgres (a build refuses while one is mid-flip); the fake // project store holds no activations, so it guards the build @@ -1321,7 +1413,8 @@ impl ProjectStoreOps for FakeProjectStore { return Ok(false); } let entry = transitions.entry(id).or_insert((ProjectTransition::None, 0)); - if entry.0 != ProjectTransition::None { + let at_rest = entry.0 == ProjectTransition::None || (entry.0 == ProjectTransition::Building && entry.1 < stale_before); + if !at_rest { return Ok(false); } *entry = (ProjectTransition::Building, crate::lease::now_unix()); @@ -1331,7 +1424,7 @@ impl ProjectStoreOps for FakeProjectStore { async fn request_cancel_build(&self, id: uuid::Uuid) -> anyhow::Result { let mut transitions = self.transitions.write().await; match transitions.get_mut(&id) { - Some(entry) if entry.0 == ProjectTransition::Building => { + Some(entry) if entry.0.is_building() => { entry.0 = ProjectTransition::CancellingBuild; Ok(true) } @@ -1339,41 +1432,38 @@ impl ProjectStoreOps for FakeProjectStore { } } - async fn finish_building(&self, id: uuid::Uuid) -> anyhow::Result { - let mut transitions = self.transitions.write().await; - match transitions.get_mut(&id) { - Some(entry) if entry.0.is_building() => { + async fn release_build_driver(&self, id: uuid::Uuid) -> anyhow::Result<()> { + if let Some(entry) = self.transitions.write().await.get_mut(&id) { + if entry.0.is_building() { + entry.1 = 0; + } + } + Ok(()) + } + + async fn settle_building(&self, project: Option, stale_before: i64) -> anyhow::Result> { + // The fake project store sees no version builds (they live in + // Postgres), so a transition nothing drives is at rest. + let mut settled = Vec::new(); + for (id, entry) in self.transitions.write().await.iter_mut() { + if entry.0.is_building() && entry.1 < stale_before && project.is_none_or(|p| p == *id) { entry.0 = ProjectTransition::None; - Ok(true) + settled.push(*id); } - _ => Ok(false), } + Ok(settled) } async fn bump_transition_heartbeat(&self, id: uuid::Uuid) -> anyhow::Result<()> { - self.transitions - .write() - .await - .entry(id) - .or_insert((ProjectTransition::None, 0)) - .1 = crate::lease::now_unix(); + // As the real store: only a build transition still driven. + if let Some(entry) = self.transitions.write().await.get_mut(&id) { + if entry.0.is_building() && entry.1 > 0 { + entry.1 = crate::lease::now_unix(); + } + } Ok(()) } - async fn list_stuck_transitions( - &self, - stale_before: i64, - ) -> anyhow::Result> { - Ok(self - .transitions - .read() - .await - .iter() - .filter(|(_, (transition, hb))| transition.is_building() && *hb < stale_before) - .map(|(id, (transition, _))| StuckTransition { id: *id, transition: *transition }) - .collect()) - } - async fn project_has_infra(&self, id: uuid::Uuid) -> anyhow::Result> { Ok(self.has_infra.read().await.get(&id).copied()) } diff --git a/crates/weft-dispatcher/src/projection.rs b/crates/weft-dispatcher/src/projection.rs index 5de821fe..f9a85b4f 100644 --- a/crates/weft-dispatcher/src/projection.rs +++ b/crates/weft-dispatcher/src/projection.rs @@ -290,11 +290,26 @@ impl ExecutionProjector { node: node_id.clone(), frames: frames.clone(), token: token.clone(), - value: effects.resumed_value.clone(), inherited_from, project_id, }] } + // A wait ends on the row that answers, skips or gives it up, + // or on its registration when that ending was on record first; + // the fold says which row ended it, and a row that came after + // the first ending paints nothing. + ExecEvent::SuspensionRegistered { .. } + | ExecEvent::SuspensionResolved { .. } + | ExecEvent::SuspensionSkipped { .. } + | ExecEvent::SuspensionGaveUp { .. } => effects.wait_ended.iter().map(|ended| DispatcherEvent::WaitEnded { + execution_id, at_unix, + node: ended.at.node_id.clone(), + frames: ended.at.frames.clone(), + token: ended.token.clone(), + ended: ended.end.clone(), + inherited_from, + project_id, + }).collect(), ExecEvent::NodeCancelled { node_id, frames, reason, .. } => { vec![DispatcherEvent::NodeCancelled { execution_id, at_unix, @@ -513,8 +528,8 @@ impl ExecutionProjector { reason: reason.clone(), at_unix, }] } - // SuspensionRegistered / SuspensionResolved / LogLine / - // RunOutput / NodeKicked / PulsesConsumed / LoopStreamEnded: + // LogLine / RunOutput / NodeKicked / PulsesConsumed / + // LoopStreamEnded: // not surfaced through DispatcherEvent. SSE consumers don't // need them for live UI; they read the journal directly when // they want full detail. A kick can still fire a group @@ -588,6 +603,7 @@ fn inherited_events( DispatcherEvent::NodeStarted { execution_id, node, frames, inherited_from, .. } | DispatcherEvent::NodeSuspended { execution_id, node, frames, inherited_from, .. } | DispatcherEvent::NodeResumed { execution_id, node, frames, inherited_from, .. } + | DispatcherEvent::WaitEnded { execution_id, node, frames, inherited_from, .. } | DispatcherEvent::NodeCompleted { execution_id, node, frames, inherited_from, .. } | DispatcherEvent::NodeSkipped { execution_id, node, frames, inherited_from, .. } => { if seed.origins.get(&Located::at(node.as_str(), frames)) != Some(&ancestor.execution_id) { continue; } @@ -1061,6 +1077,10 @@ mod tests { }); } rows.splice(3..3, [ + ExecEvent::SuspensionRegistered { + execution_id: execution_id(), node_id: "src".into(), frames: vec![], token: "answer".into(), + spec: wait_spec(), call_index: 0, at_unix: 1, + }, ExecEvent::NodeSuspended { execution_id: execution_id(), node_id: "src".into(), frames: vec![], token: "answer".into(), at_unix: 1, }, @@ -1085,20 +1105,49 @@ mod tests { assert_eq!(kinds(&events), vec![ ("node_started".into(), "src".into()), ("node_suspended".into(), "src".into()), + ("wait_ended".into(), "src".into()), ("node_resumed".into(), "src".into()), ("node_completed".into(), "src".into()), ("cost_reported".into(), "".into()), ]); - assert_eq!(events[2]["value"], "approved"); - assert_eq!(events[3]["output"]["out"], 9); - assert_eq!(events[4]["node_id"], "src"); - assert_eq!(events[4]["amount_usd"], 0.25); + assert_eq!(events[2]["ended"], json!({ "type": "answered", "value": "approved" })); + assert_eq!(events[4]["output"]["out"], 9); + assert_eq!(events[5]["node_id"], "src"); + assert_eq!(events[5]["amount_usd"], 0.25); for event in events { assert_eq!(event["execution_id"], child.to_string()); assert_eq!(event["inherited_from"], execution_id().to_string()); } } + fn wait_spec() -> weft_core::primitive::SignalSpec { + weft_core::signal::to_spec(weft_core::signal::Form { + form_type: "human_query".into(), schema: weft_core::signal::FormSchema { fields: Vec::new() }, + title: None, description: None, consumer_kind: None, + }) + } + + /// A wait the run held in its worker never suspends or resumes the + /// firing, so its ending is painted from the row that ended it: here + /// the give-up, and the answer that came after it paints nothing. + #[test] + fn a_held_wait_given_up_paints_its_ending_once() { + let mut rows: Vec = run_rows(false).into_iter().take(3).collect(); + rows.extend([ + ExecEvent::SuspensionRegistered { + execution_id: execution_id(), node_id: "src".into(), frames: vec![], token: "t".into(), + spec: wait_spec(), call_index: 0, at_unix: 2, + }, + ExecEvent::SuspensionGaveUp { execution_id: execution_id(), token: "t".into(), error: "node 'src' gave up its wait: quiet".into(), at_unix: 3 }, + ExecEvent::SuspensionResolved { execution_id: execution_id(), token: "t".into(), value: json!("late"), at_unix: 4 }, + ]); + let ended: Vec = project_all(&rows).into_iter().filter(|e| e["kind"] == "wait_ended").collect(); + assert_eq!(ended.len(), 1, "{ended:?}"); + assert_eq!(ended[0]["node"], "src"); + assert_eq!(ended[0]["token"], "t"); + assert_eq!(ended[0]["ended"], json!({ "type": "gave_up", "error": "node 'src' gave up its wait: quiet" })); + } + #[test] fn a_boundary_created_by_seed_inputs_is_visible_in_the_next_seed() { let middle = uuid::Uuid::new_v4(); diff --git a/crates/weft-dispatcher/src/reaper.rs b/crates/weft-dispatcher/src/reaper.rs index 37233b2e..6ac23998 100644 --- a/crates/weft-dispatcher/src/reaper.rs +++ b/crates/weft-dispatcher/src/reaper.rs @@ -363,22 +363,14 @@ where /// - an activation stuck `activating` -> the same wipe the activate /// rollback / cancel-activate performs (end the claim + cancel the /// leaked TriggerSetup execution + drop half-registered signals). -/// - stuck `building` / `cancelling_build` -> clear the marker; the -/// build died with its dispatcher (or keeps running harmlessly to -/// a content-addressed tag); the next verb rebuilds or cache-hits. +/// - `building` / `cancelling_build` with no live request and no +/// waiting version -> back at rest (`crate::transition::settle`): a +/// request that died before it wrote its version down, or an end +/// whose settle was missed. A waiting version keeps the marker: the +/// build loop sees it through and settles the project at its end. async fn sweep_stuck_transitions(state: DispatcherState) -> anyhow::Result<()> { let stale_before = crate::lease::now_unix() - crate::transition::heartbeat_stale_secs(); - for stuck in state.projects.list_stuck_transitions(stale_before).await? { - tracing::warn!( - target: "weft_dispatcher::reaper", - project_id = %stuck.id, - transition = stuck.transition.as_str(), - "build transition orphaned (driver heartbeat stale); clearing marker" - ); - if state.projects.finish_building(stuck.id).await? { - crate::transition::publish_transition_changed(&state, stuck.id).await; - } - } + crate::transition::settle(&state, None).await?; for stuck in state.activations.list_stuck(stale_before).await? { tracing::warn!( target: "weft_dispatcher::reaper", diff --git a/crates/weft-dispatcher/src/task_kinds/fire_signal.rs b/crates/weft-dispatcher/src/task_kinds/fire_signal.rs index 5b131322..2850b4b3 100644 --- a/crates/weft-dispatcher/src/task_kinds/fire_signal.rs +++ b/crates/weft-dispatcher/src/task_kinds/fire_signal.rs @@ -49,7 +49,7 @@ impl TaskExecutor for FireSignalExecutor { let takes_no_work = routing.standing().arrival(crate::lease::now_unix()) == weft_core::arrival::Arrival::Refused; // A retry of this same task finds the wait answered already, and // drops it as a duplicate below. - let status = match crate::api::signal::answer_run(state, &payload.token, &routing, payload.value).await { + let status = match crate::api::signal::answer_run(state, &payload.token, &routing, weft_core::primitive::WaitAnswer::Given { value: payload.value }).await { Ok(status) if takes_no_work => { let_go_of_it(state, &payload.token).await?; status diff --git a/crates/weft-dispatcher/src/transition.rs b/crates/weft-dispatcher/src/transition.rs index 4234233a..26b9a08c 100644 --- a/crates/weft-dispatcher/src/transition.rs +++ b/crates/weft-dispatcher/src/transition.rs @@ -17,10 +17,25 @@ //! transition around a version's build. The gate engages ONLY when //! an image actually has to be built (a build whose images all exist //! never flips the marker, so concurrent builds of an unchanged -//! project never serialize), and relays the user's cancel request -//! (`transition = cancelling_build`) into the builder's await loop. +//! project never serialize). +//! - [`settle`]: the one way the build transition lands back at rest. +//! +//! Who holds the build transition: the request that entered it, while its +//! builds are being started (its heartbeat, which beats until the last +//! start is recorded and the version waiting on them written down, even +//! when the request itself went away), and then the project's waiting +//! version (`crate::build::waiting`), until it registers or ends. The +//! request answers as soon as its builds run, so it cannot hold the +//! transition for their length; nor can a second marker or a task that +//! outlives it, which a dispatcher that scales to zero would lose. The +//! marker is cleared by whoever sees that nothing holds it any more +//! (`crate::project_store::build_held`): the request letting go, a +//! registration, the build loop, a cancel, and the stuck-transition +//! reaper, all through [`settle`]. The marker stays the single value every +//! guard reads (an activation's CAS, the status, the editor's cancel bar). use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::Mutex; use axum::http::StatusCode; @@ -44,10 +59,10 @@ pub fn heartbeat_stale_secs() -> i64 { weft_core::time_scale::scaled_secs(60) } -/// Drop-guarded heartbeat: bumps `transition_heartbeat_unix` every -/// [`heartbeat_interval`] until dropped. Hold it for exactly the -/// window the transition is driven in-process (the activate window, -/// the build await); dropping it stops the bumps so an orphaned row +/// Drop-guarded heartbeat of a project's build transition: bumps +/// `transition_heartbeat_unix` every [`heartbeat_interval`] until dropped. +/// Held for exactly the window a build request drives the transition +/// ([`ProjectBuildGate`]); dropping it stops the bumps so an orphaned row /// goes stale and the reaper repairs it. pub struct TransitionHeartbeat { handle: tokio::task::JoinHandle<()>, @@ -167,65 +182,44 @@ pub(crate) async fn publish_transition_changed(state: &DispatcherState, id: uuid .await; } +/// Land at rest every project in a build transition that nothing holds +/// any more (`ProjectStoreOps::settle_building`), `project` narrowing it +/// to one, and announce each. +pub(crate) async fn settle(state: &DispatcherState, project: Option) -> anyhow::Result<()> { + let stale_before = crate::lease::now_unix() - heartbeat_stale_secs(); + for id in state.projects.settle_building(project, stale_before).await? { + publish_transition_changed(state, id).await; + } + Ok(()) +} + /// The weft impl of the builder's `BuildGate`: ties the builder's /// "a real build is starting" knowledge to the project row's -/// `building` transition + heartbeat + the user's cancel request. +/// `building` transition and its heartbeat. pub struct ProjectBuildGate { state: DispatcherState, id: uuid::Uuid, engaged: AtomicBool, - saw_cancel: AtomicBool, - heartbeat: std::sync::Mutex>, + /// Beats from `begin` until `let_go`. + heartbeat: Mutex>, } impl ProjectBuildGate { fn new(state: DispatcherState, id: uuid::Uuid) -> Self { - Self { - state, - id, - engaged: AtomicBool::new(false), - saw_cancel: AtomicBool::new(false), - heartbeat: std::sync::Mutex::new(None), - } + Self { state, id, engaged: AtomicBool::new(false), heartbeat: Mutex::new(None) } } /// Whether `begin` engaged the `building` transition. fn engaged(&self) -> bool { self.engaged.load(Ordering::Acquire) } - - /// Whether the builder observed a cancel request via this gate. - fn saw_cancel(&self) -> bool { - self.saw_cancel.load(Ordering::Acquire) - } - - /// Land the transition back at rest (if engaged) and stop the - /// heartbeat. Idempotent; safe when the reaper already cleared. - async fn finish(&self) { - // Stop bumping first so a failed clear goes stale and the - // reaper repairs it, rather than us keeping a zombie fresh. - self.heartbeat.lock().expect("heartbeat mutex").take(); - if !self.engaged() { - return; - } - if let Err(e) = self.state.projects.finish_building(self.id).await { - tracing::error!( - target: "weft_dispatcher::transition", - project_id = %self.id, - error = %e, - "finish_building failed; the stuck-transition reaper will clear the \ - marker once the heartbeat goes stale" - ); - return; - } - publish_transition_changed(&self.state, self.id).await; - } } #[async_trait::async_trait] impl BuildGate for ProjectBuildGate { async fn begin(&self) -> anyhow::Result<()> { - let won = self.state.projects.try_begin_building(self.id).await?; + let stale_before = crate::lease::now_unix() - heartbeat_stale_secs(); + let won = self.state.projects.try_begin_building(self.id, stale_before).await?; if !won { // Name the blocker so the verb's 409 is actionable. let transition = self @@ -245,57 +239,77 @@ impl BuildGate for ProjectBuildGate { ); } self.engaged.store(true, Ordering::Release); - *self.heartbeat.lock().expect("heartbeat mutex") = Some(TransitionHeartbeat::spawn( - self.state.projects.clone(), - self.id, - )); + // Bumped until `let_go`, which may come after this request is gone + // (`crate::build::VersionBuilder::start_images`): the build loop + // reads a stale heartbeat as a start nobody drives any more. + *self.heartbeat.lock().expect("the gate's heartbeat slot is never poisoned") = + Some(TransitionHeartbeat::spawn(self.state.projects.clone(), self.id)); publish_transition_changed(&self.state, self.id).await; Ok(()) } - async fn cancel_requested(&self) -> anyhow::Result { - let cancelling = self - .state - .projects - .transition(self.id) - .await? - .map(|t| t == ProjectTransition::CancellingBuild) - .unwrap_or(false); - if cancelling { - self.saw_cancel.store(true, Ordering::Release); + /// Stop the heartbeat and let go of the transition (if engaged): from + /// here it rests on the project's waiting version, and lands at rest + /// now when there is none (every image was there after all, the + /// request failed, or a cancel ended it). Idempotent; safe when the + /// reaper already cleared. A failed release goes stale, and the reaper + /// settles it. + async fn let_go(&self) { + if !self.engaged() { + return; + } + drop(self.heartbeat.lock().expect("the gate's heartbeat slot is never poisoned").take()); + let released = match self.state.projects.release_build_driver(self.id).await { + Ok(()) => settle(&self.state, Some(self.id)).await, + Err(e) => Err(e), + }; + if let Err(e) = released { + tracing::error!( + target: "weft_dispatcher::transition", + project_id = %self.id, + error = %format!("{e:#}"), + "letting go of the build transition failed; the stuck-transition reaper \ + settles it once the heartbeat goes stale" + ); } - Ok(cancelling) } } /// Build one version of a project through the `building` transition: /// the gate engages only when an image actually has to be built -/// (single-flight per project, heartbeat, cancellable through -/// `/cancel-build`). A build that loses the single-flight is a 409 -/// naming the in-flight state; a user cancel is a 409 "build cancelled" -/// rather than a 500. +/// (single-flight per project, cancellable through `/cancel-build`), and +/// is let go once the version waiting on its builds is written down, +/// leaving the transition to it. A build that loses the single-flight is a +/// 409 naming the in-flight state, marked +/// [`weft_core::builds::BUILD_BUSY_HEADER`] (a caller resending an ask +/// whose answer it lost waits and sends it again); a user cancel is a 409 +/// "build cancelled" rather than a 500. pub(crate) async fn build_version_gated( state: &DispatcherState, id: uuid::Uuid, tenant: &crate::tenant::TenantId, request: &weft_core::builds::VersionBuildRequest, - hold: &crate::build::prune::ImageHold, -) -> Result { - let gate = ProjectBuildGate::new(state.clone(), id); + ask: i64, + hold: crate::build::prune::ImageHold, +) -> Result { + use axum::response::IntoResponse; + let gate = std::sync::Arc::new(ProjectBuildGate::new(state.clone(), id)); let storage = crate::storage::BrokerStorage(state); - let result = state.builder.build(&storage, id, tenant.as_str(), request, &gate, hold).await; - gate.finish().await; - match result { + // The gate is let go where it was entered, with the starts + // (`crate::build::VersionBuilder::start_images`). + match state.builder.build(&storage, id, tenant.as_str(), request, ask, gate.clone(), hold).await { Ok(built) => Ok(built), - Err(e) if gate.saw_cancel() => Err((StatusCode::CONFLICT, format!("build cancelled by user: {e}"))), + Err(e) if crate::build::cancelled(&e) => { + Err((StatusCode::CONFLICT, format!("build cancelled by user: {e:#}")).into_response()) + } // A lost single-flight (gate.begin refused) is a state // conflict, not a server fault. Err(e) if !gate.engaged() && format!("{e}").starts_with("cannot build now") => { - Err((StatusCode::CONFLICT, format!("{e}"))) + Err((StatusCode::CONFLICT, [(weft_core::builds::BUILD_BUSY_HEADER, "1")], format!("{e}")).into_response()) } // `{e:#}` (alternate) prints the FULL chain so the compile // diagnostics (`line:col message`) reach the client, not just // the outermost context. - Err(e) => Err((StatusCode::UNPROCESSABLE_ENTITY, format!("{e:#}"))), + Err(e) => Err((StatusCode::UNPROCESSABLE_ENTITY, format!("{e:#}")).into_response()), } } diff --git a/crates/weft-dispatcher/tests/db_build.rs b/crates/weft-dispatcher/tests/db_build.rs index 3d3071d6..2126af6f 100644 --- a/crates/weft-dispatcher/tests/db_build.rs +++ b/crates/weft-dispatcher/tests/db_build.rs @@ -1,26 +1,37 @@ //! A version build's image half against a real database: the ledger that -//! lets any dispatcher find and join a build, and see it through whoever -//! looks, driven through the builder with a fake `ImageBuilder`. +//! lets any dispatcher find and join a build, the build loop's look that +//! sees it through, the waiting version it registers once its builds +//! ended, and the build transition that version holds until then, driven +//! through the builder with a fake `ImageBuilder`. //! //! Same rig as `db_versions.rs`: `#[sqlx::test]` hands each test a fresh //! database; the dispatcher's whole schema is applied as a boot does. #![cfg(feature = "db-tests")] -use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicI64, AtomicUsize, Ordering}; use std::sync::Arc; use sqlx::PgPool; use weft_compiler::build_plan::{ImageKind, PlannedImage}; +use weft_core::builds::{BuiltProgram, BuildsUnderway, ImageBuild, VersionBuildStatus}; +use weft_core::projects::BuildState; use weft_dispatcher::build::ledger::{self, Claim}; use weft_dispatcher::build::prune::{ImageHold, ImageScope, KeepSet}; -use weft_dispatcher::build::{follow, BuildGate, VersionBuilder}; -use weft_platform_traits::{BuildStatus, FakeImageBuilder}; +use weft_dispatcher::build::waiting::{self, Registrar}; +use weft_dispatcher::build::{follow, BuildGate, Version, VersionBuild, VersionBuilder}; +use weft_platform_traits::{BuildStatus, FakeImageBuilder, ImageBuilder, Staging}; + +const TENANT: &str = "local"; async fn setup(pool: &PgPool) { weft_dispatcher::app::apply_core_schema(pool).await.expect("core schema"); } fn builder(pool: &PgPool, images: Arc) -> VersionBuilder { + builder_on(pool, images) +} + +fn builder_on(pool: &PgPool, images: Arc) -> VersionBuilder { VersionBuilder { bases: weft_compiler::worker_image::BaseImages { builder: "reg:5000/base:1".into(), @@ -29,7 +40,6 @@ fn builder(pool: &PgPool, images: Arc) -> VersionBuilder { images, pool: pool.clone(), compile_lanes: 2, - poll_every: std::time::Duration::from_millis(10), prunes: Default::default(), blobs: weft_dispatcher::build::blob_cache::BlobCache::in_temp_dir(), } @@ -45,11 +55,65 @@ fn image(kind: ImageKind, image_ref: &str) -> PlannedImage { } } -/// A gate that counts entries and answers a cancel when told to. +/// The asks of the tests that take no number from a project row: later +/// asks get greater numbers, as they would from one. +static ASKS: AtomicI64 = AtomicI64::new(1); + +/// `project`'s version whose worker is `binary`, running `images`, asked +/// now. +fn version(project: uuid::Uuid, binary: &str, images: &[PlannedImage]) -> Version { + version_asked(project, binary, images, ASKS.fetch_add(1, Ordering::SeqCst)) +} + +/// [`version`], asked as the project's `ask`th build. +fn version_asked(project: uuid::Uuid, binary: &str, images: &[PlannedImage], ask: i64) -> Version { + let mut refs: Vec = images.iter().map(|image| image.image_ref.clone()).collect(); + refs.sort(); + refs.dedup(); + Version { + name: "p".into(), + manifest: Default::default(), + program: BuiltProgram { + definition: serde_json::from_value(serde_json::json!({ "id": project, "nodes": [], "edges": [] })).unwrap(), + binary_hash: binary.into(), + definition_hash: format!("def-{binary}"), + infra_hash: "infra".into(), + implementations: Default::default(), + infra_images: Default::default(), + replaced_infra_images: Vec::new(), + }, + images: refs, + ask, + } +} + +/// Ask for `project`'s version `binary` of `images`, as a build request does. +async fn ask( + builder: &VersionBuilder, + pool: &PgPool, + project: uuid::Uuid, + binary: &str, + images: &[PlannedImage], + gate: Arc, +) -> anyhow::Result { + builder + .start_images(version(project, binary, images), images, &Staging::new(()), project, TENANT, gate, ImageHold::new(pool, project)) + .await +} + +/// The version build a request answered with while its builds run. +fn waiting_on(asked: VersionBuild) -> BuildsUnderway { + match asked { + VersionBuild::Waiting(underway) => underway, + VersionBuild::Ready { .. } => panic!("every image was there: nothing to wait on"), + } +} + +/// A gate that counts entries and records being let go. #[derive(Default)] struct Gate { begun: AtomicUsize, - cancel: AtomicBool, + let_go: AtomicBool, } #[async_trait::async_trait] @@ -58,181 +122,549 @@ impl BuildGate for Gate { self.begun.fetch_add(1, Ordering::SeqCst); Ok(()) } - async fn cancel_requested(&self) -> anyhow::Result { - Ok(self.cancel.load(Ordering::SeqCst)) + async fn let_go(&self) { + self.let_go.store(true, Ordering::SeqCst); } } -/// Answer the only started build with `status` once it exists. -fn finish_started(fake: Arc, status: BuildStatus) -> tokio::task::JoinHandle<()> { - tokio::spawn(async move { - loop { - if let Some(started) = fake.starts().first() { - fake.set_poll_result(&started.name, status.clone()); - return; - } - tokio::time::sleep(std::time::Duration::from_millis(5)).await; - } - }) +fn gate() -> Arc { + Arc::new(Gate::default()) +} + +/// How the row of `image_ref` stands. +async fn status_of(pool: &PgPool, image_ref: &str) -> String { + sqlx::query_scalar("SELECT status FROM image_build WHERE image_ref = $1") + .bind(image_ref) + .fetch_one(pool) + .await + .unwrap() +} + +/// Where the build `name` of `image_ref` stands, as a caller following it +/// reads it. +async fn build_state(pool: &PgPool, image_ref: &str, name: &str) -> weft_core::projects::BuildStateResponse { + let states = ledger::image_states(pool, &[ImageBuild { image: image_ref.into(), name: name.into() }]).await.unwrap(); + states.into_iter().next().unwrap().state +} + +/// How `project`'s version build `id` stands. +async fn version_state(pool: &PgPool, project: uuid::Uuid, id: uuid::Uuid) -> weft_core::builds::VersionBuildState { + waiting::state(pool, project, id).await.unwrap().expect("the version build is on record") +} + +fn crate_now() -> i64 { + std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_secs() as i64 +} + +/// Project rows registered under the test tenant, and the store the build +/// transition and registrations go through. +async fn project_store(pool: &PgPool, ids: &[uuid::Uuid]) -> weft_dispatcher::ProjectStore { + let projects: weft_dispatcher::ProjectStore = Arc::new(weft_dispatcher::PostgresProjectStore::new(pool.clone())); + for id in ids { + let empty = serde_json::from_value(serde_json::json!({ "id": id, "nodes": [], "edges": [] })).unwrap(); + projects.register_with_hashes(empty, "p", "", TENANT, None, None, None, None, None, None, None).await.unwrap(); + } + projects +} + +/// Heartbeats older than this are a request that let go. +fn stale_before() -> i64 { + crate_now() - weft_dispatcher::transition::heartbeat_stale_secs() +} + +async fn transition_of(projects: &weft_dispatcher::ProjectStore, id: uuid::Uuid) -> &'static str { + projects.transition(id).await.unwrap().unwrap().as_str() +} + +async fn running_binary(pool: &PgPool, project: uuid::Uuid) -> Option { + sqlx::query_scalar("SELECT running_binary_hash FROM project WHERE id = $1").bind(project).fetch_one(pool).await.unwrap() +} + +async fn claims_of(pool: &PgPool, claim: uuid::Uuid) -> i64 { + sqlx::query_scalar("SELECT count(*) FROM image_claim WHERE claim_id = $1").bind(claim).fetch_one(pool).await.unwrap() } /// Images already in the registry are not built, and the gate is never -/// entered: a build of an unchanged project never serializes. +/// entered: a build of an unchanged project never serializes, and the +/// request registers at once. #[sqlx::test] async fn present_images_are_skipped_without_the_gate(pool: PgPool) { setup(&pool).await; let fake = Arc::new(FakeImageBuilder::new()); fake.set_image_exists("reg:5000/weft-worker:a"); - let gate = Gate::default(); - builder(&pool, fake.clone()) - .ensure_images(&[image(ImageKind::Worker, "reg:5000/weft-worker:a")], uuid::Uuid::new_v4(), "local", &gate, &ImageHold::new(&pool, uuid::Uuid::new_v4())) + let gate = gate(); + let asked = ask(&builder(&pool, fake.clone()), &pool, uuid::Uuid::new_v4(), "a", &[image(ImageKind::Worker, "reg:5000/weft-worker:a")], gate.clone()) .await .unwrap(); + assert!(matches!(asked, VersionBuild::Ready { .. }), "nothing to wait on"); assert!(fake.starts().is_empty()); assert_eq!(gate.begun.load(Ordering::SeqCst), 0); } -/// A stale image is built once, in a lane, and its row records the end. +/// A stale image is started once, in a lane, and the request answers with +/// the version waiting on it while it still runs, never waiting for its +/// end; the build loop's look records the end later. #[sqlx::test] -async fn a_stale_image_is_built_and_recorded(pool: PgPool) { +async fn a_stale_image_is_started_and_answered_before_it_ends(pool: PgPool) { setup(&pool).await; let fake = Arc::new(FakeImageBuilder::new()); - let gate = Gate::default(); - let done = finish_started(fake.clone(), BuildStatus::Succeeded); - builder(&pool, fake.clone()) - .ensure_images( - &[image(ImageKind::Worker, "reg:5000/weft-worker:b"), image(ImageKind::Worker, "reg:5000/weft-worker:b")], - uuid::Uuid::new_v4(), - "local", - &gate, &ImageHold::new(&pool, uuid::Uuid::new_v4())) - .await - .unwrap(); - done.await.unwrap(); + let gate = gate(); + let image_ref = "reg:5000/weft-worker:b"; + let project = uuid::Uuid::new_v4(); + let underway = waiting_on( + ask(&builder(&pool, fake.clone()), &pool, project, "b", &[image(ImageKind::Worker, image_ref), image(ImageKind::Worker, image_ref)], gate.clone()) + .await + .unwrap(), + ); let starts = fake.starts(); assert_eq!(starts.len(), 1, "one build per ref, however often the plan names it"); + assert_eq!(underway.images, [ImageBuild { image: image_ref.into(), name: starts[0].name.clone() }]); assert_eq!(starts[0].build_args, vec![("WEFT_COMPILE_LANE".to_string(), "0".to_string())]); assert_eq!(gate.begun.load(Ordering::SeqCst), 1); - let (status,): (String,) = sqlx::query_as("SELECT status FROM image_build WHERE image_ref = $1") - .bind("reg:5000/weft-worker:b") - .fetch_one(&pool) + assert!(gate.let_go.load(Ordering::SeqCst), "let go once the version is written down"); + assert_eq!(status_of(&pool, image_ref).await, "running", "answered while it runs"); + assert_eq!(version_state(&pool, project, underway.build).await.state, VersionBuildStatus::Waiting); + fake.set_poll_result(&starts[0].name, BuildStatus::Succeeded); + assert!(follow::look(&pool, fake.as_ref()).await.unwrap()); + assert_eq!(status_of(&pool, image_ref).await, "succeeded"); + assert!(!follow::look(&pool, fake.as_ref()).await.unwrap(), "nothing runs any more"); +} + +/// A version whose builds all succeed registers by itself on the build +/// loop's next pass: the project runs it, the claim on its images goes, +/// and the caller following it reads it registered, with the program. +#[sqlx::test] +async fn a_waiting_version_registers_by_itself_once_its_builds_succeed(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[project]).await; + let fake = Arc::new(FakeImageBuilder::new()); + let images = [image(ImageKind::Worker, "reg:5000/weft-worker:self"), image(ImageKind::Infra, "reg:5000/weft-infra-db:self")]; + fake.set_image_exists("reg:5000/weft-infra-db:self"); + let underway = waiting_on(ask(&builder(&pool, fake.clone()), &pool, project, "self", &images, gate()).await.unwrap()); + assert_eq!(underway.images.len(), 1, "only the missing image is built"); + assert_eq!(claims_of(&pool, underway.build).await, 2, "the version's claim spares every image it runs"); + + let early = waiting::advance(&pool, projects.as_ref()).await.unwrap(); + assert!(early.waiting && early.registered.is_empty(), "nothing registers while a build runs"); + fake.set_poll_result(&underway.images[0].name, BuildStatus::Succeeded); + follow::look(&pool, fake.as_ref()).await.unwrap(); + let advanced = waiting::advance(&pool, projects.as_ref()).await.unwrap(); + assert_eq!(advanced.registered.iter().map(|s| s.id).collect::>(), [project]); + assert!(!advanced.waiting && advanced.failed.is_empty()); + + let state = version_state(&pool, project, underway.build).await; + assert_eq!(state.state, VersionBuildStatus::Registered); + assert_eq!(state.program.expect("the program registered").binary_hash, "self"); + assert_eq!(state.images[0].state.state, Some(BuildState::Succeeded)); + assert_eq!(running_binary(&pool, project).await.as_deref(), Some("self")); + assert_eq!(claims_of(&pool, underway.build).await, 0, "the claim goes with the registration"); + let uses: i64 = sqlx::query_scalar("SELECT count(*) FROM image_use WHERE project_id = $1").bind(project).fetch_one(&pool).await.unwrap(); + assert_eq!(uses, 2, "its images are the project's running version"); +} + +/// A version an image build of which failed ends failed, naming every +/// failed image and why, and never registers. +#[sqlx::test] +async fn a_failed_image_build_ends_its_version_failed(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[project]).await; + let fake = Arc::new(FakeImageBuilder::new()); + let images = [image(ImageKind::Worker, "reg:5000/weft-worker:ok"), image(ImageKind::Infra, "reg:5000/weft-infra-x:bad")]; + let underway = waiting_on(ask(&builder(&pool, fake.clone()), &pool, project, "f", &images, gate()).await.unwrap()); + for build in &underway.images { + let status = if build.image.contains("bad") { + BuildStatus::Failed { reason: "cargo: error[E0425]".into() } + } else { + BuildStatus::Succeeded + }; + fake.set_poll_result(&build.name, status); + } + follow::look(&pool, fake.as_ref()).await.unwrap(); + let advanced = waiting::advance(&pool, projects.as_ref()).await.unwrap(); + assert!(advanced.registered.is_empty() && !advanced.waiting); + let state = version_state(&pool, project, underway.build).await; + assert_eq!(state.state, VersionBuildStatus::Failed); + let reason = state.reason.expect("why it failed"); + assert!(reason.contains("weft-infra-x:bad") && reason.contains("E0425") && !reason.contains("weft-worker:ok"), "{reason}"); + assert_eq!(running_binary(&pool, project).await, None, "nothing registered"); + assert_eq!(claims_of(&pool, underway.build).await, 0, "its claim goes as it ends"); +} + +/// A newer ask of another version supersedes the one waiting, and the +/// older one never registers over it, even when its builds end last. +#[sqlx::test] +async fn a_newer_ask_supersedes_the_waiting_version(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[project]).await; + let fake = Arc::new(FakeImageBuilder::new()); + let builder = builder(&pool, fake.clone()); + let older = waiting_on(ask(&builder, &pool, project, "old", &[image(ImageKind::Worker, "reg:5000/weft-worker:old")], gate()).await.unwrap()); + let newer = waiting_on(ask(&builder, &pool, project, "new", &[image(ImageKind::Worker, "reg:5000/weft-worker:new")], gate()).await.unwrap()); + assert_ne!(older.build, newer.build); + let old_state = version_state(&pool, project, older.build).await; + assert_eq!(old_state.state, VersionBuildStatus::Superseded); + assert!(old_state.reason.is_some_and(|r| r.contains("newer build"))); + assert_eq!(claims_of(&pool, older.build).await, 0, "the superseded version's claim goes"); + + fake.set_poll_result(&newer.images[0].name, BuildStatus::Succeeded); + follow::look(&pool, fake.as_ref()).await.unwrap(); + waiting::advance(&pool, projects.as_ref()).await.unwrap(); + assert_eq!(running_binary(&pool, project).await.as_deref(), Some("new")); + fake.set_poll_result(&older.images[0].name, BuildStatus::Succeeded); + follow::look(&pool, fake.as_ref()).await.unwrap(); + let late = waiting::advance(&pool, projects.as_ref()).await.unwrap(); + assert!(late.registered.is_empty(), "the superseded version never registers"); + assert_eq!(running_binary(&pool, project).await.as_deref(), Some("new")); + + // Its registration, even attempted, does not land. + let landed = waiting::register_version( + projects.as_ref(), + &pool, + project, + TENANT, + &version(project, "old", &[image(ImageKind::Worker, "reg:5000/weft-worker:old")]), + Registrar::Loop { version: older.build }, + ) + .await + .unwrap(); + assert!(landed.is_none()); + assert_eq!(running_binary(&pool, project).await.as_deref(), Some("new")); +} + +/// Register `project`'s version `binary` as a request whose ask is `ask` +/// and that found every image; `false` when it must not land. +async fn register_asked(pool: &PgPool, projects: &weft_dispatcher::ProjectStore, project: uuid::Uuid, binary: &str, ask: i64) -> bool { + let hold = ImageHold::new(pool, project); + let version = version_asked(project, binary, &[image(ImageKind::Worker, &format!("reg:5000/weft-worker:{binary}"))], ask); + waiting::register_version(projects.as_ref(), pool, project, TENANT, &version, Registrar::Request { hold }) + .await + .unwrap() + .is_some() +} + +/// Ask for `project`'s version `binary` as its `ask`th build, which waits +/// on the image build it starts. +async fn wait_asked(pool: &PgPool, fake: &Arc, project: uuid::Uuid, binary: &str, ask: i64) -> BuildsUnderway { + let images = [image(ImageKind::Worker, &format!("reg:5000/weft-worker:{binary}"))]; + waiting_on( + builder(pool, fake.clone()) + .start_images(version_asked(project, binary, &images, ask), &images, &Staging::new(()), project, TENANT, gate(), ImageHold::new(pool, project)) + .await + .unwrap(), + ) +} + +/// A request that found every image registers at once, and settles the +/// project's waiting version by the order of their asks, never by when +/// either finished: the waiting one ends `registered` when it is the same +/// version, `superseded` when it was asked before, and keeps waiting (to +/// register over this one) when it was asked after. +#[sqlx::test] +async fn an_ask_that_registers_settles_the_waiting_version(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[project]).await; + let fake = Arc::new(FakeImageBuilder::new()); + let early = projects.next_build_ask(project).await.unwrap(); + let late = projects.next_build_ask(project).await.unwrap(); + // The later ask compiled faster and waits; the earlier one finds every + // image and registers after it. + let waiting = wait_asked(&pool, &fake, project, "late", late).await; + assert!(register_asked(&pool, &projects, project, "early", early).await, "nothing newer registered yet"); + assert_eq!(version_state(&pool, project, waiting.build).await.state, VersionBuildStatus::Waiting, "asked after it: still waits"); + fake.set_poll_result(&waiting.images[0].name, BuildStatus::Succeeded); + follow::look(&pool, fake.as_ref()).await.unwrap(); + waiting::advance(&pool, projects.as_ref()).await.unwrap(); + assert_eq!(running_binary(&pool, project).await.as_deref(), Some("late"), "the newer ask lands over the older"); + + // The other order: the waiting version was asked first, so the request + // that registers supersedes it, and it never lands after. + let first = projects.next_build_ask(project).await.unwrap(); + let older = wait_asked(&pool, &fake, project, "first", first).await; + let second = projects.next_build_ask(project).await.unwrap(); + assert!(register_asked(&pool, &projects, project, "second", second).await); + let state = version_state(&pool, project, older.build).await; + assert_eq!(state.state, VersionBuildStatus::Superseded); + assert_eq!(state.reason.as_deref(), Some("a newer build of this project registered first")); + fake.set_poll_result(&older.images[0].name, BuildStatus::Succeeded); + follow::look(&pool, fake.as_ref()).await.unwrap(); + assert!(waiting::advance(&pool, projects.as_ref()).await.unwrap().registered.is_empty()); + assert_eq!(running_binary(&pool, project).await.as_deref(), Some("second")); + + // The same version waiting is ended registered by the request. + let third = projects.next_build_ask(project).await.unwrap(); + let same = wait_asked(&pool, &fake, project, "third", third).await; + let fourth = projects.next_build_ask(project).await.unwrap(); + assert!(register_asked(&pool, &projects, project, "third", fourth).await); + let state = version_state(&pool, project, same.build).await; + assert_eq!(state.state, VersionBuildStatus::Registered, "the same version, registered by the request"); + assert_eq!(state.program.unwrap().binary_hash, "third"); +} + +/// A version registered from an ask lands nothing older after it: a slow +/// request whose ask came before finds the newer one registered and +/// registers nothing, and a version asked before that only now waits is +/// written down superseded. +#[sqlx::test] +async fn an_older_ask_never_lands_over_a_registered_newer_one(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[project]).await; + let fake = Arc::new(FakeImageBuilder::new()); + let slow = projects.next_build_ask(project).await.unwrap(); + let slower = projects.next_build_ask(project).await.unwrap(); + let newest = projects.next_build_ask(project).await.unwrap(); + let waiting = wait_asked(&pool, &fake, project, "newest", newest).await; + fake.set_poll_result(&waiting.images[0].name, BuildStatus::Succeeded); + follow::look(&pool, fake.as_ref()).await.unwrap(); + waiting::advance(&pool, projects.as_ref()).await.unwrap(); + assert_eq!(running_binary(&pool, project).await.as_deref(), Some("newest")); + + assert!(!register_asked(&pool, &projects, project, "slow", slow).await, "the older request registers nothing"); + assert_eq!(running_binary(&pool, project).await.as_deref(), Some("newest")); + let late = wait_asked(&pool, &fake, project, "slower", slower).await; + let state = version_state(&pool, project, late.build).await; + assert_eq!(state.state, VersionBuildStatus::Superseded, "written down superseded already"); + assert_eq!(claims_of(&pool, late.build).await, 0, "and its claim let go"); +} + +/// A version waiting behind a newer one still waiting is written down +/// superseded, and the newer one keeps waiting. +#[sqlx::test] +async fn an_older_ask_never_supersedes_a_newer_waiting_one(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[project]).await; + let fake = Arc::new(FakeImageBuilder::new()); + let older = projects.next_build_ask(project).await.unwrap(); + let newer = projects.next_build_ask(project).await.unwrap(); + let new = wait_asked(&pool, &fake, project, "newer", newer).await; + let old = wait_asked(&pool, &fake, project, "older", older).await; + assert_eq!(version_state(&pool, project, old.build).await.state, VersionBuildStatus::Superseded); + assert_eq!(version_state(&pool, project, new.build).await.state, VersionBuildStatus::Waiting); +} + +/// Two requests that both found every image, registering at once, leave +/// the project on the newer ask's version, whichever commits first. +#[sqlx::test] +async fn two_concurrent_registrations_leave_the_newer_ask(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[project]).await; + for _ in 0..10 { + let older = projects.next_build_ask(project).await.unwrap(); + let newer = projects.next_build_ask(project).await.unwrap(); + let (a, b) = tokio::join!( + register_asked(&pool, &projects, project, "old", older), + register_asked(&pool, &projects, project, "new", newer), + ); + assert!(b, "the newer ask always lands"); + let _ = a; + assert_eq!(running_binary(&pool, project).await.as_deref(), Some("new")); + } +} + +/// A waiting version whose stored rows no longer read ends failed with +/// why, rather than keeping the build loop awake and its project building +/// for ever. +#[sqlx::test] +async fn a_waiting_version_that_no_longer_reads_ends_failed(pool: PgPool) { + setup(&pool).await; + let (garbled_images, garbled_program) = (uuid::Uuid::new_v4(), uuid::Uuid::new_v4()); + let projects = project_store(&pool, &[garbled_images, garbled_program]).await; + for (project, program, waits_on) in [ + (garbled_images, serde_json::json!({}), serde_json::json!({ "not": "a list" })), + (garbled_program, serde_json::json!({ "not": "a program" }), serde_json::json!([])), + ] { + assert!(projects.try_begin_building(project, stale_before()).await.unwrap()); + projects.release_build_driver(project).await.unwrap(); + sqlx::query( + "INSERT INTO version_build (id, ask, project_id, tenant_id, project_name, manifest, program, images, waits_on, \ + state, asked_at) \ + VALUES ($1, 1, $1, $2, 'p', '{}', $3, '{}', $4, 'waiting', 0)", + ) + .bind(project) + .bind(TENANT) + .bind(program) + .bind(waits_on) + .execute(&pool) .await .unwrap(); - assert_eq!(status, "succeeded"); + } + let advanced = waiting::advance(&pool, projects.as_ref()).await.unwrap(); + assert!(!advanced.waiting && advanced.failed.is_empty() && advanced.registered.is_empty()); + for (project, says) in [(garbled_images, "do not read"), (garbled_program, "does not read")] { + let (state, reason): (String, Option) = + sqlx::query_as("SELECT state, reason FROM version_build WHERE id = $1").bind(project).fetch_one(&pool).await.unwrap(); + assert_eq!(state, "failed"); + assert!(reason.as_deref().is_some_and(|r| r.contains(says)), "{reason:?}"); + } + let mut settled = projects.settle_building(None, stale_before()).await.unwrap(); + settled.sort(); + let mut both = vec![garbled_images, garbled_program]; + both.sort(); + assert_eq!(settled, both, "nothing holds either project any more"); } -/// A failed build fails the verb with the builder's reason, and a later -/// build of the same ref starts afresh. +/// The build loop forgets version builds that ended over an hour ago, and +/// a removed project's go with it, a waiting one's claim included. #[sqlx::test] -async fn a_failed_build_names_its_reason_and_can_be_retried(pool: PgPool) { +async fn ended_and_removed_version_builds_are_forgotten(pool: PgPool) { setup(&pool).await; + let (old, removed) = (uuid::Uuid::new_v4(), uuid::Uuid::new_v4()); + let projects = project_store(&pool, &[]).await; let fake = Arc::new(FakeImageBuilder::new()); - let gate = Gate::default(); - let done = finish_started(fake.clone(), BuildStatus::Failed { reason: "cargo: error[E0425]".into() }); - let e = builder(&pool, fake.clone()) - .ensure_images(&[image(ImageKind::Infra, "reg:5000/weft-infra-x:c")], uuid::Uuid::new_v4(), "local", &gate, &ImageHold::new(&pool, uuid::Uuid::new_v4())) + let ended = waiting_on(ask(&builder(&pool, fake.clone()), &pool, old, "ended", &[image(ImageKind::Worker, "reg:5000/weft-worker:ended")], gate()).await.unwrap()); + builder(&pool, fake.clone()).cancel_project(old).await.unwrap(); + sqlx::query("UPDATE version_build SET ended_at = ended_at - $2 WHERE id = $1") + .bind(ended.build) + .bind(waiting::ENDED_KEPT_SECS + 1) + .execute(&pool) .await - .unwrap_err() - .to_string(); - done.await.unwrap(); - assert!(e.contains("E0425"), "{e}"); + .unwrap(); + waiting::advance(&pool, projects.as_ref()).await.unwrap(); + assert!(waiting::state(&pool, old, ended.build).await.unwrap().is_none(), "an hour after it ended"); + + let gone = waiting_on(ask(&builder(&pool, fake.clone()), &pool, removed, "removed", &[image(ImageKind::Worker, "reg:5000/weft-worker:removed")], gate()).await.unwrap()); + waiting::forget_project(&pool, removed).await.unwrap(); + assert!(waiting::state(&pool, removed, gone.build).await.unwrap().is_none()); + assert_eq!(claims_of(&pool, gone.build).await, 0); +} + +/// The same version asked again while it waits (a rerun after Ctrl+C) +/// joins it: the same version build, the same image build, never a second +/// start, and the rerun's own claim let go. +#[sqlx::test] +async fn a_rerun_of_the_same_version_joins_it(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let fake = Arc::new(FakeImageBuilder::new()); + let builder = builder(&pool, fake.clone()); + let images = [image(ImageKind::Worker, "reg:5000/weft-worker:again")]; + let first = waiting_on(ask(&builder, &pool, project, "again", &images, gate()).await.unwrap()); + let hold = ImageHold::new(&pool, project); + let rerun_claim = hold.id(); + let rerun = waiting_on( + builder + .start_images(version(project, "again", &images), &images, &Staging::new(()), project, TENANT, gate(), hold) + .await + .unwrap(), + ); + assert_eq!(rerun.build, first.build, "joined"); + assert_eq!(rerun.images, first.images); + assert_eq!(fake.starts().len(), 1, "never started twice"); + assert_eq!(claims_of(&pool, rerun_claim).await, 0, "the rerun's own claim goes"); + assert_eq!(claims_of(&pool, first.build).await, 1); + let waiting: i64 = sqlx::query_scalar("SELECT count(*) FROM version_build WHERE project_id = $1").bind(project).fetch_one(&pool).await.unwrap(); + assert_eq!(waiting, 1); +} + +/// A failed build's reason is what a caller following it reads, and a +/// later build of the same ref starts afresh. +#[sqlx::test] +async fn a_failed_build_names_its_reason_and_can_be_retried(pool: PgPool) { + setup(&pool).await; + let fake = Arc::new(FakeImageBuilder::new()); + let project = uuid::Uuid::new_v4(); + let image_ref = "reg:5000/weft-infra-x:c"; + let underway = waiting_on(ask(&builder(&pool, fake.clone()), &pool, project, "c", &[image(ImageKind::Infra, image_ref)], gate()).await.unwrap()); assert!(fake.starts()[0].build_args.is_empty(), "an infra image has no compile lane"); - let claim = ledger::claim(&pool, fake.as_ref(), "reg:5000/weft-infra-x:c", uuid::Uuid::new_v4(), "local", Some(2), 0).await.unwrap(); + fake.set_poll_result(&underway.images[0].name, BuildStatus::Failed { reason: "cargo: error[E0425]".into() }); + follow::look(&pool, fake.as_ref()).await.unwrap(); + let state = build_state(&pool, image_ref, &underway.images[0].name).await; + assert_eq!(state.state, Some(BuildState::Failed)); + assert!(state.reason.as_deref().is_some_and(|r| r.contains("E0425")), "{state:?}"); + let claim = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), TENANT, Some(2), 0).await.unwrap(); assert!(matches!(claim, Claim::Start { .. }), "{claim:?}"); } -/// A cancel of the project stops the build it started. +/// Once the project's build is cancelled, an image claim is refused: a +/// request claiming after the cancel starts nothing, and fails cancelled. #[sqlx::test] -async fn a_cancel_stops_the_projects_own_build(pool: PgPool) { +async fn a_claim_after_a_cancel_is_refused(pool: PgPool) { setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[project]).await; + assert!(projects.try_begin_building(project, stale_before()).await.unwrap()); + assert!(projects.request_cancel_build(project).await.unwrap()); let fake = Arc::new(FakeImageBuilder::new()); - let gate = Gate::default(); - gate.cancel.store(true, Ordering::SeqCst); - let e = builder(&pool, fake.clone()) - .ensure_images(&[image(ImageKind::Worker, "reg:5000/weft-worker:d")], uuid::Uuid::new_v4(), "local", &gate, &ImageHold::new(&pool, uuid::Uuid::new_v4())) - .await - .unwrap_err() - .to_string(); - assert!(e.contains("cancelled"), "{e}"); - assert_eq!(fake.releases().len(), 1, "its build is freed"); - let (status,): (String,) = sqlx::query_as("SELECT status FROM image_build WHERE image_ref = $1") - .bind("reg:5000/weft-worker:d") - .fetch_one(&pool) - .await - .unwrap(); - assert_eq!(status, "cancelled"); + let Err(e) = ask(&builder(&pool, fake.clone()), &pool, project, "d", &[image(ImageKind::Worker, "reg:5000/weft-worker:d")], gate()).await else { + panic!("a cancelled build starts nothing") + }; + assert!(weft_dispatcher::build::cancelled(&e), "{e:#}"); + assert!(fake.starts().is_empty()); + let rows: i64 = sqlx::query_scalar("SELECT count(*) FROM image_build").fetch_one(&pool).await.unwrap(); + assert_eq!(rows, 0, "nothing claimed"); } /// A second project asking for an image another one is building joins -/// that build (no second start), and its cancel leaves the other's build -/// running. +/// that build (no second start, the same build answered), and its cancel +/// leaves the other's build running while its own waiting version ends. #[sqlx::test] async fn a_shared_build_is_joined_and_never_stopped_by_the_joiner(pool: PgPool) { setup(&pool).await; let first = uuid::Uuid::new_v4(); let fake = Arc::new(FakeImageBuilder::new()); - let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), "reg:5000/weft-worker:e", first, "local", Some(2), crate_now()).await.unwrap() else { + let image_ref = "reg:5000/weft-worker:e"; + let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, first, TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; - let gate = Gate::default(); - gate.cancel.store(true, Ordering::SeqCst); - let e = builder(&pool, fake.clone()) - .ensure_images(&[image(ImageKind::Worker, "reg:5000/weft-worker:e")], uuid::Uuid::new_v4(), "local", &gate, &ImageHold::new(&pool, uuid::Uuid::new_v4())) - .await - .unwrap_err() - .to_string(); - assert!(e.contains("cancelled"), "{e}"); + let joiner = uuid::Uuid::new_v4(); + let joined = waiting_on(ask(&builder(&pool, fake.clone()), &pool, joiner, "e", &[image(ImageKind::Worker, image_ref)], gate()).await.unwrap()); + assert_eq!(joined.images, [ImageBuild { image: image_ref.into(), name: name.clone() }], "the running build is answered"); + builder(&pool, fake.clone()).cancel_project(joiner).await.unwrap(); assert!(fake.starts().is_empty(), "joined, not started again"); assert!(fake.releases().is_empty(), "another project's build is left running"); - let (status,): (String,) = sqlx::query_as("SELECT status FROM image_build WHERE build_name = $1") - .bind(&name) - .fetch_one(&pool) - .await - .unwrap(); - assert_eq!(status, "running"); + assert_eq!(status_of(&pool, image_ref).await, "running"); + assert_eq!(version_state(&pool, joiner, joined.build).await.state, VersionBuildStatus::Cancelled); } -/// Two projects waiting on one build both get its end. The one that sees -/// it end frees the build's process, so the other would find it gone: the -/// end is read from the ledger, where it is recorded first. +/// Two projects asking at once for one image start one build and each +/// waits on it; once it ends, both versions register. #[sqlx::test] -async fn every_waiter_on_one_build_gets_its_end(pool: PgPool) { +async fn every_waiter_on_one_build_registers_at_its_end(pool: PgPool) { setup(&pool).await; + let (a_project, b_project) = (uuid::Uuid::new_v4(), uuid::Uuid::new_v4()); + let projects = project_store(&pool, &[a_project, b_project]).await; let fake = Arc::new(FakeImageBuilder::new()); let wanted = [image(ImageKind::Worker, "reg:5000/weft-worker:g")]; - let (first_gate, second_gate) = (Gate::default(), Gate::default()); let (first, second) = (builder(&pool, fake.clone()), builder(&pool, fake.clone())); - let done = finish_started(fake.clone(), BuildStatus::Succeeded); - let (first_hold, second_hold) = (ImageHold::new(&pool, uuid::Uuid::new_v4()), ImageHold::new(&pool, uuid::Uuid::new_v4())); let (a, b) = tokio::join!( - first.ensure_images(&wanted, uuid::Uuid::new_v4(), "local", &first_gate, &first_hold), - second.ensure_images(&wanted, uuid::Uuid::new_v4(), "local", &second_gate, &second_hold), + ask(&first, &pool, a_project, "g", &wanted, gate()), + ask(&second, &pool, b_project, "g", &wanted, gate()), ); - // The verbs first: when both fail before starting anything, `done` - // never sees a build and would wait forever, hiding why. - a.unwrap(); - b.unwrap(); - done.await.unwrap(); + let (a, b) = (waiting_on(a.unwrap()), waiting_on(b.unwrap())); assert_eq!(fake.starts().len(), 1, "one build for both"); + assert_eq!(a.images, b.images, "both wait on the same build"); + fake.set_poll_result(&a.images[0].name, BuildStatus::Succeeded); + follow::look(&pool, fake.as_ref()).await.unwrap(); + let mut registered: Vec = waiting::advance(&pool, projects.as_ref()).await.unwrap().registered.iter().map(|s| s.id).collect(); + registered.sort(); + let mut both = vec![a_project, b_project]; + both.sort(); + assert_eq!(registered, both); assert!(!fake.releases().is_empty(), "the build is freed once the end is recorded"); } -/// A build nobody waits on any more (its verb stopped waiting, or its -/// dispatcher went away) is seen through by whoever looks next: the +/// A build nobody follows any more is seen through by the next look: the /// build loop records its end from the builder's answer. #[sqlx::test] -async fn a_build_nobody_waits_on_is_seen_through_by_the_next_look(pool: PgPool) { +async fn a_build_nobody_follows_is_seen_through_by_the_next_look(pool: PgPool) { setup(&pool).await; let fake = Arc::new(FakeImageBuilder::new()); let image_ref = "reg:5000/weft-worker:f"; - let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), "local", Some(2), crate_now()).await.unwrap() else { + let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; assert!(ledger::started(&pool, image_ref, &name, &weft_platform_traits::BuildHandle::named(name.clone())).await.unwrap()); - follow::advance(&pool, fake.as_ref(), image_ref).await.unwrap(); - assert_eq!(ledger::look(&pool, image_ref).await.unwrap(), ledger::Seen::Running, "still building"); + follow::look(&pool, fake.as_ref()).await.unwrap(); + assert_eq!(status_of(&pool, image_ref).await, "running", "still building"); fake.set_poll_result(&name, BuildStatus::Succeeded); - follow::advance(&pool, fake.as_ref(), image_ref).await.unwrap(); - assert_eq!(ledger::look(&pool, image_ref).await.unwrap(), ledger::Seen::Ended(ledger::Outcome::Succeeded)); + follow::look(&pool, fake.as_ref()).await.unwrap(); + assert_eq!(status_of(&pool, image_ref).await, "succeeded"); assert_eq!(fake.releases(), vec![name], "freed once its end is recorded"); } @@ -244,56 +676,42 @@ async fn a_build_the_builder_named_is_followed_under_that_name(pool: PgPool) { setup(&pool).await; let fake = Arc::new(FakeImageBuilder::new()); fake.name_builds_itself(); - let gate = Gate::default(); - let done = tokio::spawn({ - let fake = fake.clone(); - async move { - loop { - if let Some(started) = fake.starts().first() { - fake.set_poll_result(&format!("{}-id", started.name), BuildStatus::Succeeded); - return; - } - tokio::time::sleep(std::time::Duration::from_millis(5)).await; - } - } - }); - builder(&pool, fake.clone()) - .ensure_images(&[image(ImageKind::Worker, "reg:5000/weft-worker:n")], uuid::Uuid::new_v4(), "local", &gate, &ImageHold::new(&pool, uuid::Uuid::new_v4())) - .await - .unwrap(); - done.await.unwrap(); - let (current,): (String,) = sqlx::query_as("SELECT builder_id FROM image_build WHERE image_ref = $1") - .bind("reg:5000/weft-worker:n") - .fetch_one(&pool) - .await - .unwrap(); + let project = uuid::Uuid::new_v4(); + let underway = waiting_on(ask(&builder(&pool, fake.clone()), &pool, project, "n", &[image(ImageKind::Worker, "reg:5000/weft-worker:n")], gate()).await.unwrap()); + let state = version_state(&pool, project, underway.build).await; + let current = state.images[0].state.build.clone().expect("the builder's id is recorded"); assert!(current.ends_with("-id"), "the row names the builder's own id: {current}"); + fake.set_poll_result(¤t, BuildStatus::Succeeded); + follow::look(&pool, fake.as_ref()).await.unwrap(); + assert_eq!(status_of(&pool, "reg:5000/weft-worker:n").await, "succeeded"); } -/// A start that outlasts its hold (its dispatcher went away mid-start) -/// ends failed by name, and a start that lands on a row no longer its -/// build's (cancelled meanwhile) frees what it made instead of leaving it -/// running with nothing tracking it. +/// A start whose project no request drives any more (its dispatcher went +/// away mid-start) ends failed by name, and a start that lands on a row +/// no longer its build's (ended or cancelled meanwhile) frees what it made +/// instead of leaving it running with nothing tracking it. #[sqlx::test] async fn a_start_that_never_lands_ends_and_a_late_one_is_freed(pool: PgPool) { setup(&pool).await; let fake = Arc::new(FakeImageBuilder::new()); let image_ref = "reg:5000/weft-worker:s"; - let long_ago = crate_now() - ledger::START_HOLD_SECS - 1; - let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), "local", Some(2), long_ago).await.unwrap() else { + let project = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[project]).await; + assert!(projects.try_begin_building(project, stale_before()).await.unwrap()); + let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, project, TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; - follow::advance(&pool, fake.as_ref(), image_ref).await.unwrap(); - let ledger::Seen::Ended(ledger::Outcome::Failed(reason)) = ledger::look(&pool, image_ref).await.unwrap() else { - panic!("a start that never landed ends failed") - }; - assert!(reason.contains("never made on the builder"), "{reason}"); + projects.release_build_driver(project).await.unwrap(); + follow::look(&pool, fake.as_ref()).await.unwrap(); + let state = build_state(&pool, image_ref, &name).await; + assert_eq!(state.state, Some(BuildState::Failed), "a start that never landed ends failed"); + assert!(state.reason.as_deref().is_some_and(|r| r.contains("never made on the builder")), "{state:?}"); assert!(fake.polls().is_empty(), "a build with no id on the builder is never asked about"); let handle = weft_platform_traits::BuildHandle::named(name.clone()); assert!(!ledger::started(&pool, image_ref, &name, &handle).await.unwrap(), "the row is no longer that build's"); let image_ref = "reg:5000/weft-worker:t"; - let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), "local", Some(2), crate_now()).await.unwrap() else { + let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; assert_eq!(ledger::cancel(&pool, image_ref, &name, crate_now()).await.unwrap(), Some(None), "cancelled before it was made"); @@ -307,111 +725,83 @@ async fn a_build_its_builder_never_answers_about_is_given_up(pool: PgPool) { setup(&pool).await; let fake = Arc::new(FakeImageBuilder::new()); let image_ref = "reg:5000/weft-worker:u"; - let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), "local", Some(2), crate_now()).await.unwrap() else { + let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; assert!(ledger::started(&pool, image_ref, &name, &weft_platform_traits::BuildHandle::named(name.clone())).await.unwrap()); fake.fail_polls(&name); - follow::advance(&pool, fake.as_ref(), image_ref).await.unwrap(); - assert_eq!(ledger::look(&pool, image_ref).await.unwrap(), ledger::Seen::Running, "one unanswered look is looked at again"); + follow::look(&pool, fake.as_ref()).await.unwrap(); + assert_eq!(status_of(&pool, image_ref).await, "running", "one unanswered look is looked at again"); sqlx::query("UPDATE image_build SET failing_since = failing_since - $2 WHERE image_ref = $1") .bind(image_ref) .bind(ledger::UNANSWERED_GIVE_UP_SECS + 1) .execute(&pool) .await .unwrap(); - follow::advance(&pool, fake.as_ref(), image_ref).await.unwrap(); - let ledger::Seen::Ended(ledger::Outcome::Failed(reason)) = ledger::look(&pool, image_ref).await.unwrap() else { - panic!("given up on") - }; - assert!(reason.contains("has not answered"), "{reason}"); + follow::look(&pool, fake.as_ref()).await.unwrap(); + let state = build_state(&pool, image_ref, &name).await; + assert_eq!(state.state, Some(BuildState::Failed), "given up on"); + assert!(state.reason.as_deref().is_some_and(|r| r.contains("has not answered")), "{state:?}"); } -/// A build its builder no longer knows, once its start hold passed, ends -/// failed by name instead of being waited on for ever. +/// A build its builder no longer knows ends failed by name instead of +/// being waited on for ever. #[sqlx::test] async fn a_build_its_builder_lost_fails_by_name(pool: PgPool) { setup(&pool).await; let fake = Arc::new(FakeImageBuilder::new()); let image_ref = "reg:5000/weft-worker:l"; - let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), "local", Some(2), crate_now()).await.unwrap() else { + let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; assert!(ledger::started(&pool, image_ref, &name, &weft_platform_traits::BuildHandle::named(name.clone())).await.unwrap()); fake.set_poll_result(&name, BuildStatus::Gone); - follow::advance(&pool, fake.as_ref(), image_ref).await.unwrap(); - let ledger::Seen::Ended(ledger::Outcome::Failed(reason)) = ledger::look(&pool, image_ref).await.unwrap() else { - panic!("a lost build ends failed") - }; - assert!(reason.contains(&name) && reason.contains("gone"), "{reason}"); + follow::look(&pool, fake.as_ref()).await.unwrap(); + let state = build_state(&pool, image_ref, &name).await; + assert_eq!(state.state, Some(BuildState::Failed), "a lost build ends failed"); + assert!(state.reason.as_deref().is_some_and(|r| r.contains(&name) && r.contains("gone")), "{state:?}"); } -/// A verb cancelled while it waits on another project's build stops -/// waiting, and the build goes on to its end without it. +/// A project joining a build whose starter is still making it is +/// answered with it at once, and the build loop never calls it gone +/// while its starter still drives it, though the builder knows nothing of +/// it yet. #[sqlx::test] -async fn a_cancelled_joiner_leaves_the_build_to_finish(pool: PgPool) { +async fn a_joined_build_still_being_made_is_never_called_gone(pool: PgPool) { setup(&pool).await; - let image_ref = "reg:5000/weft-worker:h"; let fake = Arc::new(FakeImageBuilder::new()); - let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), "local", Some(2), crate_now()).await.unwrap() else { - panic!("the first claim starts") - }; - assert!(ledger::started(&pool, image_ref, &name, &weft_platform_traits::BuildHandle::named(name.clone())).await.unwrap()); - let gate = Gate::default(); - gate.cancel.store(true, Ordering::SeqCst); - let e = builder(&pool, fake.clone()) - .ensure_images(&[image(ImageKind::Worker, image_ref)], uuid::Uuid::new_v4(), "local", &gate, &ImageHold::new(&pool, uuid::Uuid::new_v4())) - .await - .unwrap_err() - .to_string(); - assert!(e.contains("cancelled"), "{e}"); - fake.set_poll_result(&name, BuildStatus::Succeeded); - follow::advance(&pool, fake.as_ref(), image_ref).await.unwrap(); - assert_eq!(ledger::look(&pool, image_ref).await.unwrap(), ledger::Seen::Ended(ledger::Outcome::Succeeded)); -} - -/// A verb joining a build whose starter is still making it never calls -/// it gone while the start hold lasts, though the builder knows nothing -/// of it yet. -#[sqlx::test] -async fn a_joiner_never_calls_a_build_still_being_made_gone(pool: PgPool) { - setup(&pool).await; - let fake = Arc::new(FakeImageBuilder::new()); - let project = uuid::Uuid::new_v4(); - let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), "reg:5000/weft-worker:j", project, "local", Some(2), crate_now()).await.unwrap() else { + let image_ref = "reg:5000/weft-worker:j"; + let starter = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[starter]).await; + assert!(projects.try_begin_building(starter, stale_before()).await.unwrap()); + let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, starter, TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; // The starter has not created the process yet: it polls as gone. fake.set_poll_result(&name, BuildStatus::Gone); - let joiner = tokio::spawn({ - let (pool, fake) = (pool.clone(), fake.clone()); - async move { - builder(&pool, fake) - .ensure_images(&[image(ImageKind::Worker, "reg:5000/weft-worker:j")], project, "local", &Gate::default(), &ImageHold::new(&pool, uuid::Uuid::new_v4())) - .await - } - }); - tokio::time::sleep(std::time::Duration::from_millis(100)).await; - assert!(!joiner.is_finished(), "the joiner waits"); - ledger::finish(&pool, "reg:5000/weft-worker:j", &name, ledger::Outcome::Succeeded, crate_now()).await.unwrap(); - joiner.await.unwrap().unwrap(); + let joiner = uuid::Uuid::new_v4(); + let underway = waiting_on(ask(&builder(&pool, fake.clone()), &pool, joiner, "j", &[image(ImageKind::Worker, image_ref)], gate()).await.unwrap()); + assert_eq!(underway.images[0].name, name); + follow::look(&pool, fake.as_ref()).await.unwrap(); + assert_eq!(status_of(&pool, image_ref).await, "running", "not gone while its start may still land"); + let state = version_state(&pool, joiner, underway.build).await; + assert_eq!((state.images[0].state.state, state.images[0].state.build.clone()), (Some(BuildState::Running), None), "followed before the builder named it"); } /// A verb that found the image missing while another verb's build of it /// was running, and claims it only once that build succeeded, builds -/// nothing: the image is there. (Building it again started a build -/// nothing ever finished, which hung the two-waiter test under load.) +/// nothing: the image is there. #[sqlx::test] async fn a_claim_after_the_build_succeeded_builds_nothing(pool: PgPool) { setup(&pool).await; let fake = FakeImageBuilder::new(); let image_ref = "reg:5000/weft-worker:k"; - let Claim::Start { name, .. } = ledger::claim(&pool, &fake, image_ref, uuid::Uuid::new_v4(), "local", Some(2), crate_now()).await.unwrap() else { + let Claim::Start { name, .. } = ledger::claim(&pool, &fake, image_ref, uuid::Uuid::new_v4(), TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; ledger::finish(&pool, image_ref, &name, ledger::Outcome::Succeeded, crate_now()).await.unwrap(); fake.set_image_exists(image_ref); - let late = ledger::claim(&pool, &fake, image_ref, uuid::Uuid::new_v4(), "local", Some(2), crate_now()).await.unwrap(); + let late = ledger::claim(&pool, &fake, image_ref, uuid::Uuid::new_v4(), TENANT, Some(2), crate_now()).await.unwrap(); assert!(matches!(late, Claim::Built), "{late:?}"); } @@ -423,21 +813,16 @@ async fn a_built_image_the_registry_lost_is_built_again(pool: PgPool) { setup(&pool).await; let fake = Arc::new(FakeImageBuilder::new()); let image_ref = "reg:5000/weft-worker:lost"; - let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), "local", Some(2), crate_now()).await.unwrap() else { + let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, uuid::Uuid::new_v4(), TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; ledger::finish(&pool, image_ref, &name, ledger::Outcome::Succeeded, crate_now()).await.unwrap(); - let done = finish_started(fake.clone(), BuildStatus::Succeeded); - let built = builder(&pool, fake.clone()) - .ensure_images(&[image(ImageKind::Worker, image_ref)], uuid::Uuid::new_v4(), "local", &Gate::default(), &ImageHold::new(&pool, uuid::Uuid::new_v4())) - .await - .unwrap(); - done.await.unwrap(); - assert_eq!(built, [image_ref]); + let underway = waiting_on(ask(&builder(&pool, fake.clone()), &pool, uuid::Uuid::new_v4(), "lost", &[image(ImageKind::Worker, image_ref)], gate()).await.unwrap()); let starts = fake.starts(); assert_eq!(starts.len(), 1, "built again"); assert_ne!(starts[0].name, name, "under a new build"); - assert_eq!(ledger::look(&pool, image_ref).await.unwrap(), ledger::Seen::Ended(ledger::Outcome::Succeeded)); + assert_eq!(underway.images, [ImageBuild { image: image_ref.into(), name: starts[0].name.clone() }]); + assert_eq!(status_of(&pool, image_ref).await, "running"); } /// Racing claims of one ref start it exactly once. @@ -448,7 +833,7 @@ async fn racing_claims_start_one_build(pool: PgPool) { for _ in 0..20 { let p = pool.clone(); claims.push(tokio::spawn(async move { - ledger::claim(&p, &FakeImageBuilder::new(), "reg:5000/weft-worker:g", uuid::Uuid::new_v4(), "local", Some(4), 0).await.unwrap() + ledger::claim(&p, &FakeImageBuilder::new(), "reg:5000/weft-worker:g", uuid::Uuid::new_v4(), TENANT, Some(4), 0).await.unwrap() })); } let mut starts = 0; @@ -460,10 +845,6 @@ async fn racing_claims_start_one_build(pool: PgPool) { assert_eq!(starts, 1); } -fn crate_now() -> i64 { - std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_secs() as i64 -} - /// A prune spares an image a build in progress claims (it found the /// image and has not registered it yet), and takes it once the claim is /// let go, without waiting on any other build. @@ -493,6 +874,31 @@ async fn a_prune_spares_an_image_a_build_claims(pool: PgPool) { other.release().await.unwrap(); } +/// The images a waiting version runs are spared by a prune for as long as +/// it waits, however long ago its request's lease ran out, and are a +/// prune's to judge once it ended. +#[sqlx::test] +async fn a_waiting_versions_images_are_spared_until_it_ends(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let fake = Arc::new(FakeImageBuilder::new()); + let runner = weft_platform_traits::FakeRunner::new("http://w"); + let found = "reg:5000/weft-worker:found".to_string(); + ledger::note_shared(&pool, &found, crate_now()).await.unwrap(); + fake.set_image_exists(&found); + let images = [image(ImageKind::Worker, &found), image(ImageKind::Infra, "reg:5000/weft-infra-db:building")]; + let underway = waiting_on(ask(&builder(&pool, fake.clone()), &pool, project, "spared", &images, gate()).await.unwrap()); + // As if the request answered long ago: its lease ran out. + sqlx::query("UPDATE image_claim SET holder_until = 0").execute(&pool).await.unwrap(); + let candidates = std::slice::from_ref(&found); + let spared = weft_dispatcher::build::prune::prune(&pool, fake.as_ref(), &runner, candidates, &KeepSet::default()).await.unwrap(); + assert_eq!(spared.in_use, std::slice::from_ref(&found), "spared while the version waits"); + assert_eq!(claims_of(&pool, underway.build).await, 2, "and its claim is not swept as lapsed"); + builder(&pool, fake.clone()).cancel_project(project).await.unwrap(); + let taken = weft_dispatcher::build::prune::prune(&pool, fake.as_ref(), &runner, candidates, &KeepSet::default()).await.unwrap(); + assert_eq!(taken.removed, std::slice::from_ref(&found), "a prune's once the version ended"); +} + /// A prune whose keep-set was read before a build registered an image as /// running must not delete it: the build claimed the image, registered it /// and let go of the claim, all between the keep-set read and the prune @@ -540,7 +946,7 @@ async fn a_prune_takes_what_failed_and_cancelled_builds_left(pool: PgPool) { let end = |image_ref: &'static str, outcome: ledger::Outcome| { let (pool, fake) = (pool.clone(), &fake); async move { - let Claim::Start { name, .. } = ledger::claim(&pool, fake, image_ref, uuid::Uuid::new_v4(), "local", Some(2), crate_now()).await.unwrap() else { + let Claim::Start { name, .. } = ledger::claim(&pool, fake, image_ref, uuid::Uuid::new_v4(), TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; ledger::finish(&pool, image_ref, &name, outcome, crate_now()).await.unwrap(); @@ -553,7 +959,7 @@ async fn a_prune_takes_what_failed_and_cancelled_builds_left(pool: PgPool) { fake.set_image_exists(pushed); fake.set_image_exists(cancelled); let running = "reg:5000/weft-worker:running"; - let Claim::Start { .. } = ledger::claim(&pool, &fake, running, uuid::Uuid::new_v4(), "local", Some(2), crate_now()).await.unwrap() else { + let Claim::Start { .. } = ledger::claim(&pool, &fake, running, uuid::Uuid::new_v4(), TENANT, Some(2), crate_now()).await.unwrap() else { panic!("the first claim starts") }; let candidates = ledger::built_images(&pool, None).await.unwrap(); @@ -565,5 +971,399 @@ async fn a_prune_takes_what_failed_and_cancelled_builds_left(pool: PgPool) { assert_eq!(report.in_use, [running], "a running build is spared"); assert_eq!(fake.deletes(), [cancelled, nothing, pushed], "a delete of an image never pushed deletes nothing, and is no error"); assert!(ledger::built_images(&pool, None).await.unwrap().is_empty(), "every ended row is forgotten"); - assert_eq!(ledger::look(&pool, running).await.unwrap(), ledger::Seen::Running); + assert_eq!(status_of(&pool, running).await, "running"); +} + +/// The build transition outlives the request that entered it: once the +/// request let go, the version waiting on its builds holds it, a settle +/// leaves it while that version waits (a build ending is not enough), and +/// the first settle after the version registered lands it at rest. A +/// project waiting on a build another project started is held the same +/// way. +#[sqlx::test] +async fn a_waiting_version_holds_the_transition_until_it_registers(pool: PgPool) { + setup(&pool).await; + let (own, joiner) = (uuid::Uuid::new_v4(), uuid::Uuid::new_v4()); + let projects = project_store(&pool, &[own, joiner]).await; + let fake = Arc::new(FakeImageBuilder::new()); + let images = [image(ImageKind::Worker, "reg:5000/weft-worker:held")]; + + assert!(projects.try_begin_building(own, stale_before()).await.unwrap()); + let underway = waiting_on(ask(&builder(&pool, fake.clone()), &pool, own, "held", &images, gate()).await.unwrap()); + assert!(projects.settle_building(None, stale_before()).await.unwrap().is_empty(), "a live request holds it"); + projects.release_build_driver(own).await.unwrap(); + + assert!(projects.try_begin_building(joiner, stale_before()).await.unwrap()); + let joined = waiting_on(ask(&builder(&pool, fake.clone()), &pool, joiner, "held", &images, gate()).await.unwrap()); + assert_eq!(joined.images, underway.images); + projects.release_build_driver(joiner).await.unwrap(); + + fake.set_poll_result(&underway.images[0].name, BuildStatus::Succeeded); + follow::look(&pool, fake.as_ref()).await.unwrap(); + assert!(projects.settle_building(None, stale_before()).await.unwrap().is_empty(), "the build ended, but both versions wait"); + assert_eq!(transition_of(&projects, own).await, "building"); + + waiting::advance(&pool, projects.as_ref()).await.unwrap(); + let mut settled = projects.settle_building(None, stale_before()).await.unwrap(); + settled.sort(); + let mut both = vec![own, joiner]; + both.sort(); + assert_eq!(settled, both, "both registered: both at rest"); + assert_eq!(transition_of(&projects, own).await, "none"); +} + +/// A request asking again while its version waits enters the transition +/// and joins it; a request still starting builds keeps every other out. +#[sqlx::test] +async fn a_request_joins_a_waiting_version_but_never_a_live_start(pool: PgPool) { + setup(&pool).await; + let id = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[id]).await; + let fake = Arc::new(FakeImageBuilder::new()); + let images = [image(ImageKind::Worker, "reg:5000/weft-worker:again")]; + assert!(projects.try_begin_building(id, stale_before()).await.unwrap()); + assert!(!projects.try_begin_building(id, stale_before()).await.unwrap(), "a live start keeps the next out"); + let first = waiting_on(ask(&builder(&pool, fake.clone()), &pool, id, "again", &images, gate()).await.unwrap()); + projects.release_build_driver(id).await.unwrap(); + assert!(projects.try_begin_building(id, stale_before()).await.unwrap(), "only its version holds it: a new request enters"); + let again = waiting_on(ask(&builder(&pool, fake.clone()), &pool, id, "again", &images, gate()).await.unwrap()); + assert_eq!(again.build, first.build, "the waiting version is answered again"); + assert_eq!(fake.starts().len(), 1, "and its build never started twice"); +} + +/// A cancel stops the builds the project started, leaves the ones another +/// project started running, ends its waiting version cancelled, and the +/// transition lands at rest. +#[sqlx::test] +async fn a_cancel_stops_own_builds_and_ends_the_waiting_version(pool: PgPool) { + setup(&pool).await; + let (id, other) = (uuid::Uuid::new_v4(), uuid::Uuid::new_v4()); + let projects = project_store(&pool, &[id]).await; + let fake = Arc::new(FakeImageBuilder::new()); + let (mine, theirs) = ("reg:5000/weft-worker:mine", "reg:5000/weft-worker:theirs"); + let Claim::Start { .. } = ledger::claim(&pool, fake.as_ref(), theirs, other, TENANT, Some(2), crate_now()).await.unwrap() else { + panic!("the first claim starts") + }; + assert!(projects.try_begin_building(id, stale_before()).await.unwrap()); + let underway = waiting_on( + ask(&builder(&pool, fake.clone()), &pool, id, "mine", &[image(ImageKind::Worker, mine), image(ImageKind::Worker, theirs)], gate()) + .await + .unwrap(), + ); + assert_eq!(underway.images.len(), 2); + projects.release_build_driver(id).await.unwrap(); + + assert!(projects.request_cancel_build(id).await.unwrap()); + assert!(projects.request_cancel_build(id).await.unwrap(), "a cancel asked twice is the same cancel"); + builder(&pool, fake.clone()).cancel_project(id).await.unwrap(); + assert_eq!(status_of(&pool, mine).await, "cancelled"); + assert_eq!(fake.releases().len(), 1, "only its own build is freed"); + assert_eq!(status_of(&pool, theirs).await, "running", "the other project's build goes on"); + let state = version_state(&pool, id, underway.build).await; + assert_eq!(state.state, VersionBuildStatus::Cancelled); + assert_eq!(claims_of(&pool, underway.build).await, 0, "its claim goes with it"); + assert_eq!(projects.settle_building(Some(id), stale_before()).await.unwrap(), [id]); + assert_eq!(transition_of(&projects, id).await, "none"); + assert!(!projects.request_cancel_build(id).await.unwrap(), "nothing left to cancel"); +} + +/// A build announces itself to the build loop the moment it is claimed, +/// before it is made on the builder: an install whose loop was asleep +/// wakes, so a build whose starter went away mid-start is ended once its +/// project's heartbeat stops, and its project leaves the build transition +/// instead of staying building for ever. +#[sqlx::test] +async fn a_claimed_build_wakes_the_loop_and_a_dead_start_lets_its_project_go(pool: PgPool) { + setup(&pool).await; + let id = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[id]).await; + let fake = FakeImageBuilder::new(); + let image_ref = "reg:5000/weft-worker:dead-start"; + let mut listener = sqlx::postgres::PgListener::connect_with(&pool).await.unwrap(); + listener.listen(follow::BUILD_CLAIMED_CHANNEL).await.unwrap(); + assert!(projects.try_begin_building(id, stale_before()).await.unwrap()); + let Claim::Start { name, .. } = ledger::claim(&pool, &fake, image_ref, id, TENANT, Some(2), crate_now()).await.unwrap() else { + panic!("the first claim starts") + }; + let woke = tokio::time::timeout(std::time::Duration::from_secs(10), listener.recv()).await.expect("the claim announces itself").unwrap(); + assert_eq!(woke.payload(), image_ref); + // Its starter dies here: the request's heartbeat stops, and nothing + // ever records the build on the builder or writes its version down. + projects.release_build_driver(id).await.unwrap(); + assert!(!follow::look(&pool, &fake).await.unwrap(), "the woken loop ends the build, and nothing runs after"); + let state = build_state(&pool, image_ref, &name).await; + assert_eq!(state.state, Some(BuildState::Failed), "ended once nobody drives its start"); + assert_eq!(projects.settle_building(Some(id), stale_before()).await.unwrap(), [id]); + assert_eq!(transition_of(&projects, id).await, "none"); +} + +/// A version written down as waiting wakes the build loop as it commits, +/// so one whose builds already ended registers at once. +#[sqlx::test] +async fn a_waiting_version_wakes_the_loop(pool: PgPool) { + setup(&pool).await; + let id = uuid::Uuid::new_v4(); + let fake = Arc::new(FakeImageBuilder::new()); + let mut listener = sqlx::postgres::PgListener::connect_with(&pool).await.unwrap(); + listener.listen(follow::BUILD_CLAIMED_CHANNEL).await.unwrap(); + let image_ref = "reg:5000/weft-worker:wakes"; + let underway = waiting_on(ask(&builder(&pool, fake.clone()), &pool, id, "wakes", &[image(ImageKind::Worker, image_ref)], gate()).await.unwrap()); + let mut payloads = Vec::new(); + while payloads.len() < 2 { + let woke = tokio::time::timeout(std::time::Duration::from_secs(10), listener.recv()).await.expect("announced").unwrap(); + payloads.push(woke.payload().to_string()); + } + assert_eq!(payloads, [image_ref.to_string(), underway.build.to_string()], "the claim, then the version"); +} + +/// A build request that fails after joining a build another project +/// started lets go of its claim, writes no version down, and its project +/// lands at rest while that build runs on. A request abandoned midway (its +/// hold dropped) lets go the same way. +#[sqlx::test] +async fn a_failed_request_lets_go_of_its_claim_and_its_project(pool: PgPool) { + setup(&pool).await; + let (id, other) = (uuid::Uuid::new_v4(), uuid::Uuid::new_v4()); + let projects = project_store(&pool, &[id]).await; + let fake = Arc::new(FakeImageBuilder::new()); + let image_ref = "reg:5000/weft-worker:theirs-still"; + let Claim::Start { name, .. } = ledger::claim(&pool, fake.as_ref(), image_ref, other, TENANT, Some(2), crate_now()).await.unwrap() else { + panic!("the first claim starts") + }; + assert!(ledger::started(&pool, image_ref, &name, &weft_platform_traits::BuildHandle::named(name.clone())).await.unwrap()); + + assert!(projects.try_begin_building(id, stale_before()).await.unwrap()); + let hold = ImageHold::new(&pool, id); + let claim = hold.id(); + // The person cancels before the request claims its image: the claim is + // refused. + let images = [image(ImageKind::Worker, image_ref)]; + let (builder, staging) = (builder(&pool, fake.clone()), Staging::new(())); + let started = builder.start_images(version(id, "x", &images), &images, &staging, id, TENANT, gate(), hold); + assert!(projects.request_cancel_build(id).await.unwrap()); + let e = started.await.expect_err("the request fails"); + assert!(weft_dispatcher::build::cancelled(&e), "{e:#}"); + projects.release_build_driver(id).await.unwrap(); + let claims = || async { + sqlx::query_scalar::<_, i64>("SELECT count(*) FROM image_claim WHERE project_id = $1").bind(id).fetch_one(&pool).await.unwrap() + }; + for _ in 0..500 { + if claims().await == 0 { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + assert_eq!(claims().await, 0, "the failed request's claim goes with it"); + assert!(waiting::state(&pool, id, claim).await.unwrap().is_none(), "no version written down"); + assert_eq!(projects.settle_building(Some(id), stale_before()).await.unwrap(), [id], "at rest while the build runs on"); + assert_eq!(status_of(&pool, image_ref).await, "running"); + + let abandoned = ImageHold::new(&pool, id); + abandoned.claim(&[image_ref.to_string()]).await.unwrap(); + drop(abandoned); + for _ in 0..500 { + if claims().await == 0 { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + assert_eq!(claims().await, 0, "an abandoned request's claim goes with it"); +} + +/// A start whose project's heartbeat went stale (its request's dispatcher +/// died between the claim and recording the build on the builder) is +/// ended by the next look, and its project settles at once instead of +/// staying building; a start whose project a live request still drives is +/// left alone, though the builder knows nothing of it yet either. +#[sqlx::test] +async fn a_start_nobody_drives_ends_at_once_and_a_driven_one_is_left(pool: PgPool) { + setup(&pool).await; + let (live, dead) = (uuid::Uuid::new_v4(), uuid::Uuid::new_v4()); + let projects = project_store(&pool, &[live, dead]).await; + let fake = FakeImageBuilder::new(); + let (live_ref, dead_ref) = ("reg:5000/weft-worker:live-start", "reg:5000/weft-worker:dead-start-2"); + let mut names = Vec::new(); + for (project, image_ref) in [(live, live_ref), (dead, dead_ref)] { + assert!(projects.try_begin_building(project, stale_before()).await.unwrap()); + let Claim::Start { name, .. } = ledger::claim(&pool, &fake, image_ref, project, TENANT, Some(2), crate_now()).await.unwrap() else { + panic!("the first claim starts") + }; + names.push(name); + } + // The dead project's request stopped bumping a while ago, without ever + // letting go. + sqlx::query("UPDATE project SET transition_heartbeat_unix = $2 WHERE id = $1") + .bind(dead) + .bind(stale_before() - 1) + .execute(&pool) + .await + .unwrap(); + assert!(follow::look(&pool, &fake).await.unwrap()); + let ended = build_state(&pool, dead_ref, &names[1]).await; + assert_eq!(ended.state, Some(BuildState::Failed), "{ended:?}"); + assert!(ended.reason.as_deref().is_some_and(|r| r.contains(&names[1]) && r.contains("never made on the builder")), "{ended:?}"); + assert_eq!(status_of(&pool, live_ref).await, "running", "a driven start is left to its starter"); + assert!(fake.polls().is_empty(), "neither has an id on the builder to ask about"); + assert_eq!(projects.settle_building(None, stale_before()).await.unwrap(), [dead], "only the dead start's project settles"); + assert_eq!(transition_of(&projects, live).await, "building"); +} + +/// A request taking over the build transition from one that died +/// mid-start ends that dead one's starts as it enters, so its own claim +/// starts the image again under a new name rather than joining a start +/// nothing will finish; the dead start, should it ever land, finds the row +/// no longer its own. Another project joins the new start. +#[sqlx::test] +async fn a_takeover_ends_the_lost_starts_it_takes_over(pool: PgPool) { + setup(&pool).await; + let (id, other) = (uuid::Uuid::new_v4(), uuid::Uuid::new_v4()); + let projects = project_store(&pool, &[id]).await; + let fake = FakeImageBuilder::new(); + let image_ref = "reg:5000/weft-worker:retaken"; + assert!(projects.try_begin_building(id, stale_before()).await.unwrap()); + let Claim::Start { name: lost, .. } = ledger::claim(&pool, &fake, image_ref, id, TENANT, Some(2), crate_now()).await.unwrap() else { + panic!("the first claim starts") + }; + // The request died: its heartbeat went stale, and the next one enters. + sqlx::query("UPDATE project SET transition_heartbeat_unix = $2 WHERE id = $1").bind(id).bind(stale_before() - 1).execute(&pool).await.unwrap(); + assert!(projects.try_begin_building(id, stale_before()).await.unwrap()); + let ended = build_state(&pool, image_ref, &lost).await; + assert_eq!(ended.state, Some(BuildState::Failed), "ended as the new request entered: {ended:?}"); + let again = ledger::claim(&pool, &fake, image_ref, id, TENANT, Some(2), crate_now()).await.unwrap(); + let Claim::Start { name: fresh, .. } = again else { panic!("started again, not joined: {again:?}") }; + assert_ne!(fresh, lost); + assert!(!ledger::started(&pool, image_ref, &lost, &weft_platform_traits::BuildHandle::named(lost.clone())).await.unwrap()); + let joined = ledger::claim(&pool, &fake, image_ref, other, TENANT, Some(2), crate_now()).await.unwrap(); + assert_eq!(joined, Claim::Join { name: fresh, ours: false }); +} + +/// A builder whose `start` waits until it is let through, so a test can +/// drop the request in the middle of one. +struct HeldStart { + inner: Arc, + entered: tokio::sync::Notify, + go: tokio::sync::Notify, +} + +#[async_trait::async_trait] +impl ImageBuilder for HeldStart { + fn image_ref(&self, tag: &str) -> String { + self.inner.image_ref(tag) + } + async fn start(&self, req: weft_platform_traits::BuildRequest) -> anyhow::Result { + self.entered.notify_one(); + self.go.notified().await; + self.inner.start(req).await + } + async fn poll(&self, handle: &weft_platform_traits::BuildHandle) -> anyhow::Result { + self.inner.poll(handle).await + } + async fn image_exists(&self, image_ref: &str) -> anyhow::Result { + self.inner.image_exists(image_ref).await + } + async fn release(&self, handle: &weft_platform_traits::BuildHandle) { + self.inner.release(handle).await + } + async fn delete_image(&self, image_ref: &str) -> anyhow::Result { + self.inner.delete_image(image_ref).await + } +} + +/// Wait, a bounded while, for `ready` to hold. +async fn until>(mut ready: impl FnMut() -> F) { + for _ in 0..500 { + if ready().await { + return; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } +} + +/// A request dropped in the middle of a start (its caller went away) does +/// not take the start with it: the start goes on, makes the build, records +/// the builder's id and writes the version down as waiting on it, so the +/// install still registers it once it is built. The gate is let go once +/// that is done, with nobody waiting. +#[sqlx::test] +async fn a_start_finishes_when_its_request_is_dropped(pool: PgPool) { + setup(&pool).await; + let fake = Arc::new(FakeImageBuilder::new()); + let held = Arc::new(HeldStart { inner: fake.clone(), entered: Default::default(), go: Default::default() }); + let project = uuid::Uuid::new_v4(); + let image_ref = "reg:5000/weft-worker:dropped"; + let gate = gate(); + let request = tokio::spawn({ + let (builder, gate, pool) = (builder_on(&pool, held.clone()), gate.clone(), pool.clone()); + async move { ask(&builder, &pool, project, "dropped", &[image(ImageKind::Worker, image_ref)], gate).await } + }); + held.entered.notified().await; + request.abort(); + assert!(request.await.unwrap_err().is_cancelled(), "the request is gone"); + held.go.notify_one(); + until(|| async { gate.let_go.load(Ordering::SeqCst) }).await; + let starts = fake.starts(); + assert_eq!(starts.len(), 1, "the build was made"); + let builder_id: Option = + sqlx::query_scalar("SELECT builder_id FROM image_build WHERE image_ref = $1").bind(image_ref).fetch_one(&pool).await.unwrap(); + assert_eq!(builder_id, Some(starts[0].name.clone()), "and its id on the builder recorded"); + assert_eq!(status_of(&pool, image_ref).await, "running"); + assert!(fake.releases().is_empty(), "nothing freed: the build loop follows it"); + let waiting: Vec = + sqlx::query_scalar("SELECT state FROM version_build WHERE project_id = $1").bind(project).fetch_all(&pool).await.unwrap(); + assert_eq!(waiting, ["waiting"], "the version waits on it with nobody asking"); + assert!(gate.let_go.load(Ordering::SeqCst), "the gate is let go once the start is done, with nobody waiting"); +} + +/// A cancel that comes while the starts run is honoured even when the +/// request that started them is gone: the version is refused, and what +/// the start made is stopped, as it would be with the request there. +#[sqlx::test] +async fn a_cancel_during_a_start_whose_request_is_dropped_stops_its_build(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let projects = project_store(&pool, &[project]).await; + assert!(projects.try_begin_building(project, stale_before()).await.unwrap()); + let fake = Arc::new(FakeImageBuilder::new()); + let held = Arc::new(HeldStart { inner: fake.clone(), entered: Default::default(), go: Default::default() }); + let image_ref = "reg:5000/weft-worker:dropped-cancel"; + let gate = gate(); + let request = tokio::spawn({ + let (builder, gate, pool) = (builder_on(&pool, held.clone()), gate.clone(), pool.clone()); + async move { ask(&builder, &pool, project, "dropped-cancel", &[image(ImageKind::Worker, image_ref)], gate).await } + }); + held.entered.notified().await; + request.abort(); + assert!(request.await.unwrap_err().is_cancelled(), "the request is gone"); + assert!(projects.request_cancel_build(project).await.unwrap()); + held.go.notify_one(); + until(|| async { gate.let_go.load(Ordering::SeqCst) }).await; + assert_eq!(status_of(&pool, image_ref).await, "cancelled", "the cancel reached the build"); + assert_eq!(fake.releases().len(), 1, "and the build on the builder was freed"); + let versions: i64 = sqlx::query_scalar("SELECT count(*) FROM version_build").fetch_one(&pool).await.unwrap(); + assert_eq!(versions, 0, "no version written down after the cancel"); +} + +/// A request whose claim on its images lapsed while it started its +/// builds (its dispatcher could not renew it in time, so a prune may have +/// taken an image it found) fails loudly, writes no version down, and +/// stops what it started. +#[sqlx::test] +async fn a_lapsed_claim_is_never_passed_to_a_waiting_version(pool: PgPool) { + setup(&pool).await; + let project = uuid::Uuid::new_v4(); + let fake = Arc::new(FakeImageBuilder::new()); + let held = Arc::new(HeldStart { inner: fake.clone(), entered: Default::default(), go: Default::default() }); + let image_ref = "reg:5000/weft-worker:lapsed"; + let request = tokio::spawn({ + let (builder, pool) = (builder_on(&pool, held.clone()), pool.clone()); + async move { ask(&builder, &pool, project, "lapsed", &[image(ImageKind::Worker, image_ref)], gate()).await } + }); + held.entered.notified().await; + sqlx::query("UPDATE image_claim SET holder_until = 0").execute(&pool).await.unwrap(); + held.go.notify_one(); + let e = request.await.unwrap().expect_err("the request fails"); + assert!(format!("{e:#}").contains("lapsed"), "{e:#}"); + let versions: i64 = sqlx::query_scalar("SELECT count(*) FROM version_build").fetch_one(&pool).await.unwrap(); + assert_eq!(versions, 0, "no version written down"); + assert_eq!(status_of(&pool, image_ref).await, "cancelled", "what it started is stopped"); } diff --git a/crates/weft-dispatcher/tests/db_install_state.rs b/crates/weft-dispatcher/tests/db_install_state.rs index 0c339335..a87f7235 100644 --- a/crates/weft-dispatcher/tests/db_install_state.rs +++ b/crates/weft-dispatcher/tests/db_install_state.rs @@ -18,7 +18,7 @@ async fn project(pool: &PgPool) -> Uuid { let id = Uuid::new_v4(); let definition = serde_json::from_value(serde_json::json!({ "id": id, "nodes": [], "edges": [] })).expect("a definition"); store - .register_with_hashes(definition, "p", "", "tenant-1", Some("bin"), Some("def"), None, None, None, None) + .register_with_hashes(definition, "p", "", "tenant-1", Some("bin"), Some("def"), None, None, None, None, None) .await .expect("register"); id diff --git a/crates/weft-dispatcher/tests/db_lifecycle.rs b/crates/weft-dispatcher/tests/db_lifecycle.rs index 908abb03..80d28118 100644 --- a/crates/weft-dispatcher/tests/db_lifecycle.rs +++ b/crates/weft-dispatcher/tests/db_lifecycle.rs @@ -76,6 +76,7 @@ async fn seed_project( None, None, None, + None, ) .await .expect("register project"); @@ -188,13 +189,13 @@ async fn registered_sources_belong_to_the_exact_program(pool: PgPool) { let implementations = std::collections::BTreeMap::new(); let source = std::collections::BTreeMap::from([("main.weft".into(), "original-file".into())]); projects.register_with_hashes(empty_project(id), "sources", "", TENANT, - Some("binary"), Some("graph"), None, None, Some(&implementations), Some(&source)).await.unwrap(); + Some("binary"), Some("graph"), None, None, Some(&implementations), Some(&source), None).await.unwrap(); let program = projects.running_program_identity(id).await.unwrap().unwrap(); assert_eq!(projects.program_source(id, &program).await.unwrap(), source); // A later build of other code moves the running program, and the // old identity no longer answers for it. projects.register_with_hashes(empty_project(id), "sources", "", TENANT, - Some("different-binary"), Some("graph"), None, None, Some(&implementations), None).await.unwrap(); + Some("different-binary"), Some("graph"), None, None, Some(&implementations), None, None).await.unwrap(); assert!(projects.program_source(id, &program).await.is_err()); let changed = weft_core::project::hash::ProgramIdentity { binary_hash: "different-binary".into(), ..program }; assert!(projects.program_source(id, &changed).await.is_err(), "changing code cannot relabel old sources"); @@ -1017,7 +1018,7 @@ async fn referenced_images_cover_projects_runs_workers_maps_and_unit_refs( /// An event waiting for its trigger, its `attempts`-th failure behind it, /// due at `not_before`. fn parked(attempts: u32, not_before: i64) -> Waiting { - Waiting { fire_id: Uuid::new_v4(), payload: json!({ "v": 1 }), caller: None, attempts, not_before, instance_gap: None } + Waiting { fire_id: Uuid::new_v4(), payload: json!({ "v": 1 }), caller: None, attempts, not_before, instance_gap: None, skipped: false } } /// Seed one signal row for `token`. Entry rows are keyed by @@ -1054,30 +1055,49 @@ async fn an_answer_reaches_its_run_once(pool: PgPool) { let project = Uuid::new_v4(); seed_project(&projects, project, "bin-A").await; seed_parked_signal(&journal, "tok-entry", project).await; - assert!(matches!(journal.answer("tok-entry", &json!(1)).await.unwrap(), Answered::Gone), "an entry is not a wait"); + assert!(matches!(journal.answer("tok-entry", &weft_core::primitive::WaitAnswer::Given { value: json!(1) }).await.unwrap(), Answered::Gone), "an entry is not a wait"); let parked_run = queued_run(&journal, project).await; sqlx::query("UPDATE run SET state = 'parked' WHERE execution_id = $1").bind(parked_run).execute(&pool).await.unwrap(); seed_wait(&journal, "tok-parked", project, parked_run).await; - let Answered::Reached { consumed } = journal.answer("tok-parked", &json!("yes")).await.unwrap() else { panic!("reached") }; + let Answered::Reached { consumed } = journal.answer("tok-parked", &weft_core::primitive::WaitAnswer::Given { value: json!("yes") }).await.unwrap() else { panic!("reached") }; assert_eq!(consumed.token, "tok-parked"); assert!(journal.signal_get("tok-parked").await.unwrap().is_none(), "answered once"); assert_eq!(run_column::(&pool, "state", parked_run).await.as_deref(), Some("queued"), "queued to carry on"); assert!(matches!(journal.events_log(parked_run).await.unwrap().last(), Some(ExecEvent::SuspensionResolved { token, value, .. }) if token == "tok-parked" && value == &json!("yes"))); - assert!(matches!(journal.answer("tok-parked", &json!("again")).await.unwrap(), Answered::Gone)); + assert!(matches!(journal.answer("tok-parked", &weft_core::primitive::WaitAnswer::Given { value: json!("again") }).await.unwrap(), Answered::Gone)); let driven = queued_run(&journal, project).await; claim(&pool, driven, project, "worker-a").await; seed_wait(&journal, "tok-driven", project, driven).await; - assert!(matches!(journal.answer("tok-driven", &json!(2)).await.unwrap(), Answered::Reached { .. })); + assert!(matches!(journal.answer("tok-driven", &weft_core::primitive::WaitAnswer::Given { value: json!(2) }).await.unwrap(), Answered::Reached { .. })); assert_eq!(journal.events_log(driven).await.unwrap().len(), 1, "nothing written into a record its worker writes"); - assert_eq!(weft_task_store::parked_fires::answers_for(&pool, driven).await.unwrap(), vec![("tok-driven".to_string(), json!(2))]); + assert_eq!( + weft_task_store::parked_fires::answers_for(&pool, driven).await.unwrap(), + vec![("tok-driven".to_string(), weft_core::primitive::WaitAnswer::Given { value: json!(2) })] + ); + + // A skip is an answer of its own kind, never a value: written as such + // into a parked run's record, and handed as such to a driven run's + // worker. + let skipped_run = queued_run(&journal, project).await; + sqlx::query("UPDATE run SET state = 'parked' WHERE execution_id = $1").bind(skipped_run).execute(&pool).await.unwrap(); + seed_wait(&journal, "tok-skipped", project, skipped_run).await; + assert!(matches!(journal.answer("tok-skipped", &weft_core::primitive::WaitAnswer::Skipped).await.unwrap(), Answered::Reached { .. })); + assert!(matches!(journal.events_log(skipped_run).await.unwrap().last(), + Some(ExecEvent::SuspensionSkipped { token, .. }) if token == "tok-skipped")); + seed_wait(&journal, "tok-driven-skipped", project, driven).await; + assert!(matches!(journal.answer("tok-driven-skipped", &weft_core::primitive::WaitAnswer::Skipped).await.unwrap(), Answered::Reached { .. })); + assert_eq!( + weft_task_store::parked_fires::answers_for(&pool, driven).await.unwrap().last(), + Some(&("tok-driven-skipped".to_string(), weft_core::primitive::WaitAnswer::Skipped)) + ); let ended = queued_run(&journal, project).await; seed_wait(&journal, "tok-ended", project, ended).await; complete(&journal, ended).await; - assert!(matches!(journal.answer("tok-ended", &json!(3)).await.unwrap(), Answered::RunEnded { .. })); + assert!(matches!(journal.answer("tok-ended", &weft_core::primitive::WaitAnswer::Given { value: json!(3) }).await.unwrap(), Answered::RunEnded { .. })); assert!(journal.signal_get("tok-ended").await.unwrap().is_none()); // An answer queued while the run's trigger was not live is the wait's @@ -1087,7 +1107,7 @@ async fn an_answer_reaches_its_run_once(pool: PgPool) { sqlx::query("UPDATE run SET state = 'parked' WHERE execution_id = $1").bind(queued_for).execute(&pool).await.unwrap(); seed_wait(&journal, "tok-queued", project, queued_for).await; assert_eq!(park(&pool, "tok-queued", &parked(0, 0), None).await.unwrap(), ParkAppend::Parked); - assert!(matches!(journal.answer("tok-queued", &json!("second")).await.unwrap(), Answered::Gone)); + assert!(matches!(journal.answer("tok-queued", &weft_core::primitive::WaitAnswer::Given { value: json!("second") }).await.unwrap(), Answered::Gone)); assert!(journal.signal_get("tok-queued").await.unwrap().is_some(), "the queued answer still has its wait"); } @@ -1530,7 +1550,7 @@ async fn removing_a_projects_executions_frees_every_table_they_touched(pool: PgP .expect("signal"); } let mut tx = pool.begin().await.unwrap(); - weft_task_store::parked_fires::hand_answer_in(&mut tx, "answered-tok", first, &json!(1)).await.expect("a handed answer"); + weft_task_store::parked_fires::hand_answer_in(&mut tx, "answered-tok", first, &weft_core::primitive::WaitAnswer::Given { value: json!(1) }).await.expect("a handed answer"); sqlx::query("INSERT INTO run_search_queue (execution_id) VALUES ($1) ON CONFLICT DO NOTHING") .bind(first) .execute(&mut *tx) diff --git a/crates/weft-dispatcher/tests/db_project_ports.rs b/crates/weft-dispatcher/tests/db_project_ports.rs index a1b2399c..24235b83 100644 --- a/crates/weft-dispatcher/tests/db_project_ports.rs +++ b/crates/weft-dispatcher/tests/db_project_ports.rs @@ -16,7 +16,7 @@ async fn project(projects: &weft_dispatcher::ProjectStore) -> Uuid { let id = Uuid::new_v4(); let definition: ProjectDefinition = serde_json::from_value(json!({ "id": id, "nodes": [], "edges": [] })).unwrap(); projects - .register_with_hashes(definition, "ports-rig", "", "tenant-1", Some("bin-A"), Some("def-1"), None, None, None, None) + .register_with_hashes(definition, "ports-rig", "", "tenant-1", Some("bin-A"), Some("def-1"), None, None, None, None, None) .await .expect("register project"); id diff --git a/crates/weft-dispatcher/tests/db_tags.rs b/crates/weft-dispatcher/tests/db_tags.rs index 050a4649..67dec23d 100644 --- a/crates/weft-dispatcher/tests/db_tags.rs +++ b/crates/weft-dispatcher/tests/db_tags.rs @@ -55,7 +55,7 @@ fn rig_project(id: Uuid) -> ProjectDefinition { async fn seed_project(projects: &weft_dispatcher::ProjectStore, id: Uuid) { projects - .register_with_hashes(rig_project(id), "db-rig", "", TENANT, Some("bin-A"), Some("def-1"), None, None, None, None) + .register_with_hashes(rig_project(id), "db-rig", "", TENANT, Some("bin-A"), Some("def-1"), None, None, None, None, None) .await .expect("register project"); } diff --git a/crates/weft-dispatcher/tests/db_versions.rs b/crates/weft-dispatcher/tests/db_versions.rs index 41764f62..fab2e603 100644 --- a/crates/weft-dispatcher/tests/db_versions.rs +++ b/crates/weft-dispatcher/tests/db_versions.rs @@ -42,7 +42,7 @@ fn rig_project(id: Uuid) -> ProjectDefinition { async fn seed_project(projects: &weft_dispatcher::ProjectStore, id: Uuid) { projects - .register_with_hashes(rig_project(id), "db-rig", "", TENANT, Some("bin-A"), Some("def-1"), None, None, None, None) + .register_with_hashes(rig_project(id), "db-rig", "", TENANT, Some("bin-A"), Some("def-1"), None, None, None, None, None) .await .expect("register project"); } @@ -179,7 +179,7 @@ async fn head_lives_on_the_project_row_and_activations_name_their_versions(pool: assert!(activations.try_begin_activating(project, &keys, setup_execution_id, None).await.unwrap().is_ok()); assert!(activations.record_activation_source(project, setup_execution_id, &activated, "v1").await.unwrap()); activations.end_activating(project, setup_execution_id, &ActivationLifecycle::active(), false, None).await.unwrap().expect("owned"); - projects.register_with_hashes(rig_project(project), "db-rig", "", TENANT, Some("bin-B"), Some("def-2"), None, None, None, None).await.unwrap(); + projects.register_with_hashes(rig_project(project), "db-rig", "", TENANT, Some("bin-B"), Some("def-2"), None, None, None, None, None).await.unwrap(); let listed = activations.list(project).await.unwrap(); assert_eq!(listed[0].program, Some(activated), "a build preserves the code activation used"); let head = versions.head(project).await.unwrap(); diff --git a/crates/weft-dispatcher/tests/tenant_isolation.rs b/crates/weft-dispatcher/tests/tenant_isolation.rs index bed4830b..ba40266a 100644 --- a/crates/weft-dispatcher/tests/tenant_isolation.rs +++ b/crates/weft-dispatcher/tests/tenant_isolation.rs @@ -33,7 +33,7 @@ fn definition(id: Uuid) -> ProjectDefinition { async fn register(store: &FakeProjectStore, id: Uuid, name: &str, tenant: &str) { store - .register_with_hashes(definition(id), name, "", tenant, None, None, None, None, None, None) + .register_with_hashes(definition(id), name, "", tenant, None, None, None, None, None, None, None) .await .expect("register"); } @@ -82,7 +82,7 @@ async fn cross_tenant_project_id_takeover_is_refused() { register(&store, shared_id, "a-owned", TENANT_A).await; let takeover = store - .register_with_hashes(definition(shared_id), "b-grab", "", TENANT_B, None, None, None, None, None, None) + .register_with_hashes(definition(shared_id), "b-grab", "", TENANT_B, None, None, None, None, None, None, None) .await; assert!(takeover.is_err(), "cross-tenant re-register must be refused"); assert_eq!( @@ -93,7 +93,7 @@ async fn cross_tenant_project_id_takeover_is_refused() { // The owner CAN re-register its own project (idempotent update). store - .register_with_hashes(definition(shared_id), "a-owned-v2", "", TENANT_A, None, None, None, None, None, None) + .register_with_hashes(definition(shared_id), "a-owned-v2", "", TENANT_A, None, None, None, None, None, None, None) .await .expect("owner re-register allowed"); } @@ -248,11 +248,11 @@ async fn a_program_is_retired_only_when_no_run_still_names_it() { // Two recorded versions; only one of them ever ran. store - .register_with_hashes(definition(project), "doomed", "", TENANT_A, None, Some("ran"), None, None, None, None) + .register_with_hashes(definition(project), "doomed", "", TENANT_A, None, Some("ran"), None, None, None, None, None) .await .expect("record the version that ran"); store - .register_with_hashes(definition(project), "doomed", "", TENANT_A, None, Some("never-ran"), None, None, None, None) + .register_with_hashes(definition(project), "doomed", "", TENANT_A, None, Some("never-ran"), None, None, None, None, None) .await .expect("record a version nothing ran"); let execution_id = Uuid::new_v4(); diff --git a/crates/weft-e2e/src/human.rs b/crates/weft-e2e/src/human.rs index 0cd73231..b8266eca 100644 --- a/crates/weft-e2e/src/human.rs +++ b/crates/weft-e2e/src/human.rs @@ -8,7 +8,8 @@ //! //! This module is a thin, intent-named layer over [`crate::signal`]: discover a //! form, answer it. The resume itself is asserted through the normal replay -//! (a `node_resumed` carrying the answer, then the node completing). +//! (a `wait_ended` carrying the answer, then the node resuming and +//! completing). use std::time::Duration; @@ -61,3 +62,11 @@ pub async fn answer_form( .context("discovered form has no signal token to submit to")?; signal::fire_token(disp, token, answer).await } + +/// Skip a discovered form, as a person declining the question does. +pub async fn skip_form(disp: &Dispatcher, form: &DiscoveredSignal) -> Result<()> { + let token = form + .token() + .context("discovered form has no signal token to skip")?; + signal::skip_token(disp, token).await +} diff --git a/crates/weft-e2e/src/signal.rs b/crates/weft-e2e/src/signal.rs index a0d4e5a6..cc0f036c 100644 --- a/crates/weft-e2e/src/signal.rs +++ b/crates/weft-e2e/src/signal.rs @@ -141,3 +141,9 @@ impl SignalScope { pub async fn fire_token(disp: &Dispatcher, token: &str, body: &Value) -> Result<()> { disp.post_empty(&format!("/signal/{token}"), body).await } + +/// Skip the waiting step behind `token` (its `/skip` door), as a person +/// declining the question does. +pub async fn skip_token(disp: &Dispatcher, token: &str) -> Result<()> { + disp.post_empty(&format!("/signal/{token}/skip"), &Value::Null).await +} diff --git a/crates/weft-e2e/tests/human_form.rs b/crates/weft-e2e/tests/human_form.rs index 8944eaef..8d86cc5b 100644 --- a/crates/weft-e2e/tests/human_form.rs +++ b/crates/weft-e2e/tests/human_form.rs @@ -43,3 +43,31 @@ async fn human_query_resumes_with_approval() -> anyhow::Result<()> { project.finish().await } + +/// A person skipping the question: the HumanQuery ends skipped (an +/// approve/reject field included), its outputs close, what reads them +/// skips, and the run still completes. +#[tokio::test] +async fn a_skipped_question_skips_its_step_and_the_run_completes() -> anyhow::Result<()> { + let disp = ensure::up().await?; + let mut project = Project::prepare("human_form", disp.clone()).await?; + let pid = project.id(); + let feed = SseFake::start().await?; + project.substitute_in_main("__E2E_FAKE_URL__", &feed.url())?; + project.activate().await?; + let before = run::executions(&disp, &pid).await?; + tokio::time::sleep(Duration::from_secs(2)).await; + feed.push_event("go", &json!({ "value": "the change" }).to_string()); + let execution_id = + run::wait_for_triggered_execution(&disp, &pid, &before, Duration::from_secs(60)).await?; + + let review = human::wait_for_form_by_node(&disp, &pid, "review").await?; + human::skip_form(&disp, &review).await?; + + let settled = SettledRun::observe(&disp, execution_id).await?; + settled.completed()?; + settled.assert_skip_reason("review", "wait_skipped")?; + settled.assert_skipped("out")?; + + project.finish().await +} diff --git a/crates/weft-e2e/tests/instances.rs b/crates/weft-e2e/tests/instances.rs index dfd00874..9edc469d 100644 --- a/crates/weft-e2e/tests/instances.rs +++ b/crates/weft-e2e/tests/instances.rs @@ -297,7 +297,7 @@ out = Debug out.data = svc.status up = Route -> (instance: String) { path: "__E2E_PATH__/up", method: "POST" } -start = StartInstanceInfra { node: "svc" } +start = StartInfra { node: "svc" } start.instance = up.instance upReply = Reply upReply.body = start.done @@ -334,7 +334,7 @@ wipeReply.body = gone.done } }; - // `StartInstanceInfra` answers once the copy runs. Ada's copy is the + // `StartInfra` answers once the copy runs. Ada's copy is the // program's first, so the project's workers move next to it: the run // asking is on the worker being replaced, and its setup lands on the // new one while it waits, so the wait ends instead of holding the diff --git a/crates/weft-engine/src/context.rs b/crates/weft-engine/src/context.rs index 1ca3f2d1..b3e48c0c 100644 --- a/crates/weft-engine/src/context.rs +++ b/crates/weft-engine/src/context.rs @@ -1352,6 +1352,10 @@ pub enum NodeTaskOutcome { /// failure is caught (`weft_core::context::catchable_message`), /// `None` for a failure that never is. Failed { message: String, catchable: Option }, + /// The step ended skipped, for `reason`: what it waited on was + /// skipped by a person (`WeftError::WaitSkipped`). Every output it + /// did not emit closes, as for any skip. + Skipped(weft_core::exec::skip::SkipReason), /// The node called `await_signal` and is now waiting on a fired /// wake signal (carries the suspension token). Waiting(String), @@ -1970,21 +1974,19 @@ impl RunnerHandle { } }; let why = match end { - HeldEnd::Answered(value) => { + HeldEnd::Answered(answer) => { self.record .journal .record_event( - &ExecEvent::SuspensionResolved { - execution_id: self.execution_id, - token, - value: value.clone(), - at_unix: now_unix(), - }, + &ExecEvent::wait_answered(self.execution_id, token, answer.clone(), now_unix()), Some(&self.worker_replica), ) .await .map_err(|e| WeftError::Suspension(format!("recording the answer to a held wait failed: {e:#}")))?; - return Ok(value); + return match answer { + weft_core::primitive::WaitAnswer::Given { value } => Ok(value), + weft_core::primitive::WaitAnswer::Skipped => Err(weft_core::primitive::wait_skipped()), + }; } HeldEnd::Pause => return Err(WeftError::Suspended { token }), HeldEnd::GaveUp(why) => why, @@ -2886,6 +2888,7 @@ impl ContextHandle for RunnerHandle { frames: self.node_frames.clone(), service: service.clone(), origin: resp.owner.clone(), + cancellation: self.cancellation.clone(), }); let client = crate::metering::connection_client( &service, @@ -4311,7 +4314,7 @@ mod replay_tests { while !handle.waits.held().pending() { tokio::task::yield_now().await; } - handle.waits.held().answer("tok-pending", serde_json::json!("approved")); + handle.waits.held().answer("tok-pending", weft_core::primitive::WaitAnswer::Given { value: serde_json::json!("approved") }); assert_eq!(waiting.await.unwrap().expect("the answer"), serde_json::json!("approved")); let events = journal.events.lock().unwrap(); assert!(matches!(events.as_slice(), [ExecEvent::SuspensionResolved { token, value, .. }] diff --git a/crates/weft-engine/src/door/limits.rs b/crates/weft-engine/src/door/limits.rs index 1f408463..b6058e73 100644 --- a/crates/weft-engine/src/door/limits.rs +++ b/crates/weft-engine/src/door/limits.rs @@ -3,12 +3,35 @@ //! at once, and how many refused tokens one address may present. //! //! Every count is kept in this worker's memory, so no call waits on a -//! database. Once a second the worker states its counts of the minute and -//! hears the project's other copies' (`/v1/door/tick`), so with several -//! copies a count is the copies' together, as of the last second: several -//! copies can let through at most about a second's worth of calls more than -//! the limit. One copy is exact. "At once" is held per copy: each takes its -//! share of the limit, the limit divided by the copies alive. +//! database. Once a second while it has work under way, the worker states +//! its counts of the minute and hears the project's other copies' +//! (`/v1/door/tick`, `super::ticks`), so with several copies a count is the +//! copies' together, as of the last second: several copies can let through +//! at most about a second's worth of calls more than the limit. One copy is +//! exact. +//! +//! "At once" is held per copy: each takes its share of the limit, the limit +//! divided by the copies alive (a copy that sleeps lets its lease lapse, and +//! so is not counted). +//! +//! Right after a copy wakes from a sleep, and until its first tick comes +//! back (under a second normally, a few seconds when the broker itself is +//! starting from zero), every call is decided at once on what the copy +//! heard before it slept: +//! - it counts itself the only copy (`Limits::asleep`), so it takes the +//! whole "at once" limit rather than refuse callers on copies that +//! stopped meanwhile; several copies waking at the same moment can each +//! let in up to the full limit; +//! - what it holds of the others' counts is from before the sleep, and +//! counts only go up within a minute, so a per-minute limit can let +//! through the calls that reach this copy in that window, and no more. +//! +//! This overshoot is a deliberate choice, accepted by the maintainer: a +//! call is never made to wait on the broker to be counted. Waiting for a +//! fresh tick after a sleep would put a round trip (and, on a serverless +//! install, the broker's own start) in front of the first call after every +//! pause, on every entry, since every entry has default limits. Do not add +//! that wait back. use std::collections::HashMap; use std::sync::{Arc, Mutex}; @@ -21,6 +44,9 @@ use weft_core::signal::limits::{failed_key, ran_key, window}; /// The counts one copy keeps (see the module doc). pub struct Limits { state: Mutex, + /// Woken whenever this copy's counts change: they are the broker's to + /// hear. + ticking: Arc, } #[derive(Default)] @@ -35,6 +61,11 @@ struct Counts { copies: u32, /// Runs of each entry going on this copy. going: HashMap, + /// How many times `mine` changed, ever, and how many of those changes + /// a tick that landed carried: the counts are unreported while they + /// differ. + changes: u64, + reported: u64, } /// Held for as long as an admitted run goes: it counts toward its entry's @@ -70,7 +101,7 @@ impl Going { pub fn failed(&self, now: i64) { let mut counts = self.limits.state.lock().expect("door limits"); counts.roll(window(now).0); - *counts.mine.entry(failed_key(&self.token)).or_default() += 1; + counts.add(failed_key(&self.token), 1); } /// The admitted work did not become a run (it waits in its trigger's @@ -81,8 +112,8 @@ impl Going { let (minute, keys) = &self.counted; if counts.window_start == *minute { for key in keys { - if let Some(hits) = counts.mine.get_mut(key) { - *hits = hits.saturating_sub(1); + if counts.mine.get(key).is_some_and(|hits| *hits > 0) { + counts.add(key.clone(), -1); } } } @@ -90,19 +121,27 @@ impl Going { } impl Limits { - pub fn new() -> Arc { - Arc::new(Self { state: Mutex::new(Counts { copies: 1, ..Counts::default() }) }) + pub fn new(ticking: Arc) -> Arc { + Arc::new(Self { state: Mutex::new(Counts { copies: 1, ..Counts::default() }), ticking }) } /// Admit one call to the entry `token` by `caller` (the identity its /// gate established, else the instance, else the address, already /// spelled as a key) at `now`: the caller's count, then the entry's, /// then its room. Counted only when admitted; a refusal is counted as - /// one for `weft status`. + /// one for `weft status`. Decided at once on what this copy last heard + /// of the others, never after a wait (see the module doc). pub fn admit(self: &Arc, token: &str, caller: &str, limits: &ResolvedLimits, now: i64) -> Result { + // Admitted or refused, the counts changed. + let admitted = self.admit_counted(token, caller, limits, now); + self.ticking.wake(); + admitted + } + + fn admit_counted(self: &Arc, token: &str, caller: &str, limits: &ResolvedLimits, now: i64) -> Result { let mut counts = self.state.lock().expect("door limits"); let (start, left) = window(now); - counts.roll(start); + let start = counts.roll(start); let checks = [ (limits.per_caller_per_minute, format!("c:{token}:{caller}"), Limited::PerCaller), (limits.per_minute, format!("e:{token}"), Limited::PerEntry), @@ -129,7 +168,7 @@ impl Limits { } } for key in &counted { - *counts.mine.entry(key.clone()).or_default() += 1; + counts.add(key.clone(), 1); } *counts.going.entry(token.to_string()).or_default() += 1; Ok(Going { limits: self.clone(), token: token.to_string(), counted: (start, counted) }) @@ -150,9 +189,12 @@ impl Limits { /// Count one refused token from `address`. pub fn token_refused(&self, address: &str, now: i64) { - let mut counts = self.state.lock().expect("door limits"); - counts.roll(window(now).0); - *counts.mine.entry(invalid_tokens_key(address)).or_default() += 1; + { + let mut counts = self.state.lock().expect("door limits"); + counts.roll(window(now).0); + counts.add(invalid_tokens_key(address), 1); + } + self.ticking.wake(); } /// How many runs of each entry go on this copy right now: what a drain @@ -161,12 +203,36 @@ impl Limits { self.state.lock().expect("door limits").going.iter().map(|(token, going)| (token.clone(), *going)).collect() } - /// This copy's counts of the minute, to state at a tick. - pub fn mine(&self, now: i64) -> (i64, Vec<(String, i64)>) { + /// This copy's counts of the minute, to state at a tick, and the + /// [`Self::reported`] mark they are as of. + pub fn mine(&self, now: i64) -> (i64, Vec<(String, i64)>, u64) { let mut counts = self.state.lock().expect("door limits"); - let start = window(now).0; - counts.roll(start); - (start, counts.mine.iter().map(|(key, hits)| (key.clone(), *hits)).collect()) + let start = counts.roll(window(now).0); + (start, counts.mine.iter().map(|(key, hits)| (key.clone(), *hits)).collect(), counts.changes) + } + + /// A tick that carried the counts as of `mark` (from [`Self::mine`]) + /// landed. + pub fn reported(&self, mark: u64) { + let mut counts = self.state.lock().expect("door limits"); + counts.reported = counts.reported.max(mark); + } + + /// Nothing of this door is under way: no run going, and the broker + /// heard every count. + pub fn settled(&self) -> bool { + let counts = self.state.lock().expect("door limits"); + counts.going.is_empty() && counts.changes == counts.reported + } + + /// The tick loop goes to sleep: from here on this copy hears nothing, + /// and the other copies it last heard of may stop meanwhile, so it + /// counts itself alone until a tick says otherwise. Its first calls + /// after the sleep may then take more than its share of "at once", + /// the same overshoot the module doc settles, and never refuse a call + /// on copies that are gone. + pub fn asleep(&self) { + self.state.lock().expect("door limits").copies = 1; } /// What a tick heard of the minute `window_start`: the other copies' @@ -182,13 +248,18 @@ impl Limits { } impl Counts { - /// Start counting the minute beginning at `start`, when it is a new one. - fn roll(&mut self, start: i64) { - if self.window_start != start { + /// Start counting the minute beginning at `start`, when it is a later + /// one, and return the minute the counts are of. A `now` read before an + /// await (a call's `now` is read before its body) can be of a minute a + /// tick already rolled past; it counts in the current one rather than + /// wiping it. + fn roll(&mut self, start: i64) -> i64 { + if start > self.window_start { self.window_start = start; self.mine.clear(); self.others.clear(); } + self.window_start } fn total(&self, key: &str) -> i64 { @@ -196,7 +267,14 @@ impl Counts { } fn refused(&mut self, token: &str, reason: Limited) { - *self.mine.entry(reason.refusal_key(token)).or_default() += 1; + self.add(reason.refusal_key(token), 1); + } + + /// Add `hits` to this copy's count of `key`: a change the broker has + /// to hear. + fn add(&mut self, key: String, hits: i64) { + *self.mine.entry(key).or_default() += hits; + self.changes += 1; } } @@ -217,7 +295,7 @@ mod tests { #[test] fn a_caller_is_held_to_its_count_and_the_entry_to_its_own() { - let door = Limits::new(); + let door = Limits::new(crate::door::ticks::Ticking::new()); let l = limits(Some(2), Some(3), None); let _a = door.admit("tok", "ip:a", &l, 0).unwrap(); let _b = door.admit("tok", "ip:a", &l, 1).unwrap(); @@ -230,13 +308,13 @@ mod tests { #[test] fn a_refused_call_is_not_counted_against_the_limit_but_is_counted_as_refused() { - let door = Limits::new(); + let door = Limits::new(crate::door::ticks::Ticking::new()); let l = limits(Some(1), None, None); let _a = door.admit("tok", "ip:a", &l, 0).unwrap(); for _ in 0..3 { assert!(door.admit("tok", "ip:a", &l, 0).is_err()); } - let (_, mine) = door.mine(0); + let (_, mine, _) = door.mine(0); let mine: HashMap = mine.into_iter().collect(); assert_eq!(mine["c:tok:ip:a"], 1); assert_eq!(mine["refused:tok:per_caller"], 3); @@ -244,7 +322,7 @@ mod tests { #[test] fn the_other_copies_counts_join_this_ones() { - let door = Limits::new(); + let door = Limits::new(crate::door::ticks::Ticking::new()); let l = limits(Some(5), None, None); door.mine(10); door.heard(0, [("c:tok:ip:a".to_string(), 4)], 2); @@ -255,9 +333,22 @@ mod tests { assert!(door.admit("tok", "ip:b", &l, 11).is_ok()); } + /// Asleep, a copy forgets the copies it last heard of: they may stop + /// meanwhile, and a call after the sleep is never refused on them. + #[test] + fn a_copy_that_slept_counts_itself_alone_until_it_hears_again() { + let door = Limits::new(crate::door::ticks::Ticking::new()); + let l = limits(None, None, Some(4)); + door.heard(0, [], 4); + let _a = door.admit("tok", "ip:a", &l, 0).unwrap(); + assert!(door.admit("tok", "ip:b", &l, 0).is_err(), "a share of 1 among 4 copies"); + door.asleep(); + let _b = door.admit("tok", "ip:b", &l, 0).expect("alone after the sleep: the whole limit"); + } + #[test] fn runs_at_once_are_held_per_copy_and_free_their_room_when_they_end() { - let door = Limits::new(); + let door = Limits::new(crate::door::ticks::Ticking::new()); let l = limits(None, None, Some(4)); door.heard(0, [], 2); let a = door.admit("tok", "ip:a", &l, 0).unwrap(); @@ -271,7 +362,7 @@ mod tests { /// this minute nor holding a room. #[test] fn work_that_did_not_start_is_uncounted() { - let door = Limits::new(); + let door = Limits::new(crate::door::ticks::Ticking::new()); let l = limits(None, Some(1), Some(1)); door.admit("tok", "ip:a", &l, 0).unwrap().uncount(); let _a = door.admit("tok", "ip:a", &l, 1).expect("the uncounted one left the minute's count and its room"); @@ -280,7 +371,7 @@ mod tests { #[test] fn an_address_guessing_tokens_is_refused_for_the_rest_of_the_minute() { - let door = Limits::new(); + let door = Limits::new(crate::door::ticks::Ticking::new()); for _ in 0..3 { door.token_refused("1.2.3.4", 0); } diff --git a/crates/weft-engine/src/door/mod.rs b/crates/weft-engine/src/door/mod.rs index 3d1487e4..b4b78b46 100644 --- a/crates/weft-engine/src/door/mod.rs +++ b/crates/weft-engine/src/door/mod.rs @@ -17,8 +17,9 @@ //! These are the checks the dispatcher ran, with the same strength: the //! routes are the same rows (held in memory, `broker`), the gate is the //! broker's same check, and the limits are counted here and shared with -//! the project's other copies once a second (`limits`). On an open route -//! nothing on the way to the run leaves this process. +//! the project's other copies once a second while the worker has work +//! under way (`limits`, `ticks`). On an open route nothing on the way to +//! the run leaves this process. //! //! A browser cannot put a credential on a socket's opening request, so it //! asks the route with a plain request and is answered a URL carrying a @@ -28,6 +29,7 @@ pub(crate) mod broker; pub(crate) mod limits; pub(crate) mod permits; +pub(crate) mod ticks; use std::collections::BTreeMap; use std::net::IpAddr; @@ -46,6 +48,7 @@ use weft_core::signal::{LiveConnectionConfig, Protocol}; pub use broker::{DoorBroker, HeldDoorBroker, HeldEntry, HeldTrigger, RunFacts, Triggers}; pub use limits::{Going, Limits}; pub use permits::{Busy, RunPermit, RunPermits}; +pub use ticks::WorkUnderWay; /// What the install allows at its public edge, as the worker holds it. #[derive(Debug, Clone, Copy, Default)] @@ -84,6 +87,8 @@ pub struct Door { pub(crate) binary_hash: String, broker: Arc, limits: Arc, + /// When the door ticks (`ticks`). + ticking: Arc, permits: Arc, edge: Edge, /// The project's secret: socket tickets are signed with a key of its @@ -187,12 +192,14 @@ impl Door { secret: weft_core::caller_token::ProjectSecret, permits: Arc, ) -> Arc { + let ticking = ticks::Ticking::new(); Arc::new(Self { project_id, tenant_id, binary_hash, broker, - limits: Limits::new(), + limits: Limits::new(ticking.clone()), + ticking, permits, edge, secret, @@ -560,45 +567,95 @@ impl Door { Ok(claims) } - /// Every second: state this worker's counts, hear the other copies - /// and keep the worker's lease (`/v1/door/tick`). A tick that fails is - /// made again the next second; the limits hold to what was last heard. + /// While the worker has work under way, every second: state this + /// worker's counts, hear the other copies and keep the worker's lease + /// (`/v1/door/tick`); with nothing under way, nothing (`ticks`). A tick + /// that fails is made again the next second; the limits hold to what + /// was last heard. `busy` is what the worker holds besides the door's + /// own runs. /// /// The first tick is made before this returns, so the worker is known - /// alive (its lease, which a run's first broker call is checked - /// against) before it takes its first call: a worker that cannot make - /// it does not start, and says why. - pub async fn start_ticks(self: &Arc) -> anyhow::Result<()> { - self.tick().await.map_err(|e| e.context("the worker's first word to the broker (its lease, and its door's counts)"))?; + /// alive before it takes its first call: a worker that cannot make it + /// does not start, and says why. + pub async fn start_ticks(self: &Arc, busy: WorkUnderWay) -> anyhow::Result<()> { + let quiet = self.tick().await.map_err(|e| e.context("the worker's first word to the broker (its lease, and its door's counts)"))?; let door = Arc::downgrade(self); + let ticking = self.ticking.clone(); tokio::spawn(async move { - let mut every = tokio::time::interval(weft_core::time_scale::scaled(std::time::Duration::from_secs(1))); + let every_second = weft_core::time_scale::scaled(ticks::TICK_EVERY); + let mut every = tokio::time::interval_at(tokio::time::Instant::now() + every_second, every_second); every.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + // The last tick landed and reported nothing in flight. + let mut quiet = quiet; + // The loop slept since its last tick: the next one goes at once. + let mut slept = false; loop { - every.tick().await; - let Some(door) = door.upgrade() else { return }; - if let Err(e) = door.tick().await { - tracing::warn!(target: "weft_engine::door", error = %format!("{e:#}"), "a tick of the door's counts failed; the next second tries again"); + match door.upgrade() { + // The last tick said all there is: sleep until work + // comes, then look again (a wake left while the loop + // was ticking finds nothing new and sleeps on). + Some(strong) if quiet && strong.limits.settled() && !busy() => { + // Asleep, this copy hears nothing of the others, + // whose leases may lapse meanwhile. + strong.limits.asleep(); + // Not held while asleep: the door goes when its + // worker does. + drop(strong); + ticking.sleep().await; + slept = true; + continue; + } + Some(_) => {} + None => return, + } + if std::mem::take(&mut slept) { + every.reset(); + } else { + every.tick().await; } + let Some(door) = door.upgrade() else { return }; + quiet = match door.tick().await { + Ok(quiet) => quiet, + Err(e) => { + tracing::warn!(target: "weft_engine::door", error = %format!("{e:#}"), "a tick of the door's counts failed; the next second tries again"); + false + } + }; } }); Ok(()) } - async fn tick(&self) -> anyhow::Result<()> { + /// One tick: whether it reported nothing in flight. Its counts are + /// marked heard once it lands. + async fn tick(&self) -> anyhow::Result { let now = crate::now_unix() as i64; - let (window_start, mine) = self.limits.mine(now); - let tokens = self.broker.triggers().await?.tokens().cloned().collect(); - let request = DoorTickRequest { - binary_hash: Some(self.binary_hash.clone()), - in_flight: self.limits.going(), - window_start, - counts: mine.into_iter().map(|(key, hits)| DoorCount { key, hits }).collect(), - tokens, - }; - let heard = self.broker.tick(&request).await?; + let (window_start, mine, mark) = self.limits.mine(now); + let in_flight = self.limits.going(); + let quiet = in_flight.is_empty(); + let heard = async { + let tokens = self.broker.triggers().await?.tokens().cloned().collect(); + let request = DoorTickRequest { + binary_hash: Some(self.binary_hash.clone()), + in_flight, + window_start, + counts: mine.into_iter().map(|(key, hits)| DoorCount { key, hits }).collect(), + tokens, + }; + self.broker.tick(&request).await + } + .await?; + self.limits.reported(mark); self.limits.heard(window_start, heard.others.into_iter().map(|count| (count.key, count.hits)), heard.copies); - Ok(()) + Ok(quiet) + } + + /// A run came onto this worker other than through the door (a claim of + /// a queued run), once it is counted where the tick looks: the tick + /// goes out now and renews the lease behind the run, which does not + /// wait for it (`ticks`). + pub(crate) fn wake(&self) { + self.ticking.wake(); } } @@ -673,7 +730,13 @@ pub(crate) mod fake { pub identity: Mutex>, pub verified: Mutex>, pub instance_tokens: Mutex>, + /// Every tick tried, landed or not. pub ticks: Mutex>, + /// How long a tick takes to answer, and whether it fails. + pub tick_takes: Mutex, + pub tick_fails: std::sync::atomic::AtomicBool, + /// What a tick hears: the other copies' counts, and the copies alive. + pub heard: Mutex<(Vec, u32)>, /// The fires parked, in order, with their token. pub parked: Mutex>, } @@ -692,6 +755,9 @@ pub(crate) mod fake { verified: Mutex::new(Vec::new()), instance_tokens: Mutex::new(HashMap::new()), ticks: Mutex::new(Vec::new()), + tick_takes: Mutex::new(std::time::Duration::ZERO), + tick_fails: std::sync::atomic::AtomicBool::new(false), + heard: Mutex::new((Vec::new(), 1)), parked: Mutex::new(Vec::new()), } } @@ -725,7 +791,15 @@ pub(crate) mod fake { } async fn tick(&self, request: &DoorTickRequest) -> anyhow::Result { self.ticks.lock().unwrap().push(request.clone()); - Ok(DoorTick { others: Vec::new(), copies: 1 }) + let takes = *self.tick_takes.lock().unwrap(); + if !takes.is_zero() { + tokio::time::sleep(takes).await; + } + if self.tick_fails.load(std::sync::atomic::Ordering::SeqCst) { + anyhow::bail!("the broker is away"); + } + let (others, copies) = self.heard.lock().unwrap().clone(); + Ok(DoorTick { others, copies }) } async fn park_fire( &self, diff --git a/crates/weft-engine/src/door/ticks.rs b/crates/weft-engine/src/door/ticks.rs new file mode 100644 index 00000000..1a679c7e --- /dev/null +++ b/crates/weft-engine/src/door/ticks.rs @@ -0,0 +1,282 @@ +//! When the door ticks (`/v1/door/tick`, [`super::Door::start_ticks`]): +//! once a second while this worker has something under way, and not at +//! all once it has nothing. A worker with no runs and no callers makes no +//! broker call, so the broker and its database can go to sleep behind it. +//! +//! ```text +//! work arrives ─► wake ─► tick now, then every second +//! nothing under way, and the last tick said so ─► sleep until woken +//! ``` +//! +//! "Under way" is: a run of the door going (`Limits::going`), counts of +//! this minute the broker has not heard yet, or what the worker says it +//! still holds (the runs it drives, the records it still has to write: +//! [`WorkUnderWay`]). The last tick before a sleep is the one that +//! reported nothing in flight and every count, so the broker's picture is +//! the latest one. +//! +//! The lease is what the tick keeps: a run on record as this worker's is +//! read as lost once the lease lapsed and a margin as long again went by +//! (`weft_dispatcher::worker_door`). Every way a run comes onto this worker +//! wakes the tick, and the run goes on at once without waiting for it: the +//! tick it woke renews the lease a round trip later, long before the +//! margin runs out. + +use std::time::Duration; + +/// How often the loop ticks while something is under way (scaled). +pub(super) const TICK_EVERY: Duration = Duration::from_secs(1); + +/// What the worker says it still holds, besides the door's own runs: true +/// while the tick has to go on. +pub type WorkUnderWay = Box bool + Send + Sync>; + +/// The tick's sleep and wake (see the module doc). +pub struct Ticking { + /// Notified on every wake. With nobody waiting it keeps one permit, so + /// a wake that lands just before the loop starts waiting is not lost: + /// the loop finds it and looks again. + woken: tokio::sync::Notify, +} + +impl Ticking { + pub fn new() -> std::sync::Arc { + std::sync::Arc::new(Self { woken: tokio::sync::Notify::new() }) + } + + /// Something came under way: a sleeping tick loop ticks now. Called + /// after the work is counted where the loop looks for it (the door's + /// limits, the worker's [`WorkUnderWay`]), so the loop, woken now or + /// finding the permit this leaves, sees the work. + pub fn wake(&self) { + self.woken.notify_one(); + } + + /// Wait for the next [`Self::wake`], or return at once on a permit one + /// left meanwhile. The loop looks again either way: a permit left + /// while it was ticking is no reason to tick. + pub(super) async fn sleep(&self) { + self.woken.notified().await; + } +} + +#[cfg(test)] +mod tests { + use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::Arc; + + use axum::http::HeaderMap; + use futures::FutureExt as _; + use tokio::time::{sleep, Instant}; + use weft_broker_client::protocol::DoorTickRequest; + + use super::super::fake::*; + use super::super::{Arrived, Call, Door}; + use super::*; + + fn secs(n: f64) -> Duration { + Duration::from_secs_f64(n) + } + + async fn arrive(door: &Door, headers: &HeaderMap) -> Arrived { + let call = Call { method: "GET", raw_path: "/x", raw_query: "", headers, peer: "10.0.0.1".parse().unwrap(), relayed: false, route_prefix: "" }; + door.arrive(call, axum::body::Body::empty()).await + } + + /// A door serving `x`, ticking, with nothing under way but what + /// `busy` says. + async fn ticking(busy: impl Fn() -> bool + Send + Sync + 'static) -> (Arc, Arc) { + let broker = FakeDoorBroker::new(vec![route("t1", "route", "x")]); + let door = door(broker.clone()); + door.start_ticks(Box::new(busy)).await.unwrap(); + (door, broker) + } + + fn ticks(broker: &FakeDoorBroker) -> Vec { + broker.ticks.lock().unwrap().clone() + } + + fn the_minute() -> i64 { + weft_core::signal::limits::window(crate::now_unix() as i64).0 + } + + /// A call keeps the door ticking every second while its run goes; once + /// it ends, one tick says so with the latest counts, and then nothing. + #[tokio::test(start_paused = true)] + async fn the_door_ticks_while_a_run_goes_then_once_more_then_never() { + let (door, broker) = ticking(|| false).await; + sleep(secs(10.0)).await; + assert_eq!(ticks(&broker).len(), 1, "only the first tick, at startup: nothing came since"); + + let minute = the_minute(); + let Arrived::Run(admitted, _) = arrive(&door, &HeaderMap::new()).await else { panic!("admitted") }; + sleep(secs(3.5)).await; + let busy = ticks(&broker); + assert!(busy.len() >= 5, "the call is decided on what the copy last heard, then its tick goes out at once, then one a second: {}", busy.len()); + assert!(busy[2..].iter().all(|tick| tick.in_flight.get("t1") == Some(&1)), "every tick after the admission says the run is in flight"); + + drop(admitted); + sleep(secs(3.0)).await; + let after = ticks(&broker); + assert_eq!(after.len(), busy.len() + 1, "one last tick once the run ended"); + let last = after.last().unwrap(); + assert!(last.in_flight.is_empty(), "the last tick says nothing is in flight"); + if last.window_start == minute { + let ran = weft_core::signal::limits::ran_key("t1"); + assert!(last.counts.iter().any(|count| count.key == ran && count.hits == 1), "the last tick carries the minute's counts"); + } + sleep(secs(30.0)).await; + assert_eq!(ticks(&broker).len(), after.len(), "silence once idle"); + } + + /// After a long sleep the lease has lapsed: a run that comes then goes + /// on at once, and the tick it woke goes out at once and renews the + /// lease behind it, however slow the broker is to answer. + #[tokio::test(start_paused = true)] + async fn a_run_after_a_long_sleep_does_not_wait_and_its_tick_goes_out() { + let holds = Arc::new(AtomicBool::new(false)); + let (door, broker) = ticking({ + let holds = holds.clone(); + move || holds.load(Ordering::SeqCst) + }) + .await; + sleep(secs(20.0)).await; + assert_eq!(ticks(&broker).len(), 1, "asleep since the first tick"); + *broker.tick_takes.lock().unwrap() = secs(2.0); + holds.store(true, Ordering::SeqCst); + let asked = Instant::now(); + door.wake(); + assert_eq!(asked.elapsed(), Duration::ZERO, "the run waited for nothing"); + sleep(secs(0.5)).await; + assert_eq!(ticks(&broker).len(), 2, "the tick it woke went out at once"); + } + + /// The broker being away when the worker wakes refuses nothing: the + /// call is let in, and the tick tries again each second. + #[tokio::test(start_paused = true)] + async fn a_lapsed_lease_and_a_failing_tick_refuse_no_call() { + let (door, broker) = ticking(|| false).await; + sleep(secs(20.0)).await; + broker.tick_fails.store(true, Ordering::SeqCst); + let Arrived::Run(_admitted, _) = arrive(&door, &HeaderMap::new()).await else { panic!("admitted") }; + sleep(secs(3.5)).await; + assert!(ticks(&broker).len() >= 4, "a failed tick is tried again each second"); + } + + /// A wake left while the loop was ticking does not keep it awake: the + /// door still goes quiet after one last tick. + #[tokio::test(start_paused = true)] + async fn a_wake_while_ticking_leaves_no_extra_tick() { + let holds = Arc::new(AtomicBool::new(true)); + let (door, broker) = ticking({ + let holds = holds.clone(); + move || holds.load(Ordering::SeqCst) + }) + .await; + sleep(secs(2.5)).await; + door.wake(); + door.wake(); + holds.store(false, Ordering::SeqCst); + let before = ticks(&broker).len(); + sleep(secs(30.0)).await; + assert_eq!(ticks(&broker).len(), before + 1, "one last tick, then silence"); + } + + /// What the worker holds (a run it claimed, a record on its way) keeps + /// the door ticking as long as it does. + #[tokio::test(start_paused = true)] + async fn the_door_ticks_while_the_worker_holds_work() { + let holds = Arc::new(AtomicBool::new(false)); + let (door, broker) = ticking({ + let holds = holds.clone(); + move || holds.load(Ordering::SeqCst) + }) + .await; + sleep(secs(5.0)).await; + holds.store(true, Ordering::SeqCst); + door.wake(); + sleep(secs(3.5)).await; + let held = ticks(&broker).len(); + assert!(held >= 5, "a tick at once, then one a second: {held}"); + holds.store(false, Ordering::SeqCst); + sleep(secs(1.5)).await; + let after = ticks(&broker).len(); + sleep(secs(30.0)).await; + assert_eq!(ticks(&broker).len(), after, "silence once it holds nothing"); + } + + /// A refusal is a count too: it wakes the tick, is reported once, and + /// the door goes quiet again. + #[tokio::test(start_paused = true)] + async fn a_refusal_while_idle_is_reported_once() { + let (door, broker) = ticking(|| false).await; + sleep(secs(10.0)).await; + let minute = the_minute(); + let mut headers = HeaderMap::new(); + headers.insert(weft_core::instance::INSTANCE_TOKEN_HEADER, "nope".parse().unwrap()); + assert!(matches!(arrive(&door, &headers).await, Arrived::Answer(_)), "refused"); + sleep(secs(30.0)).await; + let ticks = ticks(&broker); + assert_eq!(ticks.len(), 2, "one tick for the refusal, then silence"); + if ticks[1].window_start == minute { + assert!(ticks[1].counts.iter().any(|count| count.key.starts_with("t:-:")), "the refused token is counted"); + } + } + + /// A call after a sleep is decided at once on what this copy last + /// heard, never after a wait (the overshoot the limits module doc + /// settles): another copy ran the entry's one run of the minute + /// meanwhile, so the first call goes over by one, and the tick it woke + /// brings that count in for the next call. + #[tokio::test(start_paused = true)] + async fn a_call_after_a_sleep_is_decided_at_once_and_its_tick_corrects_the_next() { + let mut trigger = route("t1", "route", "x"); + armed(&mut trigger).spec.limits = weft_core::signal::EntryLimits { per_minute: Some(1), ..Default::default() }; + let broker = FakeDoorBroker::new(vec![trigger]); + let door = door(broker.clone()); + door.start_ticks(Box::new(|| false)).await.unwrap(); + sleep(secs(20.0)).await; + assert_eq!(ticks(&broker).len(), 1, "asleep since the first tick"); + + // Meanwhile another copy ran the minute's one run. + let minute = the_minute(); + let ran = weft_broker_client::protocol::DoorCount { key: "e:t1".into(), hits: 1 }; + *broker.heard.lock().unwrap() = (vec![ran], 2); + *broker.tick_takes.lock().unwrap() = secs(2.0); + let asked = Instant::now(); + let Arrived::Run(_admitted, _) = arrive(&door, &HeaderMap::new()).await else { panic!("admitted on what it last heard") }; + assert_eq!(asked.elapsed(), Duration::ZERO, "the call waited for nothing"); + sleep(secs(3.0)).await; + if the_minute() != minute { + return; // crossed into a new minute, which counts from nothing + } + assert!(matches!(arrive(&door, &HeaderMap::new()).await, Arrived::Answer(_)), "the next call is refused on the count its tick heard"); + } + + /// While the door ticks every second, a call is decided on what it heard + /// last and waits for nothing, however slow the broker is right now. + #[tokio::test(start_paused = true)] + async fn a_call_while_ticking_waits_for_no_tick() { + let holds = Arc::new(AtomicBool::new(true)); + let (door, broker) = ticking({ + let holds = holds.clone(); + move || holds.load(Ordering::SeqCst) + }) + .await; + sleep(secs(3.5)).await; + *broker.tick_takes.lock().unwrap() = secs(2.0); + let asked = Instant::now(); + let Arrived::Run(_admitted, _) = arrive(&door, &HeaderMap::new()).await else { panic!("admitted") }; + assert_eq!(asked.elapsed(), Duration::ZERO, "no wait while ticking"); + } + + /// A wake that comes before the loop starts waiting is kept for it: the + /// sleep that follows returns at once. + #[tokio::test] + async fn a_wake_before_the_sleep_is_not_lost() { + let ticking = Ticking::new(); + ticking.wake(); + assert!(ticking.sleep().now_or_never().is_some(), "the wake left for the sleeper"); + assert!(ticking.sleep().now_or_never().is_none(), "one wake, one return"); + } +} diff --git a/crates/weft-engine/src/execution_driver.rs b/crates/weft-engine/src/execution_driver.rs index a88e2eb5..f2b472fc 100644 --- a/crates/weft-engine/src/execution_driver.rs +++ b/crates/weft-engine/src/execution_driver.rs @@ -299,9 +299,14 @@ pub(crate) async fn run_one_execution_observed( bus_coordinator.shutdown().await; drop(bus_coordinator); // Its spend: a charge still open is a call that spent and whose - // amount no response stated, booked as the unknown it is. - clients.open_charges.flush_execution_id(execution_id, "the run ended before the job was read back"); - record.costs.wait_zero().await; + // amount no response stated, booked as the unknown it is, and every + // figure still being worked out lands before the ending. A call whose + // answer is still being read holds this wait, and the metering tap + // bounds it: it cuts an answer silent for its limit, or any answer once + // the run is cancelled (`crate::metering::ANSWER_SILENCE_LIMIT`). + clients.open_charges + .close_execution_id(execution_id, &record.costs, "the run ended before the job was read back") + .await; // Its caller's last words. A durable run's whole answer that waits // for its record leaves once the ending is on it too: one write holds // both. @@ -512,7 +517,7 @@ async fn take_answers( let at_unix = now_unix(); let resolved: Vec = answers .into_iter() - .map(|answer| ExecEvent::SuspensionResolved { execution_id, token: answer.token, value: answer.value, at_unix }) + .map(|answer| ExecEvent::wait_answered(execution_id, answer.token, answer.answer, at_unix)) .collect(); JournalClient::record_events(journal, &resolved, Some(replica)).await?; journal.flush().await?; @@ -1522,7 +1527,7 @@ async fn drive( .held() .early_for(&suspended_tokens(executions)) .into_iter() - .map(|(token, value)| weft_broker_client::protocol::RunAnswer { token, value }) + .map(|(token, answer)| weft_broker_client::protocol::RunAnswer { token, answer }) .collect(); resume_answered( drive_journal, fold, execution_id, replica, early, executions, pulses, kicked, &mut awaited_sequences, @@ -2435,6 +2440,9 @@ async fn drive( Err(weft_core::error::WeftError::Suspended { token }) => { NodeTaskOutcome::Waiting(token) } + Err(weft_core::error::WeftError::WaitSkipped(_)) => { + NodeTaskOutcome::Skipped(weft_core::exec::skip::SkipReason::WaitSkipped) + } Err(e) => NodeTaskOutcome::Failed { message: runner_for_bake.with_infra_hint(format!("{e}")), catchable: weft_core::context::catchable_message(&e), @@ -2742,7 +2750,7 @@ async fn drive( let suspended_on = suspended_tokens(executions); let (suspended, held): (Vec<_>, Vec<_>) = answers.into_iter().partition(|answer| suspended_on.contains(answer.token.as_str())); for answer in held { - waits.held().answer(&answer.token, answer.value); + waits.held().answer(&answer.token, answer.answer); } resume_answered( drive_journal, fold, execution_id, replica, suspended, executions, pulses, kicked, &mut awaited_sequences, @@ -3718,7 +3726,7 @@ async fn handle_node_skip( journal: &dyn JournalClient, replica: &str, ) { - mark_skipped(executions, node_id, execution_id, frames); + mark_skipped(executions, node_id, execution_id, frames, reason); // A node inside a gated scope carries NO closures: every member of // that scope gets its own `ScopeSkipped` row from the scope's // sweep (nested scopes included), and the scope's outward closures @@ -3842,10 +3850,12 @@ async fn cancel_cleanup( dispatchable: Option<&std::collections::HashSet>, ) { // An in-flight node's future is aborted immediately; it needs no window - // to wrap up first. A paid call's cost is measured by the metering tap - // BELOW the node's future, which finalizes on drop and resolves the - // figure detached (tracked by the process's pending-cost records), so an - // aborted node body never loses money bookkeeping. + // to wrap up first. A paid call's answer may be read on a task of a + // client library's own, which this abort does not reach: the metering + // tap watches the run's cancel flag, already tripped, and cuts such a + // call itself, booking it and releasing the run's ending + // (`crate::metering`). A call whose answer lives in the node's future + // is booked by the tap's drop. // 1. Stop every body. bodies.shutdown().await; // 2. Drain the task channel in one FIFO pass (see the doc above). @@ -5031,6 +5041,24 @@ async fn apply_one_task_msg( ) .await; } + NodeTaskOutcome::Skipped(reason) => { + retire_consumer_streams(project, &loc, task_execution_id, pulses, journal, replica, stream_rt) + .await; + // Same stale-terminal guard as the Completed arm. + if firing_already_terminal(executions, &loc.node_id, task_execution_id, &loc.frames) { + return; + } + // A body that ended skipped closes what it never emitted + // on, as a completed one does; the NodeSkipped row is the + // fact the fold sweeps the same ports from. + mark_skipped(executions, &loc.node_id, task_execution_id, &loc.frames, &reason); + let mentioned = mentioned_ports(executions, &loc.node_id, task_execution_id, &loc.frames); + build_unmentioned_closures( + &loc.node_id, &mentioned, task_execution_id, &loc.frames, + project, program_idx, pulses, executions, reason.inherited_failure(), + ); + ship_node_skipped(journal, replica, task_execution_id, &loc.node_id, &loc.frames, &reason).await; + } NodeTaskOutcome::Waiting(token) => { mark_waiting(executions, &loc.node_id, task_execution_id, &loc.frames, &token); ship_node_suspended(journal, replica, task_execution_id, &loc.node_id, &loc.frames, &token) @@ -5083,7 +5111,7 @@ async fn drain_task_msgs_for_cancel( .await; } TaskMsg::Terminal { loc, execution_id: task_execution_id, outcome } => match outcome { - NodeTaskOutcome::Completed { .. } | NodeTaskOutcome::Failed { .. } => { + NodeTaskOutcome::Completed { .. } | NodeTaskOutcome::Failed { .. } | NodeTaskOutcome::Skipped(_) => { tracing::debug!( target: "weft_engine::execution_driver", execution_id = %task_execution_id, node = %loc.node_id, frames = ?loc.frames, @@ -5230,15 +5258,19 @@ fn mark_skipped( node_id: &str, execution_id: ExecutionId, frames: &weft_core::frames::LoopFrames, + reason: &weft_core::exec::skip::SkipReason, ) { if let Some(e) = latest_firing_mut(executions, node_id, execution_id, frames) { e.status = NodeExecutionStatus::Skipped; e.completed_at = Some(now_unix()); - // A skip is the firing's whole outcome: the body never ran, so - // a port's refusal found at readiness is not a fact about it. - // The `NodeSkipped` row carries no error, and the fold reads - // none; the live record says the same. + // A skip is the firing's whole outcome (decided before the body + // ran, or the body's wait skipped by a person), so a port's + // refusal found at readiness is not a fact about it. The + // `NodeSkipped` row carries no error and the reason, and the fold + // reads them so; the live record says the same. e.error = None; + e.skip_reason = Some(reason.clone()); + e.callback_id = None; } } diff --git a/crates/weft-engine/src/execution_driver_tests/rig.rs b/crates/weft-engine/src/execution_driver_tests/rig.rs index 620c41d5..575a28f1 100644 --- a/crates/weft-engine/src/execution_driver_tests/rig.rs +++ b/crates/weft-engine/src/execution_driver_tests/rig.rs @@ -225,7 +225,10 @@ /// An answer to the wait `token`, as the install parks one for the /// run's worker to take. pub(crate) fn answer(&self, token: impl Into, value: Value) { - self.waiting.lock().unwrap().push(weft_broker_client::protocol::RunAnswer { token: token.into(), value }); + self.waiting.lock().unwrap().push(weft_broker_client::protocol::RunAnswer { + token: token.into(), + answer: weft_core::primitive::WaitAnswer::Given { value }, + }); self.came.notify_waiters(); } } diff --git a/crates/weft-engine/src/held_waits.rs b/crates/weft-engine/src/held_waits.rs index 4930e27a..79873b81 100644 --- a/crates/weft-engine/src/held_waits.rs +++ b/crates/weft-engine/src/held_waits.rs @@ -24,7 +24,7 @@ use std::collections::{HashMap, HashSet}; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Arc, Mutex, MutexGuard, PoisonError, Weak}; -use serde_json::Value; +use weft_core::primitive::WaitAnswer; use tokio::sync::Notify; use weft_core::liveness::{wait_on, FiringLocation, WaitLiveness, WaitSource}; @@ -87,7 +87,8 @@ pub(crate) fn gave_up_because(why: Unsuspendable, hold_secs: u32) -> String { /// How a held wait ended. #[derive(Debug, Clone, PartialEq)] pub(crate) enum HeldEnd { - Answered(Value), + /// Its answer: a value, or a person skipping it. + Answered(WaitAnswer), /// Given up, and why. GaveUp(String), /// The run can pause now: the wait suspends. @@ -112,7 +113,7 @@ struct HeldState { /// on it and is about to read as suspended (`Self::early_for` hands /// those to the drive's resume). Kept by token until one of those /// takes it. - early: HashMap, + early: HashMap, /// Waits given up: an answer that comes for one after is dropped. given_up: HashSet, /// Monotone generation of endings, for [`WaitSource`]. @@ -160,7 +161,7 @@ impl HeldWaits { /// The answer to the wait `token`, taken by the drive: to the call /// holding it, else kept for whoever takes it next (see `early`). One /// for a wait given up is dropped. - pub(crate) fn answer(&self, token: &str, value: Value) { + pub(crate) fn answer(&self, token: &str, value: WaitAnswer) { let mut state = self.lock(); if state.given_up.contains(token) { return; @@ -182,7 +183,7 @@ impl HeldWaits { /// The answers kept for waits that turned out to be suspended steps /// (their tokens in `suspended`), for the drive to resume them with. - pub(crate) fn early_for(&self, suspended: &HashSet<&str>) -> Vec<(String, Value)> { + pub(crate) fn early_for(&self, suspended: &HashSet<&str>) -> Vec<(String, WaitAnswer)> { let mut state = self.lock(); let tokens: Vec = state.early.keys().filter(|token| suspended.contains(token.as_str())).cloned().collect(); tokens.into_iter().filter_map(|token| state.early.remove(&token).map(|value| (token, value))).collect() @@ -267,6 +268,10 @@ impl WaitSource for HeldWaits { mod tests { use super::*; + fn given(value: serde_json::Value) -> WaitAnswer { + WaitAnswer::Given { value } + } + fn here() -> FiringLocation { FiringLocation::new("form", Vec::new()) } @@ -282,8 +287,8 @@ mod tests { async fn an_answer_ends_its_wait_even_when_it_came_first() { let (_tracker, weak) = liveness(); let held = HeldWaits::new(); - held.answer("early", serde_json::json!(1)); - assert_eq!(held.hold(&weak, here(), "early").await, HeldEnd::Answered(serde_json::json!(1))); + held.answer("early", given(serde_json::json!(1))); + assert_eq!(held.hold(&weak, here(), "early").await, HeldEnd::Answered(given(serde_json::json!(1)))); let waiting = tokio::spawn({ let held = held.clone(); async move { held.hold(&weak, here(), "late").await } @@ -291,8 +296,8 @@ mod tests { while !held.pending() { tokio::task::yield_now().await; } - held.answer("late", serde_json::json!(2)); - assert_eq!(waiting.await.unwrap(), HeldEnd::Answered(serde_json::json!(2))); + held.answer("late", given(serde_json::json!(2))); + assert_eq!(waiting.await.unwrap(), HeldEnd::Answered(given(serde_json::json!(2)))); assert!(!held.pending()); } @@ -310,7 +315,7 @@ mod tests { tokio::task::yield_now().await; } assert_eq!(held.give_up("nothing moved"), 1); - held.answer("t", serde_json::json!("late")); + held.answer("t", given(serde_json::json!("late"))); assert_eq!(waiting.await.unwrap(), HeldEnd::GaveUp("nothing moved".into())); { let state = held.lock(); @@ -325,12 +330,12 @@ mod tests { #[test] fn an_early_answer_waits_for_its_step() { let held = HeldWaits::new(); - held.answer("suspends", serde_json::json!(1)); - held.answer("zero", serde_json::json!(2)); + held.answer("suspends", given(serde_json::json!(1))); + held.answer("zero", given(serde_json::json!(2))); held.gave_up("zero"); assert!(held.early_for(&HashSet::from(["other"])).is_empty()); - assert_eq!(held.early_for(&HashSet::from(["suspends", "zero"])), vec![("suspends".to_string(), serde_json::json!(1))]); - held.answer("zero", serde_json::json!(3)); + assert_eq!(held.early_for(&HashSet::from(["suspends", "zero"])), vec![("suspends".to_string(), given(serde_json::json!(1)))]); + held.answer("zero", given(serde_json::json!(3))); assert!(held.lock().early.is_empty(), "a wait given up takes no answer"); } @@ -348,9 +353,9 @@ mod tests { tokio::task::yield_now().await; } held.pause(); - held.answer("t", serde_json::json!("yes")); + held.answer("t", given(serde_json::json!("yes"))); assert_eq!(waiting.await.unwrap(), HeldEnd::Pause); - assert_eq!(held.early_for(&HashSet::from(["t"])), vec![("t".to_string(), serde_json::json!("yes"))]); + assert_eq!(held.early_for(&HashSet::from(["t"])), vec![("t".to_string(), given(serde_json::json!("yes")))]); } /// A held wait is parked to the run's tracker: a run whose only step diff --git a/crates/weft-engine/src/journal_writer.rs b/crates/weft-engine/src/journal_writer.rs index 4c16730b..ee0317d6 100644 --- a/crates/weft-engine/src/journal_writer.rs +++ b/crates/weft-engine/src/journal_writer.rs @@ -232,17 +232,21 @@ impl WorkerJournal { let idle = lane.idle.notified(); tokio::pin!(idle); idle.as_mut().enable(); - { - let queue = lane.queue.lock().expect("record writer"); - if queue.runs.is_empty() && !queue.in_flight { - break; - } + if lane.queue.lock().expect("record writer").idle() { + break; } idle.await; } } } + /// Whether every event handed so far, of every run, was written or + /// failed: until then a run's record may still be on its way, and the + /// worker keeps its lease (`crate::door::ticks`). + pub fn settled(&self) -> bool { + self.lanes.iter().all(|lane| lane.queue.lock().expect("record writer").idle()) + } + /// End `execution_id`, a run this worker drives whose record it can no /// longer write (a write of it failed, see [`DriveJournal`], "Poison"): /// the broker writes its ending, failed with `why`, after the last row @@ -448,6 +452,13 @@ struct LaneQueue { closed: bool, } +impl LaneQueue { + /// Nothing queued, nothing on its way. + fn idle(&self) -> bool { + self.runs.is_empty() && !self.in_flight + } +} + /// One run's side of the writer. Locks are taken in the order: its lane's /// queue, its state, its poison. struct RunLog { diff --git a/crates/weft-engine/src/metering.rs b/crates/weft-engine/src/metering.rs index f51ea988..7e1e6830 100644 --- a/crates/weft-engine/src/metering.rs +++ b/crates/weft-engine/src/metering.rs @@ -33,12 +33,29 @@ //! The resolve + record run detached from the node's future (the call may //! be cut by a cancel), tracked per run by [`PendingCostRecords`]: a run //! waits for its spend to be written down before its ending is, so the -//! figure is on its record, before its ending. +//! figure is on its record, before its ending. The token that keeps the +//! ending waiting is taken when the call goes out, not when its response +//! ends: a response body can outlive the node that asked for it (a client +//! library reading the stream on a task of its own finishes the body after +//! the node already returned), and a token taken only then would arrive +//! after the run had ended. +//! +//! That same library task is why aborting the node does not end a call: +//! the body lives on the library's task, not in the node's future. So the +//! tap itself stops reading a call's answer when the provider sends +//! nothing for [`ANSWER_SILENCE_LIMIT`] while it is waited on, or when the +//! call's run is cancelled (the run's own cancel flag, on the +//! [`CostSink`]). The reader then gets an error, which ends the library's +//! task; the call is booked as cut, naming why; and its token goes, so the +//! run's ending never waits on a provider that stays connected and silent. +//! A call the run's cancel catches before its answer even started is let +//! go the same way. use std::pin::Pin; use std::sync::atomic::Ordering; use std::sync::Arc; use std::task::{Context, Poll}; +use std::time::Duration; use bytes::Bytes; use futures::stream::BoxStream; @@ -53,10 +70,12 @@ use weft_providers::{ // ---------- Pending-record tracking ---------- -/// Counts one run's cost resolutions still in flight, so the run's ending -/// waits while a call's money is still being written down. Incremented -/// when a metered response ends (the resolve task is spawned), decremented -/// when its figure is on the run's record (or loudly failed). +/// Counts one run's spend not yet on its record, so the run's ending +/// waits while a call's money is still being written down. A token is +/// taken when a metered call goes out and travels with it: the response's +/// tap carries it, then the open charge or the figure being written, and +/// it is released once the figure is on the run's record (or loudly +/// failed). pub struct PendingCostRecords { inner: Arc, } @@ -71,12 +90,10 @@ impl PendingCostRecords { self.inner.count() } - fn begin(&self) { - self.inner.begin(); - } - - fn end(&self) { - self.inner.end(); + /// Take a token for spend not on record yet; it is released when it + /// drops. + pub(crate) fn hold(&self) -> weft_core::in_flight::InFlightToken { + self.inner.token() } /// Resolve once no cost records are in flight. @@ -84,8 +101,8 @@ impl PendingCostRecords { /// Every resolve is internally bounded (the follow-up client has a /// request timeout, the ledger poll a fixed budget). An OPEN charge /// holds a token until its run closes it - /// (`OpenCharges::flush_execution_id`, which a run ending calls before - /// it waits here); a report being read holds one of its own until the + /// (`OpenCharges::close_execution_id`, which a run ending calls, and + /// which waits here); a report being read holds one of its own until the /// read returns (bounded the same way); and each figure being written /// holds one until it is on record. pub async fn wait_zero(&self) { @@ -141,13 +158,31 @@ struct OpenCharge { /// call that opened it, so the sink that books it is carried with it /// rather than looked up later. sink: Arc, + /// The billable call's own pending token, carried on until the charge + /// is booked, so its run's ending waits for it. + held: weft_core::in_flight::InFlightToken, } /// One owner per charge, with reports queued in arrival order. A closing /// execution drains received reports before declaring the amount unknown. pub struct OpenCharges { next_token: std::sync::atomic::AtomicU64, - by_id: std::sync::Mutex>, + book: std::sync::Mutex, +} + +/// The charges held open, and the runs whose ending already closed theirs. +/// One lock over both, so a charge cannot slip in between a run's close +/// and the mark that refuses it. +#[derive(Default)] +struct Book { + by_id: std::collections::HashMap, + /// Runs being ended (`OpenCharges::close_execution_id`). A response + /// body can finish after its run's charges were closed (a client + /// library reading the stream on a task of its own), and a charge it + /// opened then would hold the run's token with nothing left to close + /// it, so the ending would wait forever. Such a charge is booked as + /// unknown at once instead. + ending: std::collections::HashSet, } struct ChargeState { @@ -169,23 +204,32 @@ impl OpenCharges { pub fn new() -> Arc { Arc::new(Self { next_token: std::sync::atomic::AtomicU64::new(0), - by_id: std::sync::Mutex::new(std::collections::HashMap::new()), + book: std::sync::Mutex::new(Book::default()), }) } - fn by_id(&self) -> std::sync::MutexGuard<'_, std::collections::HashMap> { - self.by_id.lock().unwrap_or_else(|poisoned| poisoned.into_inner()) + fn book(&self) -> std::sync::MutexGuard<'_, Book> { + self.book.lock().unwrap_or_else(|poisoned| poisoned.into_inner()) } - pub fn count(&self) -> usize { self.by_id().len() } + pub fn count(&self) -> usize { self.book().by_id.len() } fn open(&self, service: &'static str, id: String, mut charge: OpenCharge) { let id = ChargeKey { service, id }; charge.token = self.next_token.fetch_add(1, Ordering::SeqCst); - charge.sink.pending.begin(); - let replaced = self.by_id().insert(id.clone(), ChargeState { - charge, reports: Default::default(), processing: false, closing: None, - }); + let replaced = { + let mut book = self.book(); + if book.ending.contains(&charge.sink.execution_id) { + drop(book); + tracing::warn!(target: "weft_engine::metering", charge = %id, + "a call opened a charge after its run began ending; booking it as unknown"); + book_open_charge(charge, "the call's response finished after its run began ending"); + return; + } + book.by_id.insert(id.clone(), ChargeState { + charge, reports: Default::default(), processing: false, closing: None, + }) + }; if let Some(replaced) = replaced { tracing::error!(target: "weft_engine::metering", charge = %id, unread = replaced.reports.len(), "two billable calls claimed the same charge id; booking the displaced call as unknown \ @@ -207,8 +251,8 @@ impl OpenCharges { let id = ChargeKey { service: meter.service(), id: reported.to_string() }; let (done, received) = tokio::sync::oneshot::channel(); let (known, start) = { - let mut charges = self.by_id(); - match charges.get_mut(&id) { + let mut book = self.book(); + match book.by_id.get_mut(&id) { None => (false, None), Some(state) => { state.reports.push_back(ChargeReport { @@ -244,9 +288,9 @@ impl OpenCharges { } Err(_) => { let held = { - let mut charges = self.by_id(); - if charges.get(&id).is_some_and(|state| state.charge.token == token) { - charges.remove(&id) + let mut book = self.book(); + if book.by_id.get(&id).is_some_and(|state| state.charge.token == token) { + book.by_id.remove(&id) } else { None } }; if let Some(held) = held { @@ -261,7 +305,8 @@ impl OpenCharges { async fn drain_reports(&self, id: ChargeKey, token: u64) { loop { let next = { - let mut charges = self.by_id(); + let mut book = self.book(); + let charges = &mut book.by_id; let Some(state) = charges.get_mut(&id).filter(|state| state.charge.token == token) else { return }; match state.reports.pop_front() { Some(report) => Ok((report, state.charge.scratch.clone(), state.charge.sink.clone())), @@ -283,13 +328,14 @@ impl OpenCharges { // released under it (a second call displacing the charge books it // as unknown), and a figure this read still produces must not // land after `wait_zero` let the process exit. - sink.pending.begin(); + let reading = sink.pending.hold(); let result = std::panic::AssertUnwindSafe(report.meter.fold_report( &report.path, report.observed, &mut scratch, report.follow_up.follow_up(report.meter.base_url()), )).catch_unwind().await; let completed = { - let mut charges = self.by_id(); + let mut book = self.book(); + let charges = &mut book.by_id; match charges.get_mut(&id).filter(|state| state.charge.token == token) { Some(state) => { state.charge.scratch = scratch; @@ -316,13 +362,12 @@ impl OpenCharges { tracing::warn!(target: "weft_engine::metering", charge = %id, "the amount for this charge arrived after a second call displaced it; it is \ on the trail twice, once as unknown and once with the figure"); - sink.clone().book_resolved(cost); + sink.clone().book_resolved(reading, cost); } Ok(None) => {} Err(_) => tracing::error!(target: "weft_engine::metering", charge = %id, "the meter panicked while reading a cost report for a charge a second call had displaced"), } - sink.pending.end(); let _ = report.done.send(()); return; } @@ -330,11 +375,7 @@ impl OpenCharges { }; if let Some((charge, result, unread)) = completed { match result { - Ok(Some(cost)) => { - let sink = charge.sink; - sink.clone().book_resolved(cost); - sink.pending.end(); - } + Ok(Some(cost)) => charge.sink.book_resolved(charge.held, cost), Err(_) => { tracing::error!(target: "weft_engine::metering", charge = %id, "the meter panicked while reading a cost report"); @@ -355,19 +396,32 @@ impl OpenCharges { tracing::warn!(target: "weft_engine::metering", charge = %id, unread, "reports received after the one that priced this charge were not read"); } - sink.pending.end(); + drop(reading); let _ = report.done.send(()); return; } - sink.pending.end(); + drop(reading); let _ = report.done.send(()); } } pub fn flush(&self, why: &str) { self.close_where(|_| true, why); } - pub fn flush_execution_id(&self, execution_id: weft_core::ExecutionId, why: &str) { + /// End a run's spend: book every charge it still holds as unknown, then + /// wait until each of its figures is on record. Until that wait returns, + /// a charge the run opens late (a body that finished after this began) + /// is booked as unknown at once rather than held. Once it returns no + /// call of the run is left to open one, so the mark is dropped, and a + /// run that paused can open charges again when it resumes. + pub async fn close_execution_id(&self, execution_id: ExecutionId, pending: &PendingCostRecords, why: &str) { + struct Ending<'a>(&'a OpenCharges, ExecutionId); + impl Drop for Ending<'_> { + fn drop(&mut self) { self.0.book().ending.remove(&self.1); } + } + self.book().ending.insert(execution_id); + let _ending = Ending(self, execution_id); self.close_where(|charge| charge.sink.execution_id == execution_id, why); + pending.wait_zero().await; } /// Book every matching charge as unknown and drop it. A charge whose @@ -378,7 +432,8 @@ impl OpenCharges { /// that read, which is bounded by the follow-up client's timeout. fn close_where(&self, matches: impl Fn(&OpenCharge) -> bool, why: &str) { let held = { - let mut charges = self.by_id(); + let mut book = self.book(); + let charges = &mut book.by_id; let ids: Vec<_> = charges.iter_mut().filter_map(|(id, state)| { if !matches(&state.charge) { return None; } if state.processing { @@ -410,11 +465,9 @@ fn book_open_charge(charge: OpenCharge, why: &str) { "the call spent, and no response ever stated what it cost ({why})" )); let model = metadata["model"].as_str().map(str::to_string); - let sink = charge.sink.clone(); - // Book first, release the charge's token after: the count must not - // pass through zero while a figure is still to be written down. - sink.clone().book_resolved(MeasuredCost { amount_usd: None, model, metadata }); - sink.pending.end(); + // The charge's token moves into the booking, so the count cannot pass + // through zero while the figure is still to be written down. + charge.sink.book_resolved(charge.held, MeasuredCost { amount_usd: None, model, metadata }); } // ---------- The cost sink (where a measured figure lands) ---------- @@ -441,34 +494,38 @@ pub struct CostSink { /// Whose credential the connection rides; recorded on every figure /// so the cost trail says whose account spent. pub origin: weft_core::CredentialOwner, + /// The run's cancel flag: a call still going when it trips is let go + /// (see the module doc). + pub cancellation: Arc, } impl CostSink { /// Book one finished observation's figure: resolve `cost` and write /// it down durably, detached from the caller's future (which may be - /// aborted at any point) and tracked by the pending counter so the - /// process cannot exit while money is still being written. The one - /// begin/spawn/record/end sequence behind every finalizer, HTTP and - /// session alike; a session hands a ready future, an HTTP call the + /// aborted at any point). `held` is the pending token the spend has + /// carried since its call went out; it is released once the figure is + /// on record, so the run cannot end while money is still being + /// written. The one spawn/record sequence behind every finalizer, HTTP + /// and session alike; a session hands a ready future, an HTTP call the /// meter's resolve. pub(crate) fn book( self: Arc, + held: weft_core::in_flight::InFlightToken, cost: impl std::future::Future + Send + 'static, ) { - self.pending.begin(); match tokio::runtime::Handle::try_current() { Ok(handle) => { handle.spawn(async move { let cost = cost.await; self.record(cost).await; - self.pending.end(); + drop(held); }); } Err(_) => { // No runtime to spawn on (the process is tearing down // outside tokio): the record cannot be written. Say so // loudly; never drop money silently. - self.pending.end(); + drop(held); tracing::error!( target: "weft_engine::metering", "COST RECORD LOST for node {} ({}): the async runtime was already torn \ @@ -482,8 +539,8 @@ impl CostSink { /// Book a figure that is already resolved. The twin of [`Self::book`] /// for a charge whose amount arrived on a later response, where there /// is nothing left to await by the time it is known. - pub(crate) fn book_resolved(self: Arc, cost: MeasuredCost) { - self.book(async move { cost }); + pub(crate) fn book_resolved(self: Arc, held: weft_core::in_flight::InFlightToken, cost: MeasuredCost) { + self.book(held, std::future::ready(cost)); } /// Write the figure down: a `CostReported` on the run's record, one per @@ -781,7 +838,41 @@ impl reqwest_middleware::Middleware for MeteringMiddleware { let request_bytes: Vec = req.body().and_then(|b| b.as_bytes()).map(|b| b.to_vec()).unwrap_or_default(); - let response = next.run(req, extensions).await?; + // Named in the warning if the tap has to cut the call. + let host = req.url().host_str().unwrap_or("").to_string(); + + // The run's ending waits from here: the token rides the response's + // tap into the figure's record, and a body still being read after + // the node returned keeps it held. Until the answer starts, a + // cancel of the run lets the call go (the send may be on a task + // the node's abort does not reach). A run already cancelled sends + // nothing: no call goes out, so nothing is booked. + if self.sink.cancellation.is_cancelled() { + return Err(middleware_err(format!( + "the run was cancelled: the call to '{}' was not sent", + self.sink.service + ))); + } + let unanswered = Unanswered { + sink: self.sink.clone(), + host: host.clone(), + held: Some(self.sink.pending.hold()), + spends: matches!(class, RouteClass::Billable(_)), + }; + let response = tokio::select! { + biased; + sent = next.run(req, extensions) => match sent { + Ok(response) => response, + Err(e) => { + unanswered.failed(&e); + return Err(e); + } + }, + () = self.sink.cancellation.cancelled() => { + return Err(middleware_err(format!("{CANCELLED_BEFORE_ANSWER}: the call to '{}' was let go", self.sink.service))); + } + }; + let held = unanswered.answered(); // Tap the response: the caller sees every chunk in real time while // the observer reads what it needs in passing. When the stream @@ -798,7 +889,9 @@ impl reqwest_middleware::Middleware for MeteringMiddleware { // provider's own figure never has to re-derive one. observer.on_headers(&headers); let tapped = TapStream { - inner: response.bytes_stream().boxed(), + inner: Some(response.bytes_stream().boxed()), + host, + watch: None, finalizer: Some(Finalizer { observer, meter, @@ -807,6 +900,7 @@ impl reqwest_middleware::Middleware for MeteringMiddleware { open_charges: self.sink.open_charges.clone(), follow_up: self.follow_up.clone(), sink: self.sink.clone(), + held, }), }; // The rebuild carries the wire-visible surface only (status, @@ -838,6 +932,8 @@ struct Finalizer { /// What the meter's own follow-up queries ride. follow_up: FollowUpLane, sink: Arc, + /// The call's pending token, taken when it went out. + held: weft_core::in_flight::InFlightToken, } impl Finalizer { @@ -848,17 +944,18 @@ impl Finalizer { /// may answer 200 with a failure body it never bills, or bill a /// call it then refuses), so the declared price is the meter's /// input, never the middleware's verdict. - fn finish(self, interrupted: bool) { - let observed = self.observer.end(interrupted); - let meter = self.meter; - let route = self.route; - let follow_up = self.follow_up; - let open_charges = self.open_charges; - let sink = self.sink; + /// + /// `cut` is why the tap stopped reading the answer, when it did: it + /// lands on the figure (under `cut`), whatever the meter makes of the + /// part it saw. + fn finish(self, interrupted: bool, cut: Option) { + let Finalizer { observer, meter, route, class, open_charges, follow_up, sink, held } = self; + let observed = observer.end(interrupted); // A response that reports on an earlier spend books nothing of its - // own: it closes the charge that spend opened. - if matches!(self.class, RouteClass::Reports) { + // own: it closes the charge that spend opened, which holds its own + // run's token, so this call's token is released here. + if matches!(class, RouteClass::Reports) { let Some(id) = meter.charge_reported_on(&route, &observed) else { return }; drop(open_charges.report(meter, &id, &route, observed, &follow_up)); return; @@ -881,30 +978,224 @@ impl Finalizer { other => serde_json::json!({ "meterState": other }), }; scratch["route"] = serde_json::json!(route); - open_charges.open(meter.service(), id, OpenCharge { token: 0, scratch, sink }); + if let Some(cut) = cut { + scratch["cut"] = serde_json::json!(cut); + } + open_charges.open(meter.service(), id, OpenCharge { token: 0, scratch, sink, held }); return; } - sink.book(async move { - meter.resolve(&route, observed, follow_up.follow_up(meter.base_url())).await + sink.book(held, async move { + let mut cost = meter.resolve(&route, observed, follow_up.follow_up(meter.base_url())).await; + if let Some(cut) = cut { + // Shape-checked like `scratch` above, for the same reason. + if !cost.metadata.is_object() { + cost.metadata = serde_json::json!({ "meterState": cost.metadata }); + } + cost.metadata["cut"] = serde_json::json!(cut); + } + cost }); } } +/// How long a metered call's answer may send nothing while it is waited +/// on before the tap stops reading it (see the module doc). Real time, +/// never scaled: it waits on a provider, not on weft +/// (`weft_core::time_scale`). A streamed answer sends every token. The +/// wait for an answer's status and headers is before the tap exists, so +/// a provider that takes minutes to answer at all is not affected; one +/// that answers its headers and then sends nothing for this long is. +pub(crate) const ANSWER_SILENCE_LIMIT: Duration = Duration::from_secs(60); + +/// Why a call cancelled before its answer started was let go. +const CANCELLED_BEFORE_ANSWER: &str = "the run was cancelled before the provider answered"; + +/// A metered call between going out and its answer starting: the call's +/// pending token, and what a call let go here books. A call that spends +/// and ends here without an answer (its run cancelled, its caller gone, its +/// send failed after the connection was made) may have been charged, so it +/// is booked as an unknown spend, never as nothing. Only a call that never +/// reached the provider books nothing. +struct Unanswered { + sink: Arc, + host: String, + held: Option, + /// A billable route; a reporting route spends nothing. + spends: bool, +} + +impl Unanswered { + /// The answer started: its tap carries the token on. + fn answered(mut self) -> weft_core::in_flight::InFlightToken { + self.held.take().expect("the token is only taken here or by `failed`") + } + + /// The send failed, and the error goes back to the caller. A failure to + /// connect means the provider never saw the call: nothing is booked. + /// Any other (a timeout, a reset once the request went out) may come + /// after the provider started work it bills, so it is booked as unknown. + fn failed(mut self, error: &reqwest_middleware::Error) { + if never_connected(error) { + self.held = None; + return; + } + self.book_unknown(&format!("the call failed before the provider answered: {error}")); + } + + /// Book the call as an unknown spend, naming `why`, and release its + /// token once that is on record. A reporting route spends nothing. + fn book_unknown(&mut self, why: &str) { + let Some(held) = self.held.take() else { return }; + if !self.spends { + return; + } + tracing::warn!( + target: "weft_engine::metering", + execution_id = %self.sink.execution_id, + node = %self.sink.node_id, + service = %self.sink.service, + host = %self.host, + "{why}; its cost is booked as unknown", + ); + let cost = MeasuredCost { + amount_usd: None, + model: None, + metadata: serde_json::json!({ "resolution": format!("unknown: {why}") }), + }; + self.sink.clone().book_resolved(held, cost); + } +} + +impl Drop for Unanswered { + fn drop(&mut self) { + let why = if self.sink.cancellation.is_cancelled() { + CANCELLED_BEFORE_ANSWER + } else { + "the call was dropped before the provider answered" + }; + self.book_unknown(why); + } +} + +/// Whether a send failed before any connection to the provider was made: +/// a connect error anywhere in its chain. +fn never_connected(error: &reqwest_middleware::Error) -> bool { + let mut cause: Option<&(dyn std::error::Error + 'static)> = match error { + reqwest_middleware::Error::Reqwest(e) => Some(e), + reqwest_middleware::Error::Middleware(e) => Some(e.as_ref()), + }; + while let Some(e) = cause { + if e.downcast_ref::().is_some_and(reqwest::Error::is_connect) { + return true; + } + cause = e.source(); + } + false +} + +/// Why the tap stopped reading an answer. +#[derive(Debug, Clone, Copy)] +enum Cut { + Silent, + Cancelled, +} + +impl std::fmt::Display for Cut { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Cut::Silent => write!(f, "the provider sent nothing for {}s", ANSWER_SILENCE_LIMIT.as_secs()), + Cut::Cancelled => f.write_str("the run was cancelled before the call's answer finished"), + } + } +} + +impl std::error::Error for Cut {} + +/// What the tap's reader is handed: the body's own error, or a [`Cut`]. +type TapError = Box; + /// The response-body tap: forwards every chunk untouched and unbuffered, /// feeding the observer in passing. Finalizes exactly once: on clean end, -/// on stream error, or on drop (the caller hung up mid-stream). +/// on stream error, on a cut, or on drop (the caller hung up mid-stream). struct TapStream { - inner: BoxStream<'static, reqwest::Result>, + /// `None` once the tap cut the answer: the body is dropped, and its + /// connection with it. + inner: Option>>, finalizer: Option, + /// The host the call went to, named if the tap cuts it. + host: String, + /// What can cut the answer, made the first time the reader waits. + watch: Option, +} + +/// The two ways a waited-on answer is cut (see the module doc). +struct Watch { + cancelled: Pin + Send>>, + silence: Pin>, + /// The reader is waiting, since the silence deadline was last set: a + /// chunk clears it, and the next wait sets the deadline again. + waiting: bool, +} + +impl TapStream { + /// Nothing is ready for the reader: whether the answer is to be cut. + /// The silence counts from when the reader started waiting, so a + /// reader that was busy elsewhere never cuts an answer that waited + /// for it. + fn stalled(&mut self, cx: &mut Context<'_>) -> Option { + let cancellation = &self.finalizer.as_ref()?.sink.cancellation; + let watch = self.watch.get_or_insert_with(|| { + let cancellation = cancellation.clone(); + Watch { + cancelled: Box::pin(async move { cancellation.cancelled().await }), + silence: Box::pin(tokio::time::sleep(ANSWER_SILENCE_LIMIT)), + waiting: true, + } + }); + if !watch.waiting { + watch.waiting = true; + watch.silence.as_mut().reset(tokio::time::Instant::now() + ANSWER_SILENCE_LIMIT); + } + if watch.cancelled.poll_unpin(cx).is_ready() { + return Some(Cut::Cancelled); + } + if watch.silence.poll_unpin(cx).is_ready() { + return Some(Cut::Silent); + } + None + } + + /// Stop reading the answer: drop the body, book the call as cut, and + /// hand the reader the reason as the body's error. + fn cut(&mut self, cut: Cut) -> Poll>> { + self.inner = None; + self.watch = None; + if let Some(f) = self.finalizer.take() { + tracing::warn!( + target: "weft_engine::metering", + execution_id = %f.sink.execution_id, + node = %f.sink.node_id, + service = %f.sink.service, + host = %self.host, + "{cut}: weft stopped reading the call's answer, and its cost is booked from what arrived", + ); + f.finish(true, Some(cut.to_string())); + } + Poll::Ready(Some(Err(Box::new(cut)))) + } } impl Stream for TapStream { - type Item = reqwest::Result; + type Item = Result; fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - match self.inner.poll_next_unpin(cx) { + let Some(inner) = self.inner.as_mut() else { return Poll::Ready(None) }; + match inner.poll_next_unpin(cx) { Poll::Ready(Some(Ok(chunk))) => { + if let Some(watch) = self.watch.as_mut() { + watch.waiting = false; + } if let Some(f) = self.finalizer.as_mut() { f.observer.on_chunk(&chunk); } @@ -912,17 +1203,20 @@ impl Stream for TapStream { } Poll::Ready(Some(Err(e))) => { if let Some(f) = self.finalizer.take() { - f.finish(true); + f.finish(true, None); } - Poll::Ready(Some(Err(e))) + Poll::Ready(Some(Err(Box::new(e)))) } Poll::Ready(None) => { if let Some(f) = self.finalizer.take() { - f.finish(false); + f.finish(false, None); } Poll::Ready(None) } - Poll::Pending => Poll::Pending, + Poll::Pending => match self.stalled(cx) { + Some(cut) => self.cut(cut), + None => Poll::Pending, + }, } } } @@ -930,7 +1224,7 @@ impl Stream for TapStream { impl Drop for TapStream { fn drop(&mut self) { if let Some(f) = self.finalizer.take() { - f.finish(true); + f.finish(true, None); } } } @@ -1129,7 +1423,7 @@ pub(crate) mod tests { relay_url: Option, ) -> (reqwest_middleware::ClientWithMiddleware, Arc, Arc) { - rig_owned(base, relay_url, weft_core::CredentialOwner::Author, None) + rig_owned(base, relay_url, weft_core::CredentialOwner::Author, None, weft_core::cancellation::CancellationFlag::new_arc()) } fn rig_owned( @@ -1137,6 +1431,7 @@ pub(crate) mod tests { relay_url: Option, origin: weft_core::CredentialOwner, fixed_usd: Option, + cancellation: Arc, ) -> (reqwest_middleware::ClientWithMiddleware, Arc, Arc) { let tasks = Arc::new(RecordedCosts::default()); @@ -1152,6 +1447,7 @@ pub(crate) mod tests { frames: LoopFrames::default(), service: "testprov".into(), origin, + cancellation, }; // The follow-up client of the rig: same signed-in shape the // production composition builds (auth-free here; the test meter @@ -1193,16 +1489,6 @@ pub(crate) mod tests { .collect() } - async fn wait_recorded(tasks: &RecordedCosts, pending: &PendingCostRecords) { - pending.wait_zero().await; - for _ in 0..100 { - if !tasks.events.lock().unwrap().is_empty() { - return; - } - tokio::time::sleep(std::time::Duration::from_millis(10)).await; - } - } - // L1-shaped stress pin for the pending counter's wakeup: `end()` firing // concurrently with `wait_zero` arming must never be missed (a lost // wakeup HANGS the await; the explicit timeout turns that hang into a @@ -1220,14 +1506,10 @@ pub(crate) mod tests { for i in 0..400usize { let pending = PendingCostRecords::new(); let records = if i % 2 == 0 { 1 } else { 3 }; - for _ in 0..records { - pending.begin(); - } - let enders: Vec<_> = (0..records) - .map(|_| { - let ender = pending.clone(); - tokio::spawn(async move { ender.end() }) - }) + let held: Vec<_> = (0..records).map(|_| pending.hold()).collect(); + let enders: Vec<_> = held + .into_iter() + .map(|token| tokio::spawn(async move { drop(token) })) .collect(); tokio::time::timeout(std::time::Duration::from_secs(5), pending.wait_zero()) .await @@ -1282,6 +1564,7 @@ pub(crate) mod tests { frames: LoopFrames::default(), service: "testprov".into(), origin: weft_core::CredentialOwner::Author, + cancellation: weft_core::cancellation::CancellationFlag::new_arc(), }; // The exact production stack (metering outer, auth inner), // with the meter injected directly since the global @@ -1319,7 +1602,7 @@ pub(crate) mod tests { .expect("send"); response.bytes().await.expect("body"); - wait_recorded(&tasks, &pending).await; + pending.wait_zero().await; let payloads = recorded_payloads(&tasks); assert_eq!(payloads.len(), 1, "the sign-in call was measured"); assert_eq!(payloads[0].amount_usd, Some(0.5)); @@ -1331,6 +1614,64 @@ pub(crate) mod tests { assert_eq!(received.lock().unwrap()[0]["usage"]["include"], true); } + // L3, the order a client library that reads the body on a task of its + // own produces (an LLM client parsing the SSE stream, say): the node has + // its answer once the usage chunk arrives and returns, while that task + // is still reading the stream's tail, so the tap finishes after the node + // is gone. The run's ending must still wait for the call's figure and + // find it on record once it opens. Stress-looped: the reader task, the + // tap and the gate meet on a shared multi-thread runtime. + weft_core::stress_test! { + name: a_body_read_after_the_node_returned_still_holds_the_runs_ending, + runs: 32, + worker_threads: 4, + async fn body() { + let (base, chunk_tx, _received) = spawn_sse_server().await; + let base: &'static str = Box::leak(base.into_boxed_str()); + let (client, tasks, pending) = rig(base, None); + + // The answer and its usage arrive; the stream's tail has not. + chunk_tx.send(Bytes::from("data: {\"usage\":{\"cost\":0.25}}\n\n")).unwrap(); + let response = client + .post(format!("{base}/chat/completions")) + .json(&serde_json::json!({"model": "m", "messages": []})) + .send() + .await + .expect("send"); + let (answer_tx, answer_rx) = tokio::sync::oneshot::channel(); + let reader = tokio::spawn(async move { + let mut body = response.bytes_stream(); + let first = body.next().await.expect("a chunk").expect("readable"); + let _ = answer_tx.send(first); + while let Some(chunk) = body.next().await { + chunk.expect("readable"); + } + }); + let answer = answer_rx.await.expect("the node got its answer"); + assert!(String::from_utf8_lossy(&answer).contains("0.25")); + + // The node returned and the run ends: its gate stays shut while + // the tap still holds the call's spend. + let ending = pending.wait_zero(); + tokio::pin!(ending); + assert!( + futures::poll!(&mut ending).is_pending(), + "the run's ending opened while its call's body was still being read" + ); + + // The tail arrives and the reader finishes the body. + chunk_tx.send(Bytes::from("data: [DONE]\n\n")).unwrap(); + drop(chunk_tx); + tokio::time::timeout(std::time::Duration::from_secs(10), ending) + .await + .expect("the ending opens once the figure is on record"); + let payloads = recorded_payloads(&tasks); + assert_eq!(payloads.len(), 1, "the figure is on record before the ending"); + assert_eq!(payloads[0].amount_usd, Some(0.25)); + reader.await.unwrap(); + } + } + /// A FIXED-price route still resolves THROUGH its meter (whether a /// fixed-priced call was charged is provider knowledge): the record /// carries the declared price and names the fixed resolution. @@ -1339,7 +1680,7 @@ pub(crate) mod tests { let (base, chunk_tx, _received) = spawn_sse_server().await; let base: &'static str = Box::leak(base.into_boxed_str()); let (client, tasks, pending) = - rig_owned(base, None, weft_core::CredentialOwner::Author, Some(0.001)); + rig_owned(base, None, weft_core::CredentialOwner::Author, Some(0.001), weft_core::cancellation::CancellationFlag::new_arc()); chunk_tx.send(Bytes::from("data: [DONE]\n\n")).unwrap(); drop(chunk_tx); @@ -1351,7 +1692,7 @@ pub(crate) mod tests { .expect("send"); response.bytes().await.expect("body"); - wait_recorded(&tasks, &pending).await; + pending.wait_zero().await; let payloads = recorded_payloads(&tasks); assert_eq!(payloads.len(), 1, "the fixed call was booked"); assert_eq!(payloads[0].amount_usd, Some(0.001)); @@ -1384,6 +1725,7 @@ pub(crate) mod tests { frames: LoopFrames::default(), service: "no_such_meterless_service".into(), origin: weft_core::CredentialOwner::Author, + cancellation: weft_core::cancellation::CancellationFlag::new_arc(), }; let client = connection_client("no_such_meterless_service", steps, None, Arc::new(sink), weft_core::shared::Shared::new(std::time::Duration::MAX)) @@ -1416,6 +1758,7 @@ pub(crate) mod tests { frames: LoopFrames::default(), service: "no_such_meterless_service".into(), origin: weft_core::CredentialOwner::Platform, + cancellation: weft_core::cancellation::CancellationFlag::new_arc(), }; let err = connection_client( "no_such_meterless_service", @@ -1470,7 +1813,7 @@ pub(crate) mod tests { chunk.expect("chunk ok"); } - wait_recorded(&tasks, &pending).await; + pending.wait_zero().await; let payloads = recorded_payloads(&tasks); assert_eq!(payloads.len(), 1); assert_eq!(payloads[0].amount_usd, Some(0.000031)); @@ -1501,7 +1844,7 @@ pub(crate) mod tests { // Hang up before the usage chunk ever arrives. drop(stream); - wait_recorded(&tasks, &pending).await; + pending.wait_zero().await; let payloads = recorded_payloads(&tasks); assert_eq!(payloads.len(), 1, "an interrupted call still gets its record"); assert_eq!( @@ -1511,6 +1854,222 @@ pub(crate) mod tests { assert_eq!(payloads[0].metadata["interrupted"], true); } + /// A tap over `body` for a call of the run whose cancel flag is + /// `cancellation`, as the middleware builds it once the answer started, + /// booking into the returned record. + fn tap_over( + body: impl Stream> + Send + 'static, + cancellation: Arc, + ) -> (TapStream, Arc, Arc, Arc) { + let tasks = Arc::new(RecordedCosts::default()); + let pending = PendingCostRecords::new(); + let open_charges = OpenCharges::new(); + let sink = Arc::new(CostSink { + journal: tasks.clone(), + writer: test_writer(), + replica: "w".into(), + pending: pending.clone(), + open_charges: open_charges.clone(), + execution_id: uuid::Uuid::nil(), + node_id: "node-x".into(), + frames: LoopFrames::default(), + service: "testprov".into(), + origin: weft_core::CredentialOwner::Author, + cancellation, + }); + let meter: &'static TestMeter = Box::leak(Box::new(TestMeter { base: "http://provider.test/api/v1", fixed_usd: None })); + let follow_up = FollowUpLane { + http: reqwest_middleware::ClientBuilder::new(reqwest::Client::new()).build(), + shared: weft_core::shared::Shared::new(Duration::MAX), + }; + let tap = TapStream { + inner: Some(body.boxed()), + finalizer: Some(Finalizer { + observer: meter.observe("chat/completions", "", b""), + meter, + route: "chat/completions".into(), + class: RouteClass::Billable(weft_providers::Pricing::Metered), + open_charges: open_charges.clone(), + follow_up, + sink, + held: pending.hold(), + }), + host: "provider.test".into(), + watch: None, + }; + (tap, tasks, pending, open_charges) + } + + /// What a client library's own reading task does with an answer: read + /// it to its end or its first error, then drop it. + async fn read_to_end(mut tap: TapStream) -> Result { + let mut chunks = 0; + while let Some(chunk) = tap.next().await { + chunk.map_err(|e| e.to_string())?; + chunks += 1; + } + Ok(chunks) + } + + const CONTENT: &str = "data: {\"choices\":[{\"delta\":{\"content\":\"hi\"}}]}\n\n"; + + /// A provider that stays connected and sends nothing: once the reader + /// waited the silence limit, the tap cuts the answer, the reader gets + /// the reason as an error (so the library's task ends), the call is + /// booked unknown with that reason, and the run's ending returns. + #[tokio::test(start_paused = true)] + async fn a_stalled_answer_is_cut_after_the_silence_limit_and_the_run_ends() { + let body = futures::stream::iter([Ok(Bytes::from(CONTENT))]).chain(futures::stream::pending()); + let (tap, tasks, pending, charges) = tap_over(body, weft_core::cancellation::CancellationFlag::new_arc()); + let started = tokio::time::Instant::now(); + let library = tokio::spawn(read_to_end(tap)); + + charges.close_execution_id(uuid::Uuid::nil(), &pending, "the run ended").await; + assert!(started.elapsed() >= ANSWER_SILENCE_LIMIT, "cut once the silence limit passed, not before"); + assert_eq!(library.await.unwrap(), Err(Cut::Silent.to_string()), "the reader is told why"); + let payloads = recorded_payloads(&tasks); + assert_eq!(payloads.len(), 1); + assert_eq!(payloads[0].amount_usd, None, "nothing stated the cost: unknown, never $0"); + assert_eq!(payloads[0].metadata["cut"], "the provider sent nothing for 60s"); + assert_eq!(payloads[0].metadata["interrupted"], true); + } + + /// A cancel of the run cuts an answer still being read at once, on a + /// task the node's abort never reaches, and the run's ending returns. + #[tokio::test(start_paused = true)] + async fn a_cancelled_runs_answer_is_cut_at_once_and_the_run_ends() { + let cancellation = weft_core::cancellation::CancellationFlag::new_arc(); + let body = futures::stream::iter([Ok(Bytes::from(CONTENT))]).chain(futures::stream::pending()); + let (tap, tasks, pending, charges) = tap_over(body, cancellation.clone()); + let library = tokio::spawn(read_to_end(tap)); + tokio::task::yield_now().await; + + let started = tokio::time::Instant::now(); + cancellation.cancel_because(weft_core::exec::CancelCause::User); + charges.close_execution_id(uuid::Uuid::nil(), &pending, "the run was cancelled").await; + assert!(started.elapsed() < ANSWER_SILENCE_LIMIT, "cut by the cancel, not by the silence limit"); + assert_eq!(library.await.unwrap(), Err(Cut::Cancelled.to_string())); + let payloads = recorded_payloads(&tasks); + assert_eq!(payloads.len(), 1); + assert_eq!(payloads[0].amount_usd, None); + assert_eq!(payloads[0].metadata["cut"], "the run was cancelled before the call's answer finished"); + } + + /// A live answer whose gaps each stay under the limit (together far + /// over it), read by a reader that is then busy elsewhere for longer + /// than the limit, is never cut and books its stated cost. + #[tokio::test(start_paused = true)] + async fn a_slow_but_live_answer_is_never_cut() { + let (tx, rx) = tokio::sync::mpsc::unbounded_channel::>(); + let (mut tap, tasks, pending, _charges) = + tap_over(tokio_stream::wrappers::UnboundedReceiverStream::new(rx), weft_core::cancellation::CancellationFlag::new_arc()); + let gap = ANSWER_SILENCE_LIMIT - Duration::from_secs(1); + tokio::spawn(async move { + for _ in 0..3 { + tokio::time::sleep(gap).await; + tx.send(Ok(Bytes::from(CONTENT))).unwrap(); + } + // Sent while the reader is busy elsewhere. + tokio::time::sleep(ANSWER_SILENCE_LIMIT).await; + tx.send(Ok(Bytes::from("data: {\"usage\":{\"cost\":0.000031}}\n\n"))).unwrap(); + }); + for _ in 0..3 { + tap.next().await.expect("open").expect("a live chunk is never cut"); + } + tokio::time::sleep(ANSWER_SILENCE_LIMIT * 2).await; + tap.next().await.expect("open").expect("the chunk that waited for the reader"); + assert!(tap.next().await.is_none(), "the answer ends cleanly"); + + pending.wait_zero().await; + let payloads = recorded_payloads(&tasks); + assert_eq!(payloads.len(), 1); + assert_eq!(payloads[0].amount_usd, Some(0.000031)); + assert!(payloads[0].metadata.get("cut").is_none(), "nothing was cut"); + } + + /// A call whose answer has not even started when its run is cancelled + /// is let go at once (its send may be on a task the node's abort does + /// not reach), and booked as an unknown spend. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn a_cancel_lets_go_of_a_call_still_waiting_for_its_answer() { + let app = axum::Router::new() + .route("/api/v1/chat/completions", axum::routing::post(std::future::pending::<&'static str>)); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let base: &'static str = Box::leak(format!("http://{}/api/v1", listener.local_addr().unwrap()).into_boxed_str()); + tokio::spawn(async move { axum::serve(listener, app).await.unwrap() }); + let cancellation = weft_core::cancellation::CancellationFlag::new_arc(); + let (client, tasks, pending) = rig_owned(base, None, weft_core::CredentialOwner::Author, None, cancellation.clone()); + + let send = tokio::spawn(async move { + client.post(format!("{base}/chat/completions")).json(&serde_json::json!({"model": "m", "messages": []})).send().await.map(drop) + }); + tokio::time::timeout(Duration::from_secs(5), async { + while pending.count() == 0 { + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await + .expect("the call went out"); + cancellation.cancel_because(weft_core::exec::CancelCause::User); + let err = tokio::time::timeout(Duration::from_secs(5), send).await.expect("let go at once").unwrap().unwrap_err(); + assert!(format!("{err:?}").contains(CANCELLED_BEFORE_ANSWER), "{err:?}"); + tokio::time::timeout(Duration::from_secs(5), pending.wait_zero()).await.expect("the run's ending is not held"); + let payloads = recorded_payloads(&tasks); + assert_eq!(payloads.len(), 1); + assert_eq!(payloads[0].amount_usd, None); + assert_eq!(payloads[0].metadata["resolution"], format!("unknown: {CANCELLED_BEFORE_ANSWER}")); + } + + /// A billable call on a run already cancelled is refused before it is + /// sent: the provider never sees it, and nothing is booked. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn a_call_on_a_cancelled_run_is_not_sent_and_books_nothing() { + let (base, _chunk_tx, received) = spawn_sse_server().await; + let base: &'static str = Box::leak(base.into_boxed_str()); + let cancellation = weft_core::cancellation::CancellationFlag::new_arc(); + cancellation.cancel_because(weft_core::exec::CancelCause::User); + let (client, tasks, pending) = rig_owned(base, None, weft_core::CredentialOwner::Author, None, cancellation); + + let err = client.post(format!("{base}/chat/completions")).json(&serde_json::json!({"model": "m", "messages": []})).send().await.unwrap_err(); + assert!(format!("{err:?}").contains("was not sent"), "{err:?}"); + pending.wait_zero().await; + assert!(recorded_payloads(&tasks).is_empty(), "no call went out, so nothing is booked"); + assert!(received.lock().unwrap().is_empty(), "the provider never saw it"); + } + + /// A send that never connected books nothing; one that failed once the + /// request reached the provider (here: it hung up without answering) + /// may have been billed, so it is booked as an unknown spend. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn a_failed_send_books_unknown_only_once_the_provider_was_reached() { + let closed = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let closed_base: &'static str = Box::leak(format!("http://{}/api/v1", closed.local_addr().unwrap()).into_boxed_str()); + drop(closed); + let (client, tasks, pending) = rig(closed_base, None); + let err = client.post(format!("{closed_base}/chat/completions")).json(&serde_json::json!({"model": "m", "messages": []})).send().await.unwrap_err(); + assert!(never_connected(&err), "{err:?}"); + pending.wait_zero().await; + assert!(recorded_payloads(&tasks).is_empty(), "a call that never connected books nothing"); + + let hangs_up = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let hangs_up_base: &'static str = Box::leak(format!("http://{}/api/v1", hangs_up.local_addr().unwrap()).into_boxed_str()); + tokio::spawn(async move { + use tokio::io::AsyncReadExt as _; + let (mut socket, _) = hangs_up.accept().await.unwrap(); + let mut request = [0u8; 1024]; + let _ = socket.read(&mut request).await; + }); + let (client, tasks, pending) = rig(hangs_up_base, None); + let err = client.post(format!("{hangs_up_base}/chat/completions")).json(&serde_json::json!({"model": "m", "messages": []})).send().await.unwrap_err(); + assert!(!never_connected(&err), "{err:?}"); + tokio::time::timeout(Duration::from_secs(5), pending.wait_zero()).await.expect("booked and released"); + let payloads = recorded_payloads(&tasks); + assert_eq!(payloads.len(), 1); + assert_eq!(payloads[0].amount_usd, None); + let resolution = payloads[0].metadata["resolution"].as_str().unwrap(); + assert!(resolution.starts_with("unknown: the call failed before the provider answered"), "{resolution}"); + } + /// The runtime-credential allowlist: on origin Ours, an UNKNOWN /// route (or a URL off the provider's API) is refused loudly and /// nothing is sent, an explicitly FREE route passes; the same @@ -1520,7 +2079,7 @@ pub(crate) mod tests { let (base, _chunk_tx, received) = spawn_sse_server().await; let leaked: &'static str = Box::leak(base.clone().into_boxed_str()); let (ours, tasks, _pending) = - rig_owned(leaked, None, weft_core::CredentialOwner::Platform, None); + rig_owned(leaked, None, weft_core::CredentialOwner::Platform, None, weft_core::cancellation::CancellationFlag::new_arc()); // Unknown route: refused, the server never sees it. let err = ours @@ -1797,6 +2356,7 @@ mod open_charge_tests { frames: LoopFrames::default(), service: "queued".into(), origin: weft_core::CredentialOwner::Author, + cancellation: weft_core::cancellation::CancellationFlag::new_arc(), }) } @@ -1818,6 +2378,7 @@ mod open_charge_tests { frames: LoopFrames::default(), service: "queued".into(), origin: weft_core::CredentialOwner::Author, + cancellation: weft_core::cancellation::CancellationFlag::new_arc(), }) } @@ -1847,7 +2408,7 @@ mod open_charge_tests { let sink = sink(tasks.clone(), pending.clone()); let charges = sink.open_charges.clone(); - charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, sink }); + charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, held: sink.pending.hold(), sink }); assert_eq!(charges.count(), 1, "the charge is held between spend and figure"); assert!(recorded_payloads(&tasks).is_empty(), "nothing is booked yet"); @@ -1876,7 +2437,7 @@ mod open_charge_tests { let pending = PendingCostRecords::new(); let sink = sink(tasks.clone(), pending.clone()); let charges = sink.open_charges.clone(); - charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, sink }); + charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, held: sink.pending.hold(), sink }); // No `units`: the job is still running. let pending_report = ObservedCall { @@ -1903,7 +2464,7 @@ mod open_charge_tests { let pending = PendingCostRecords::new(); let execution_id = uuid::Uuid::new_v4(); let sink = sink_on(tasks.clone(), pending.clone(), charges.clone(), execution_id); - charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, sink }); + charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, held: sink.pending.hold(), sink }); drop(charges.report(&QUEUED, "req-1", "requests/req-1", ObservedCall { interrupted: false, status: 200, data: serde_json::json!({"status": "IN_PROGRESS"}), @@ -1911,8 +2472,7 @@ mod open_charge_tests { drop(charges.report(&QUEUED, "req-1", "requests/req-1", ObservedCall { interrupted: false, status: 200, data: serde_json::json!({"units": 3.0}), }, &lane())); - charges.flush_execution_id(execution_id, "the execution ended before the job was read back"); - pending.wait_zero().await; + charges.close_execution_id(execution_id, &pending, "the execution ended before the job was read back").await; assert_eq!(charges.count(), 0); let booked = recorded_payloads(&tasks); @@ -1936,7 +2496,7 @@ mod open_charge_tests { let pending = PendingCostRecords::new(); let execution_id = weft_core::ExecutionId::new_v4(); let sink = sink_on(tasks.clone(), pending.clone(), charges.clone(), execution_id); - charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, sink: sink.clone() }); + charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, held: sink.pending.hold(), sink: sink.clone() }); // The fake meter yields once inside `fold_report`; the displacing // open lands in that window. let read = charges.report(&QUEUED, "req-1", "requests/req-1", ObservedCall { @@ -1944,10 +2504,9 @@ mod open_charge_tests { }, &lane()); // Let the spawned read reach the meter's own await before displacing. tokio::task::yield_now().await; - charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, sink }); + charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, held: sink.pending.hold(), sink }); read.await; - charges.flush_execution_id(execution_id, "the execution ended"); - pending.wait_zero().await; + charges.close_execution_id(execution_id, &pending, "the execution ended").await; assert_eq!(charges.count(), 0); let mut amounts: Vec> = recorded_payloads(&tasks).iter().map(|record| record.amount_usd).collect(); amounts.sort_by(|a, b| a.partial_cmp(b).unwrap()); @@ -1955,6 +2514,72 @@ mod open_charge_tests { "the displaced charge as unknown, the second charge as unknown at close, and the figure the read produced"); } + /// A response body that finishes after its run began ending (a client + /// library reading the stream on its own task) opens its charge too + /// late for the close to see it. The charge carries the call's token, + /// taken when the call went out, so held open it would keep the ending + /// waiting forever; it is booked as unknown at once and the ending + /// completes. + #[tokio::test] + async fn a_charge_opened_after_its_run_began_ending_is_booked_and_the_ending_completes() { + let tasks = Arc::new(RecordedCosts::default()); + let charges = OpenCharges::new(); + let pending = PendingCostRecords::new(); + let execution_id = weft_core::ExecutionId::new_v4(); + let sink = sink_on(tasks.clone(), pending.clone(), charges.clone(), execution_id); + // The call went out: its token is taken, its body still streaming. + let held = sink.pending.hold(); + + let ending = charges.close_execution_id(execution_id, &pending, "the run ended before the job was read back"); + tokio::pin!(ending); + assert!(futures::poll!(&mut ending).is_pending(), "the ending waits on the call still in flight"); + + // The body finishes now, and its meter opens a charge. + charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, held, sink }); + tokio::time::timeout(std::time::Duration::from_secs(5), ending) + .await + .expect("a charge opened after the close must not hold the ending open"); + + assert_eq!(charges.count(), 0, "nothing is left held for a run that ended"); + let booked = recorded_payloads(&tasks); + assert_eq!(booked.len(), 1); + assert!(booked[0].amount_usd.is_none(), "spend with no figure is unknown, never zero"); + assert!(charges.book().ending.is_empty(), "the mark goes once the ending's wait returned"); + } + + // The same late open racing the close from another thread: whichever + // side wins the lock, the charge is booked and the ending completes. + weft_core::stress_test! { + name: a_late_charge_racing_the_close_never_holds_the_ending, + runs: 200, + worker_threads: 4, + async fn body() { + let tasks = Arc::new(RecordedCosts::default()); + let charges = OpenCharges::new(); + let pending = PendingCostRecords::new(); + let execution_id = weft_core::ExecutionId::new_v4(); + let sink = sink_on(tasks.clone(), pending.clone(), charges.clone(), execution_id); + let held = sink.pending.hold(); + let opener = { + let charges = charges.clone(); + tokio::spawn(async move { + charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, held, sink }); + }) + }; + tokio::time::timeout( + std::time::Duration::from_secs(5), + charges.close_execution_id(execution_id, &pending, "the run ended"), + ) + .await + .expect("the ending completes whichever side won"); + opener.await.unwrap(); + assert_eq!(charges.count(), 0); + let booked = recorded_payloads(&tasks); + assert_eq!(booked.len(), 1); + assert!(booked[0].amount_usd.is_none()); + } + } + /// A figure that arrives after its execution closed the charge has /// nowhere to go: the charge is on the trail as unknown, exactly /// once, and the late report neither books again nor panics. @@ -1965,8 +2590,8 @@ mod open_charge_tests { let pending = PendingCostRecords::new(); let execution_id = weft_core::ExecutionId::new_v4(); let sink = sink_on(tasks.clone(), pending.clone(), charges.clone(), execution_id); - charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, sink }); - charges.flush_execution_id(execution_id, "the execution ended before the job was read back"); + charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, held: sink.pending.hold(), sink }); + charges.close_execution_id(execution_id, &pending, "the execution ended before the job was read back").await; charges.report(&QUEUED, "req-1", "requests/req-1", ObservedCall { interrupted: false, status: 200, data: serde_json::json!({"units": 3.0}), }, &lane()).await; @@ -1985,16 +2610,15 @@ mod open_charge_tests { let mine_pending = PendingCostRecords::new(); let execution_id = uuid::Uuid::new_v4(); let mine = sink_on(tasks.clone(), mine_pending.clone(), charges.clone(), execution_id); - charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, sink: mine }); + charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, held: mine.pending.hold(), sink: mine }); // Another execution on the same process, still going. Its own // pending tracker, so waiting for this execution's records does // not wait on a charge that is meant to stay open. let other = sink_on(tasks.clone(), PendingCostRecords::new(), charges.clone(), uuid::Uuid::new_v4()); - charges.open(QUEUED.service(), "req-2".into(), OpenCharge { token: 0, scratch: submitted("req-2").data, sink: other }); + charges.open(QUEUED.service(), "req-2".into(), OpenCharge { token: 0, scratch: submitted("req-2").data, held: other.pending.hold(), sink: other }); - charges.flush_execution_id(execution_id, "the execution ended before the job was read back"); - mine_pending.wait_zero().await; + charges.close_execution_id(execution_id, &mine_pending, "the execution ended before the job was read back").await; assert_eq!(charges.count(), 1, "the other execution's charge is untouched"); let booked = recorded_payloads(&tasks); @@ -2017,15 +2641,14 @@ mod open_charge_tests { let execution_id = uuid::Uuid::new_v4(); let first = sink_on(tasks.clone(), pending.clone(), charges.clone(), execution_id); - charges.open("queued", "job-1".into(), OpenCharge { token: 0, scratch: submitted("job-1").data, sink: first }); + charges.open("queued", "job-1".into(), OpenCharge { token: 0, scratch: submitted("job-1").data, held: first.pending.hold(), sink: first }); let second = sink_on(tasks.clone(), pending.clone(), charges.clone(), execution_id); - charges.open("otherprov", "job-1".into(), OpenCharge { token: 0, scratch: submitted("job-1").data, sink: second }); + charges.open("otherprov", "job-1".into(), OpenCharge { token: 0, scratch: submitted("job-1").data, held: second.pending.hold(), sink: second }); assert_eq!(charges.count(), 2, "neither displaced the other"); assert!(recorded_payloads(&tasks).is_empty(), "nothing was booked as displaced"); - charges.flush_execution_id(execution_id, "the execution ended"); - pending.wait_zero().await; + charges.close_execution_id(execution_id, &pending, "the execution ended").await; assert_eq!(recorded_payloads(&tasks).len(), 2, "both spends are written down"); } @@ -2037,7 +2660,7 @@ mod open_charge_tests { let pending = PendingCostRecords::new(); let sink = sink(tasks.clone(), pending.clone()); let charges = sink.open_charges.clone(); - charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, sink }); + charges.open(QUEUED.service(), "req-1".into(), OpenCharge { token: 0, scratch: submitted("req-1").data, held: sink.pending.hold(), sink }); charges.flush("the replica shut down"); pending.wait_zero().await; diff --git a/crates/weft-engine/src/socket.rs b/crates/weft-engine/src/socket.rs index 86b7eaf3..04fb11da 100644 --- a/crates/weft-engine/src/socket.rs +++ b/crates/weft-engine/src/socket.rs @@ -219,7 +219,7 @@ impl SocketDial for ConnectionSocketDial { Ok(ProviderSocket::new(Box::new(MeteredSocket { stream, finalizer: observation - .map(|observer| SessionFinalizer { observer, sink: self.sink.clone() }), + .map(|observer| SessionFinalizer { observer, held: self.sink.pending.hold(), sink: self.sink.clone() }), }))) } } @@ -238,6 +238,9 @@ struct MeteredSocket { struct SessionFinalizer { observer: Box, sink: Arc, + /// The session's pending token, taken when it opened: a session held + /// past its node's return keeps the run's ending waiting for its figure. + held: weft_core::in_flight::InFlightToken, } impl SessionFinalizer { @@ -247,7 +250,7 @@ impl SessionFinalizer { /// the whole measurement, so the cost is handed over ready). fn finish(self, interrupted: bool) { let cost = self.observer.end(interrupted); - self.sink.book(std::future::ready(cost)); + self.sink.book(self.held, std::future::ready(cost)); } } @@ -548,6 +551,7 @@ mod tests { frames: weft_core::frames::LoopFrames::default(), service: "bytesvc".into(), origin: weft_core::CredentialOwner::Author, + cancellation: weft_core::cancellation::CancellationFlag::new_arc(), }) } diff --git a/crates/weft-engine/src/test_rig.rs b/crates/weft-engine/src/test_rig.rs index b52d1423..6c092ef6 100644 --- a/crates/weft-engine/src/test_rig.rs +++ b/crates/weft-engine/src/test_rig.rs @@ -304,8 +304,9 @@ impl LiveTestRunner { for handle in handles { failures.extend(handle.close_opened_accesses().await); } - self.clients.open_charges.flush_execution_id(self.execution_id, "the node test ended before the job was read back"); - self.record.costs.wait_zero().await; + self.clients.open_charges + .close_execution_id(self.execution_id, &self.record.costs, "the node test ended before the job was read back") + .await; let ending = weft_journal::ExecEvent::ExecutionCompleted { execution_id: self.execution_id, at_unix: crate::now_unix() }; let ended = match weft_journal::JournalClient::record_event(self.journal.as_ref(), &ending, Some(&self.replica)).await { Ok(()) => self.journal.leave(Leaving::Ended).await, diff --git a/crates/weft-engine/src/worker.rs b/crates/weft-engine/src/worker.rs index 6797de7f..bff6d4f3 100644 --- a/crates/weft-engine/src/worker.rs +++ b/crates/weft-engine/src/worker.rs @@ -374,6 +374,10 @@ impl Worker { () = self.stopping.cancelled() => return Ok(None), registered = self.cancels.claim_once_left(execution_id) => registered, }; + // The claim makes the run this worker's on record, which its lease + // keeps: the registration counts it where the door's tick looks, so + // the tick goes out now, and the claim does not wait for it. + self.door.wake(); let Some(claimed) = self.clients.runs.claim(execution_id).await.context("claim the run")? else { return Ok(Some(RunAnswer::NothingToRun)); }; @@ -1108,12 +1112,15 @@ pub async fn serve(catalog: Arc, clients: EngineClients, config config.secret.clone(), crate::door::RunPermits::from_env()?, ); - door.start_ticks().await?; // The line is held open while runs are driven (the cancel watch) and // while records wait to be written (the writer's own calls); idle, it // closes as any line does, and the door's copy of its triggers reads // the broker until it is back (`weft_task_store::held_copy`). let worker = new_worker(catalog, clients, &config, door.clone()); + // The door ticks while this worker owns anything on record: a run it + // drives, or a record still on its way. + let (cancels, writer) = (worker.cancels.clone(), worker.clients.writer.clone()); + door.start_ticks(Box::new(move || cancels.count() > 0 || !writer.settled())).await?; spawn_cancel_watch(worker.clone()); // What the runs share goes once nothing used it for the idle window. worker.clients.shared.sweep_every(std::time::Duration::from_secs(30)); @@ -1289,6 +1296,13 @@ impl Worker { if self.stopping.is_cancelled() { return Err(Turned::Stopping); } + // Everything the run reads from here on leans on this worker's + // lease: its plan and image (an image prune keeps what a live + // worker may be running, a run its door just bore included, before + // anything of it is on record), then its birth, which makes the run + // this worker's on record. The door woke its tick when it let the + // run in (`Limits::admit`), so the lease is being renewed behind + // this without the run waiting for it. let plan = self.plans.plan(&self.clients, self.door.broker().as_ref(), trigger, entry, instance, triggers_version).await.map_err(Turned::Unread)?; let startable = plan.start.as_ref().map_err(|unready| Turned::Unready(unready.clone()))?; let opening = startable.opening(execution_id, &payload).map_err(|e| Turned::Internal(format!("the run's first state: {e}")))?; diff --git a/crates/weft-infra-supervisor/src/lifecycle.rs b/crates/weft-infra-supervisor/src/lifecycle.rs index 7d96584e..681cbc9b 100644 --- a/crates/weft-infra-supervisor/src/lifecycle.rs +++ b/crates/weft-infra-supervisor/src/lifecycle.rs @@ -366,7 +366,9 @@ async fn run_command( .err() .map(|e| e.downcast_ref::().is_some()) .unwrap_or(false); - let error = result.as_ref().err().map(|e| e.to_string()); + // The whole chain: the outermost context alone ("start the agent of + // unit ..") hides the reason the person needs. + let error = result.as_ref().err().map(|e| format!("{e:#}")); // `command_complete` is Gone if the row was already completed // (remove_node cascade cancelled it) and Displaced if this supervisor no // longer owns the project (drain / lease takeover moved it @@ -393,13 +395,13 @@ async fn run_command( if cancelled { tracing::info!( command_id = cmd.id, - halt = %e, + halt = format!("{e:#}"), "lifecycle command halted by user cancel; marked cancelled" ); } else { tracing::warn!( command_id = cmd.id, - error = %e, + error = format!("{e:#}"), "lifecycle command failed; marked complete with error" ); } diff --git a/crates/weft-journal/src/events.rs b/crates/weft-journal/src/events.rs index 7c4603bb..b1aeea55 100644 --- a/crates/weft-journal/src/events.rs +++ b/crates/weft-journal/src/events.rs @@ -240,10 +240,10 @@ pub enum ExecEvent { node_id: String, frames: LoopFrames, /// Resume cause: - /// - `Some(token)`: the firing was Suspended and a - /// `SuspensionResolved` for `token` arrived (its value is on - /// that row). The fold clears the `suspensions` and - /// `pending_deliveries` entries for `token`. + /// - `Some(token)`: the firing was Suspended and its wait ended + /// by `SuspensionResolved` (the value is on that row) or + /// `SuspensionSkipped`. The fold clears the `suspensions` + /// entry for `token`. /// - `None`: a group boundary the worker was firing when it /// went away, re-fired by the next worker (its state is all /// journaled, so the re-fire is safe; a STEP a dead worker @@ -459,6 +459,17 @@ pub enum ExecEvent { at_unix: u64, }, + /// A person skipped the wait `token` (its `/skip` door): the waiting + /// `ctx` call ends its step skipped, and a replay of its body ends the + /// same call the same way. The wait's other ending, an answer, is + /// `SuspensionResolved`; the first one on record stands. + SuspensionSkipped { + #[serde(skip)] + execution_id: ExecutionId, + token: String, + at_unix: u64, + }, + RunOutput { #[serde(skip)] execution_id: ExecutionId, @@ -750,6 +761,7 @@ macro_rules! execution_id_of { | Self::TriggerCaptured { execution_id, .. } | Self::SuspensionResolved { execution_id, .. } | Self::SuspensionGaveUp { execution_id, .. } + | Self::SuspensionSkipped { execution_id, .. } | Self::RunOutput { execution_id, .. } | Self::CostReported { execution_id, .. } | Self::LogLine { execution_id, .. } @@ -771,6 +783,17 @@ macro_rules! execution_id_of { } impl ExecEvent { + /// How the wait `token` of `execution_id` was answered, as its record + /// holds it: `SuspensionResolved` with the value, or + /// `SuspensionSkipped`. The one spelling every writer of an answer + /// uses. + pub fn wait_answered(execution_id: ExecutionId, token: String, answer: weft_core::primitive::WaitAnswer, at_unix: u64) -> Self { + match answer { + weft_core::primitive::WaitAnswer::Given { value } => Self::SuspensionResolved { execution_id, token, value, at_unix }, + weft_core::primitive::WaitAnswer::Skipped => Self::SuspensionSkipped { execution_id, token, at_unix }, + } + } + /// Whether this event ends the execution: completed, failed, or /// cancelled. The ONE definition of the terminal set; the run's row /// keeps the ending it records (`crate::record::Ending`). @@ -818,6 +841,7 @@ impl ExecEvent { | Self::TriggerCaptured { at_unix, .. } | Self::SuspensionResolved { at_unix, .. } | Self::SuspensionGaveUp { at_unix, .. } + | Self::SuspensionSkipped { at_unix, .. } | Self::RunOutput { at_unix, .. } | Self::CostReported { at_unix, .. } | Self::LogLine { at_unix, .. } @@ -860,6 +884,7 @@ impl ExecEvent { Self::TriggerCaptured { .. } => "trigger_captured", Self::SuspensionResolved { .. } => "suspension_resolved", Self::SuspensionGaveUp { .. } => "suspension_gave_up", + Self::SuspensionSkipped { .. } => "suspension_skipped", Self::RunOutput { .. } => "run_output", Self::CostReported { .. } => "cost_reported", Self::LogLine { .. } => "log_line", @@ -931,6 +956,7 @@ impl ExecEvent { Self::TriggerCaptured { node_id, port_snapshot, .. } => node_id.len() + value_bytes(port_snapshot), Self::SuspensionResolved { token, value, .. } => token.len() + value_bytes(value), Self::SuspensionGaveUp { token, error, .. } => token.len() + error.len(), + Self::SuspensionSkipped { token, .. } => token.len(), Self::RunOutput { node_id, frames: at, name, value, .. } => { node_id.len() + frames(at) + name.len() + value_bytes(value) } diff --git a/crates/weft-journal/src/fold.rs b/crates/weft-journal/src/fold.rs index 1e789025..21fc706d 100644 --- a/crates/weft-journal/src/fold.rs +++ b/crates/weft-journal/src/fold.rs @@ -55,6 +55,15 @@ use weft_core::ExecutionId; use crate::events::ExecEvent; +/// A wait that ended: the firing that waited, the wait's token, and how +/// it ended. +#[derive(Debug, Clone, PartialEq)] +pub struct EndedWait { + pub at: FiringLocation, + pub token: String, + pub end: AwaitEnd, +} + /// What one applied row put in the world, beyond the snapshot: the /// live bridge turns these into the events the editor paints. #[derive(Default)] @@ -65,9 +74,10 @@ pub struct FoldEffects { pub emissions: Vec, /// The group boundaries this row made ready, fired in order. pub boundaries: Vec, - /// On a `NodeResumed` that resumed a suspension: the value the - /// fire delivered (read before the delivery is cleared). - pub resumed_value: Option, + /// The wait this row ended, the first ending on record: an answer, a + /// skip or a give-up row for a registered wait, or the registration + /// of a wait whose ending was already on record. + pub wait_ended: Option, /// On a `LoopOutFired`: the body's `done` vote as the firing read /// it. pub loop_out_done: Option>, @@ -580,7 +590,6 @@ impl Fold { // re-fired after a refold has no token to remove. if let Some(t) = token { self.snap.suspensions.remove(t); - effects.resumed_value = self.snap.pending_deliveries.remove(t); } } ExecEvent::NodeCompleted { node_id, frames, at_unix, .. } => { @@ -765,16 +774,21 @@ impl Fold { // (written while nobody drives the run), so nothing orders // the two, and an answer can be on record first. If the // resolution already landed, `pending_deliveries` holds its - // value; stamp it now so the entry is born resolved. + // value; stamp it now so the entry is born resolved, and + // this row is the one that ended the wait. // Without this, the SuspensionResolved arm found no entry to // mark (not registered yet), the entry lands `ended: // None`, and the await never resumes (permanent hang, fire // consumed). Making the fold order-insensitive for the // Registered/Resolved pair is the right invariant. - let ended = self.snap.pending_deliveries.get(token).map(|value| AwaitEnd::Answered { value: value.clone() }); + let ended = self.snap.pending_deliveries.remove(token); + let at = FiringLocation::new(node_id.clone(), frames.clone()); + if let Some(end) = &ended { + effects.wait_ended = Some(EndedWait { at: at.clone(), token: token.clone(), end: end.clone() }); + } self.snap .awaited_sequences - .entry(FiringLocation::new(node_id.clone(), frames.clone())) + .entry(at) .or_default() .push(AwaitedEntry { call_index: *call_index, @@ -792,18 +806,13 @@ impl Fold { }); } ExecEvent::SuspensionResolved { token, value, .. } => { - // An answer to a wait already given up is ignored: the - // call it would have answered failed, and stays failed. - // One that comes before its registration waits in - // `pending_deliveries`, and there too the first one stands. - if self.end_await(token, AwaitEnd::Answered { value: value.clone() }) { - self.snap.pending_deliveries.entry(token.clone()).or_insert_with(|| value.clone()); - } + effects.wait_ended = self.end_await(token, AwaitEnd::Answered { value: value.clone() }); } + ExecEvent::SuspensionSkipped { token, .. } => effects.wait_ended = self.end_await(token, AwaitEnd::Skipped), // Written by the worker that held the wait, after its // `SuspensionRegistered`, so the entry is always there. ExecEvent::SuspensionGaveUp { token, error, .. } => { - self.end_await(token, AwaitEnd::GaveUp { error: error.clone() }); + effects.wait_ended = self.end_await(token, AwaitEnd::GaveUp { error: error.clone() }); } // A metered call's cost record: the cost of a firing belongs on // its execution record. A record may already be terminal when @@ -1147,9 +1156,12 @@ impl Fold { let Some(record_id) = self.firing_record_id(node_id, frames, CorruptionSite::NodeLifecycle, ev) else { return; }; - // A skip is the whole group's consumption: whatever was left - // pending for its feeds (a stream's items) goes with it. - if status == NodeExecutionStatus::Skipped { + // A skip decided before the body ran is the whole group's + // consumption: whatever was left pending for its feeds (a + // stream's items) goes with it. A body that ran and ended + // skipped (a person skipped what it waited for) ends like one that + // completed: what arrived meanwhile stays for the next firing. + if status == NodeExecutionStatus::Skipped && skip_reason != Some(&SkipReason::WaitSkipped) { let more = self.absorb_pending(node_id, frames, true); if let Some(e) = self.record_mut(node_id, record_id) { for id in more { @@ -1263,25 +1275,30 @@ impl Fold { } } - /// End the wait `token` with `end`, unless it already ended: the first - /// ending on record stands. False when it had, so the caller drops - /// what it was about to apply; true too for a wait not registered yet - /// (an answer may land before its registration). - fn end_await(&mut self, token: &str, end: AwaitEnd) -> bool { - for entries in self.snap.awaited_sequences.values_mut() { + /// End the wait `token` with `end`, unless it already ended: the + /// first ending on record stands, so an answer to a wait already given + /// up is ignored (the call it would have answered failed, and stays + /// failed). Answers the wait it ended, `None` when it had already + /// ended or is not registered yet. An ending that comes before its + /// registration (an answer the dispatcher wrote while nobody drove the + /// run) waits in `pending_deliveries`, and there too the first one + /// stands. + fn end_await(&mut self, token: &str, end: AwaitEnd) -> Option { + for (at, entries) in self.snap.awaited_sequences.iter_mut() { for entry in entries.iter_mut() { if let AwaitedEntryKind::Await { token: t, ended } = &mut entry.kind { if t == token { if ended.is_some() { - return false; + return None; } - *ended = Some(end); - return true; + *ended = Some(end.clone()); + return Some(EndedWait { at: at.clone(), token: token.to_string(), end }); } } } } - true + self.snap.pending_deliveries.entry(token.to_string()).or_insert(end); + None } /// The record a row about `(node, frames)` is about: the same @@ -2490,8 +2507,8 @@ mod tests { } /// Suspend, resolve, resume, complete: one record the whole way, - /// the resume clears the suspension and hands the value to the - /// bridge. + /// the answer ends the wait for the bridge, and the resume clears the + /// suspension. #[test] fn lifecycle_one_record_per_frames() { let token = "tok-1".to_string(); @@ -2515,12 +2532,9 @@ mod tests { completed("a", vec![], 0), ]; let mut fold = Fold::new(execution_id(), fan_out_project()); - let mut resumed = None; + let mut ended = Vec::new(); for ev in &events { - let effects = fold.apply(ev); - if let Some(v) = effects.resumed_value { - resumed = Some(v); - } + ended.extend(fold.apply(ev).wait_ended); } let snap = fold.into_snapshot(); assert!(snap.corruptions.is_empty(), "{:?}", snap.corruptions); @@ -2529,7 +2543,7 @@ mod tests { assert_eq!(execs[0].status, NodeExecutionStatus::Completed); assert!(snap.suspensions.is_empty(), "suspensions cleared after resume+complete"); assert!(snap.pending_deliveries.is_empty()); - assert_eq!(resumed, Some(json!("approved"))); + assert_eq!(ended, vec![EndedWait { at: FiringLocation::new("a", vec![]), token, end: AwaitEnd::Answered { value: json!("approved") } }], "the answer ends the wait, once"); let seq = &snap.awaited_sequences[&FiringLocation::new("a", vec![])]; assert!(matches!(&seq[0].kind, AwaitedEntryKind::Await { ended: Some(AwaitEnd::Answered { value }), .. } if value == &json!("approved"))); } @@ -2625,7 +2639,7 @@ mod tests { #[test] fn suspension_resolved_before_registered_still_resolves() { - let events = vec![ + let events = [ ExecEvent::SuspensionResolved { execution_id: execution_id(), token: "t".into(), value: json!(5), at_unix: 0 }, ExecEvent::SuspensionRegistered { execution_id: execution_id(), @@ -2637,16 +2651,21 @@ mod tests { at_unix: 0, }, ]; - let snap = fold_to_snapshot(execution_id(), fan_out_project(), &events); + let mut fold = Fold::new(execution_id(), fan_out_project()); + let ended: Vec> = events.iter().map(|ev| fold.apply(ev).wait_ended).collect(); + assert_eq!(ended, vec![None, Some(EndedWait { at: FiringLocation::new("a", vec![]), token: "t".into(), end: AwaitEnd::Answered { value: json!(5) } })], + "an answer on record before its wait ends the wait at the registration"); + let snap = fold.into_snapshot(); let seq = &snap.awaited_sequences[&FiringLocation::new("a", vec![])]; assert!(matches!(&seq[0].kind, AwaitedEntryKind::Await { ended: Some(AwaitEnd::Answered { value }), .. } if value == &json!(5))); + assert!(snap.pending_deliveries.is_empty(), "the registration took the answer"); } /// A wait given up stays given up: the answer that comes after it is /// ignored, so a replay fails the call the way it failed live. #[test] fn an_answer_after_a_wait_was_given_up_is_ignored() { - let events = vec![ + let events = [ ExecEvent::SuspensionRegistered { execution_id: execution_id(), node_id: "a".into(), @@ -2659,7 +2678,11 @@ mod tests { ExecEvent::SuspensionGaveUp { execution_id: execution_id(), token: "t".into(), error: "gave up".into(), at_unix: 0 }, ExecEvent::SuspensionResolved { execution_id: execution_id(), token: "t".into(), value: json!(5), at_unix: 0 }, ]; - let snap = fold_to_snapshot(execution_id(), fan_out_project(), &events); + let mut fold = Fold::new(execution_id(), fan_out_project()); + let ended: Vec> = events.iter().map(|ev| fold.apply(ev).wait_ended).collect(); + assert_eq!(ended, vec![None, Some(EndedWait { at: FiringLocation::new("a", vec![]), token: "t".into(), end: AwaitEnd::GaveUp { error: "gave up".into() } }), None], + "the give-up ends the wait; the late answer ends nothing"); + let snap = fold.into_snapshot(); let seq = &snap.awaited_sequences[&FiringLocation::new("a", vec![])]; assert!(matches!(&seq[0].kind, AwaitedEntryKind::Await { ended: Some(AwaitEnd::GaveUp { error }), .. } if error == "gave up")); assert!(snap.pending_deliveries.is_empty(), "nothing is left to deliver"); diff --git a/crates/weft-journal/src/lib.rs b/crates/weft-journal/src/lib.rs index 8e3ab5c4..e84c635a 100644 --- a/crates/weft-journal/src/lib.rs +++ b/crates/weft-journal/src/lib.rs @@ -18,7 +18,7 @@ pub mod traits; pub mod unrecorded; pub use events::{redacted, ExecEvent, Seed}; -pub use fold::{fold_to_snapshot, FiringView, Fold, FoldEffects}; +pub use fold::{fold_to_snapshot, EndedWait, FiringView, Fold, FoldEffects}; pub use seed::{fold_seeded, seed_chain, LiveFold, SeedChain}; pub use traits::{BatchError, JournalClient, JournalRow, NoopJournal, Position, RecordClient}; diff --git a/crates/weft-journal/src/record.rs b/crates/weft-journal/src/record.rs index f1d6d168..144374b2 100644 --- a/crates/weft-journal/src/record.rs +++ b/crates/weft-journal/src/record.rs @@ -184,7 +184,9 @@ impl Written { match event { ExecEvent::CostReported { amount_usd: Some(usd), .. } => self.cost_micro_usd += micro_usd(*usd), ExecEvent::NodeSkipped { .. } => self.skipped += 1, - ExecEvent::SuspensionResolved { token, .. } => self.resolved.push(token.clone()), + ExecEvent::SuspensionResolved { token, .. } | ExecEvent::SuspensionSkipped { token, .. } => { + self.resolved.push(token.clone()) + } _ => {} } if self.ended.is_none() { @@ -670,7 +672,8 @@ pub async fn append_locked_in( /// Write the answers handed to `execution_id` while a worker drove it /// (`weft_task_store::parked_fires::hand_answer_in`) into its record, as -/// `SuspensionResolved` at `last_seq + 1`, and queue it for a worker to +/// `SuspensionResolved` (or `SuspensionSkipped` for a skip) at +/// `last_seq + 1`, and queue it for a worker to /// carry on, on the caller's transaction with its row locked as `locked` /// and no owner left (its worker just let go of it, or went away). Answers /// whether any were waiting; with none, nothing is written. @@ -682,7 +685,7 @@ pub async fn resolve_handed_in(conn: &mut PgConnection, execution_id: ExecutionI let at_unix = unix_now() as u64; let resolved: Vec = handed .into_iter() - .map(|(token, value)| ExecEvent::SuspensionResolved { execution_id, token, value, at_unix }) + .map(|(token, answer)| ExecEvent::wait_answered(execution_id, token, answer, at_unix)) .collect(); match append_locked_in(conn, execution_id, locked, &resolved, writer, Then::Queued).await? { Appended::At(_) => Ok(true), diff --git a/crates/weft-platform-gcp/src/images.rs b/crates/weft-platform-gcp/src/images.rs index 0193b3d7..ef6c40e6 100644 --- a/crates/weft-platform-gcp/src/images.rs +++ b/crates/weft-platform-gcp/src/images.rs @@ -157,6 +157,8 @@ impl ImageBuilder for CloudBuildImages { format!("{}/{tag}", self.gcp.artifact_registry.trim_end_matches('/')) } + /// Uploads the whole context before it answers: Cloud Build reads the + /// bucket's copy, so `req.staging` goes when this returns. async fn start(&self, req: BuildRequest) -> anyhow::Result { let dir = req.context_dir.clone(); let bytes = tokio::task::spawn_blocking(move || tar_context(&dir)).await??; @@ -270,6 +272,7 @@ mod tests { project_id: uuid::Uuid::from_u128(1), tenant: "local".into(), context_dir: "/ctx".into(), + staging: weft_platform_traits::Staging::new(()), image_ref: "us-central1-docker.pkg.dev/acme/weft/weft-worker:abc".into(), build_args: vec![("WEFT_COMPILE_LANE".into(), "0".into())], }; diff --git a/crates/weft-platform-local/src/images.rs b/crates/weft-platform-local/src/images.rs index 49f30071..b8cac3e2 100644 --- a/crates/weft-platform-local/src/images.rs +++ b/crates/weft-platform-local/src/images.rs @@ -98,24 +98,33 @@ impl ImageBuilder for DockerImageBuilder { /// runs `docker build` otherwise. async fn start(&self, req: BuildRequest) -> anyhow::Result { let name = req.name.clone(); - let args = build_args(&req); + let image_ref = req.image_ref.clone(); let docker = self.docker.clone(); let builds = self.builds.clone(); let task_name = name.clone(); + // An image another install on this daemon holds is taken as it is, + // with nothing built, and its staging let go of at once. + if adopt(docker.as_ref(), &image_ref).await? { + self.builds.lock().insert(name.clone(), Build::Ended(BuildStatus::Succeeded)); + return Ok(BuildHandle::named(name)); + } + let args = build_args(&req); + // `docker build` reads the context while it runs, after the + // request that staged it answered: the build task keeps the + // staging until docker is done with it, and an abort (`release`) + // drops it with the task; either way the directory goes on a + // blocking thread (`Staging`). + let staging = req.staging; // Held until the build is registered, so a fast build never // records its end before its start. let mut table = self.builds.lock(); - let image_ref = req.image_ref.clone(); let task = tokio::spawn(async move { - let status = match adopt(docker.as_ref(), &image_ref).await { - Ok(true) => BuildStatus::Succeeded, - Ok(false) => match docker.exec(&args).await { - Ok(out) if out.success() => BuildStatus::Succeeded, - Ok(out) => BuildStatus::Failed { reason: tail(&format!("{}\n{}", out.stdout, out.stderr), FAILURE_LOG_LINES) }, - Err(e) => BuildStatus::Failed { reason: format!("{e:#}") }, - }, + let status = match docker.exec(&args).await { + Ok(out) if out.success() => BuildStatus::Succeeded, + Ok(out) => BuildStatus::Failed { reason: tail(&format!("{}\n{}", out.stdout, out.stderr), FAILURE_LOG_LINES) }, Err(e) => BuildStatus::Failed { reason: format!("{e:#}") }, }; + drop(staging); if let Some(entry) = builds.lock().get_mut(&task_name) { *entry = Build::Ended(status); } @@ -192,13 +201,32 @@ pub async fn bound_build_cache(docker: &dyn Docker) -> anyhow::Result<()> { mod tests { use super::*; use crate::docker::fake::FakeDocker; + use weft_platform_traits::Staging; + + /// A staged directory, removed when dropped, as the dispatcher's + /// staging temp dir is. + struct Staged(std::path::PathBuf); + impl Drop for Staged { + fn drop(&mut self) { + std::fs::remove_dir_all(&self.0).unwrap(); + } + } + + /// A request for a staged context (a Dockerfile and a file it copies) + /// that the request holds only through its staging, as a build request + /// that answered does. fn req(name: &str) -> BuildRequest { + let dir = std::env::temp_dir().join(format!("weft-staged-{}", uuid::Uuid::new_v4().simple())); + std::fs::create_dir_all(dir.join("src")).unwrap(); + std::fs::write(dir.join("Dockerfile"), "FROM scratch\n").unwrap(); + std::fs::write(dir.join("src/main.rs"), "fn main() {}\n").unwrap(); BuildRequest { name: name.into(), project_id: uuid::Uuid::from_u128(1), tenant: "local".into(), - context_dir: "/ctx/a".into(), + context_dir: dir.clone(), + staging: Staging::new(Staged(dir)), image_ref: "weft-worker:abc".into(), build_args: vec![("WEFT_COMPILE_LANE".into(), "2".into())], } @@ -213,18 +241,65 @@ mod tests { } } + /// Wait for the staged directory to go, which happens on a blocking + /// thread once the last holder lets go. + async fn gone(dir: &std::path::Path) { + for _ in 0..500 { + if !dir.exists() { + return; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + panic!("the staging {} was never let go of", dir.display()); + } + + /// A Docker whose `docker build` runs until the test lets it end. + struct HeldBuilds { + docker: FakeDocker, + ends: tokio::sync::Semaphore, + } + + #[async_trait] + impl Docker for HeldBuilds { + async fn exec(&self, args: &[String]) -> anyhow::Result { + let out = self.docker.exec(args).await; + if args.first().is_some_and(|verb| verb == "build") { + self.ends.acquire().await.unwrap().forget(); + } + out + } + } + + /// `docker build` reads the staged context itself, untouched, and the + /// build keeps it on disk while it runs, letting go once it ends. #[tokio::test] - async fn a_build_runs_docker_build_and_reports_how_it_ended() { - let docker = Arc::new(FakeDocker::new()); + async fn a_build_runs_docker_build_on_the_staging_it_keeps() { + let docker = Arc::new(HeldBuilds { docker: FakeDocker::new(), ends: tokio::sync::Semaphore::new(0) }); let b = DockerImageBuilder::new(docker.clone(), weft_core::infra::Install::default_install()); - let h = b.start(req("b1")).await.unwrap(); - assert_eq!(settled(&b, &h).await, BuildStatus::Succeeded); + let first = req("b1"); + let dir = first.context_dir.clone(); + let h = b.start(first).await.unwrap(); + for _ in 0..10 { + tokio::task::yield_now().await; + } + assert_eq!(b.poll(&h).await.unwrap(), BuildStatus::Pending); + assert!(dir.join("src/main.rs").exists(), "the staging stays while the build runs"); + let call = &docker.docker.calls_to("build")[0]; assert_eq!( - docker.calls_to("build")[0], - vec!["build", "--tag", "weft-worker:abc", "--build-arg", "WEFT_COMPILE_LANE=2", "--file", "/ctx/a/Dockerfile", "/ctx/a"] + call, + &["build", "--tag", "weft-worker:abc", "--build-arg", "WEFT_COMPILE_LANE=2", "--file"] + .into_iter() + .map(String::from) + .chain([dir.join("Dockerfile").display().to_string(), dir.display().to_string()]) + .collect::>(), + "built from the staged context itself" ); + docker.ends.add_permits(1); + assert_eq!(settled(&b, &h).await, BuildStatus::Succeeded); + gone(&dir).await; - docker.fail(&["build"], "error: cargo failed"); + docker.docker.fail(&["build"], "error: cargo failed"); + docker.ends.add_permits(1); let h = b.start(req("b2")).await.unwrap(); assert!(matches!(settled(&b, &h).await, BuildStatus::Failed { reason } if reason.contains("cargo failed"))); @@ -233,6 +308,21 @@ mod tests { assert_eq!(b.poll(&BuildHandle::named("never")).await.unwrap(), BuildStatus::Gone); } + /// A build stopped while it runs lets go of its staging with it. + #[tokio::test] + async fn a_stopped_build_lets_go_of_its_staging() { + let docker = Arc::new(HeldBuilds { docker: FakeDocker::new(), ends: tokio::sync::Semaphore::new(0) }); + let b = DockerImageBuilder::new(docker, weft_core::infra::Install::default_install()); + let r = req("b1"); + let dir = r.context_dir.clone(); + let h = b.start(r).await.unwrap(); + tokio::task::yield_now().await; + assert!(dir.exists(), "kept while it runs"); + b.release(&h).await; + gone(&dir).await; + assert_eq!(b.poll(&h).await.unwrap(), BuildStatus::Gone); + } + /// A content another install already holds is given this install's /// name instead of being built; with nobody holding it, it is built. #[tokio::test] @@ -243,10 +333,12 @@ mod tests { let mut r = req("b1"); r.image_ref = b.image_ref("weft-worker:abc"); assert_eq!(r.image_ref, "localhost/weft-cell7/weft-worker:abc"); + let dir = r.context_dir.clone(); let h = b.start(r.clone()).await.unwrap(); assert_eq!(settled(&b, &h).await, BuildStatus::Succeeded); assert_eq!(docker.calls_to("tag"), vec![vec!["tag", "weft-worker:abc", "localhost/weft-cell7/weft-worker:abc"]]); assert!(docker.calls_to("build").is_empty(), "nothing built"); + assert!(dir.exists(), "the test's own clone still holds it"); let docker = Arc::new(FakeDocker::new()); docker.answer(&["image", "ls"], "evil/weft-worker:abc\n"); @@ -255,6 +347,7 @@ mod tests { assert_eq!(settled(&b, &h).await, BuildStatus::Succeeded); assert!(docker.calls_to("tag").is_empty(), "an image no install holds is never taken"); assert_eq!(docker.calls_to("build").len(), 1); + gone(&dir).await; } /// The bound caps every record, least recently used first: `--all` diff --git a/crates/weft-platform-traits/src/images.rs b/crates/weft-platform-traits/src/images.rs index 62714299..c51409d6 100644 --- a/crates/weft-platform-traits/src/images.rs +++ b/crates/weft-platform-traits/src/images.rs @@ -36,7 +36,13 @@ pub trait ImageBuilder: Send + Sync { fn image_ref(&self, tag: &str) -> String; /// Launch the build of a staged context, producing `req.image_ref`. - /// Returns as soon as it runs; the build goes on on its own. + /// Returns as soon as it runs; the build goes on on its own. The + /// context stays on disk exactly as long as somebody holds + /// `req.staging`: the request that staged it answers without waiting + /// for the build and lets go of its own, so a builder that reads the + /// context after this returns keeps `req.staging` until it is done + /// reading (the build ended, or was stopped), and one that read it + /// all here lets go of it here. async fn start(&self, req: BuildRequest) -> anyhow::Result; /// How `handle`'s build is doing. Answers from the name alone, so a @@ -67,11 +73,52 @@ pub struct BuildRequest { pub project_id: uuid::Uuid, pub tenant: String, pub context_dir: PathBuf, + /// What keeps `context_dir` on disk ([`Staging`]). + pub staging: Staging, pub image_ref: String, /// `ARG`s the Dockerfile declares (the worker's compile lane). pub build_args: Vec<(String, String)>, } +/// What keeps a staged build context on disk: the context exists while +/// any clone of this is held, and goes with the last one. Built from +/// whatever owns the directory (the staging request's temp dir), so the +/// files are never copied and keep the times they were staged with, which +/// a compile cache relies on. +/// +/// Dropping one never blocks the async runtime: the last one lets go of +/// the directory on a blocking thread when a runtime is there to run it. +#[derive(Clone)] +pub struct Staging { + _owner: std::sync::Arc, +} + +impl Staging { + /// The context exists for as long as `owner` lives. + pub fn new(owner: impl std::any::Any + Send + Sync) -> Self { + Self { _owner: std::sync::Arc::new(OffThread(Some(owner))) } + } +} + +impl std::fmt::Debug for Staging { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("Staging") + } +} + +/// A staging owner, dropped (by the last [`Staging`] clone) on a blocking +/// thread when a runtime is there to run it. +struct OffThread(Option); + +impl Drop for OffThread { + fn drop(&mut self) { + let Some(owner) = self.0.take() else { return }; + if let Ok(runtime) = tokio::runtime::Handle::try_current() { + runtime.spawn_blocking(move || drop(owner)); + } + } +} + /// The builder's own id for a build, persisted on the `image_build` row /// so a poll survives a restart. The builder may choose it (Cloud Build /// names its builds itself), so it is the one [`ImageBuilder::start`] @@ -258,6 +305,7 @@ mod tests { project_id: uuid::Uuid::from_u128(0x100), tenant: "t".into(), context_dir: PathBuf::from("/tmp/ctx"), + staging: Staging::new(()), image_ref: "reg/weft-worker:x".into(), build_args: Vec::new(), }) diff --git a/crates/weft-platform-traits/src/lib.rs b/crates/weft-platform-traits/src/lib.rs index 63d5258a..1fdcbc86 100644 --- a/crates/weft-platform-traits/src/lib.rs +++ b/crates/weft-platform-traits/src/lib.rs @@ -46,7 +46,7 @@ pub use identity::{CallerIdentity, FixedToken, IdentityRefused, IdentityTokens, pub use domains::DomainHosting; pub use frontends::{FrontendHosting, FrontendSite, HostedFrontend}; pub use holder_pool::HolderPool; -pub use images::{BuildHandle, BuildRequest, BuildStatus, ImageBuilder, ImageDeleted}; +pub use images::{BuildHandle, BuildRequest, BuildStatus, ImageBuilder, ImageDeleted, Staging}; pub use infra_host::{EndpointAt, InfraHost, UnitObservation, UnitRunState}; pub use object_store::{ object_store_for, ObjectEntry, ObjectStore, ObjectStoreConfig, PresignAudience, diff --git a/crates/weft-task-store/migrations/image_build/20261010T093458_image_build_version_build.sql b/crates/weft-task-store/migrations/image_build/20261010T093458_image_build_version_build.sql new file mode 100644 index 00000000..ff343962 --- /dev/null +++ b/crates/weft-task-store/migrations/image_build/20261010T093458_image_build_version_build.sql @@ -0,0 +1,41 @@ +CREATE TABLE version_build ( + asked_at bigint NOT NULL, + ended_at bigint, + id uuid NOT NULL, + images text[] NOT NULL, + manifest jsonb NOT NULL, + program jsonb NOT NULL, + project_id uuid NOT NULL, + project_name text NOT NULL, + reason text, + registered jsonb, + state text NOT NULL, + tenant_id text NOT NULL, + waits_on jsonb NOT NULL +); + +ALTER TABLE version_build ADD CONSTRAINT version_build_asked_at_not_null NOT NULL asked_at; + +ALTER TABLE version_build ADD CONSTRAINT version_build_id_not_null NOT NULL id; + +ALTER TABLE version_build ADD CONSTRAINT version_build_images_not_null NOT NULL images; + +ALTER TABLE version_build ADD CONSTRAINT version_build_manifest_not_null NOT NULL manifest; + +ALTER TABLE version_build ADD CONSTRAINT version_build_pkey PRIMARY KEY (id); + +ALTER TABLE version_build ADD CONSTRAINT version_build_program_not_null NOT NULL program; + +ALTER TABLE version_build ADD CONSTRAINT version_build_project_id_not_null NOT NULL project_id; + +ALTER TABLE version_build ADD CONSTRAINT version_build_project_name_not_null NOT NULL project_name; + +ALTER TABLE version_build ADD CONSTRAINT version_build_state_check CHECK ((state = ANY (ARRAY['waiting'::text, 'registered'::text, 'failed'::text, 'cancelled'::text, 'superseded'::text]))); + +ALTER TABLE version_build ADD CONSTRAINT version_build_state_not_null NOT NULL state; + +ALTER TABLE version_build ADD CONSTRAINT version_build_tenant_id_not_null NOT NULL tenant_id; + +ALTER TABLE version_build ADD CONSTRAINT version_build_waits_on_not_null NOT NULL waits_on; + +CREATE UNIQUE INDEX version_build_one_waiting ON version_build USING btree (project_id) WHERE (state = 'waiting'::text); diff --git a/crates/weft-task-store/migrations/image_build/20261010T100248_image_build_build_ask_order.sql b/crates/weft-task-store/migrations/image_build/20261010T100248_image_build_build_ask_order.sql new file mode 100644 index 00000000..64cf35d9 --- /dev/null +++ b/crates/weft-task-store/migrations/image_build/20261010T100248_image_build_build_ask_order.sql @@ -0,0 +1,4 @@ +-- Fails if the table holds any row: add it nullable, backfill, then SET NOT NULL. +ALTER TABLE version_build ADD COLUMN ask bigint NOT NULL; + +ALTER TABLE version_build ADD CONSTRAINT version_build_ask_not_null NOT NULL ask; diff --git a/crates/weft-task-store/migrations/parked_fire/20261009T114508_parked_fire_skipped_answers.sql b/crates/weft-task-store/migrations/parked_fire/20261009T114508_parked_fire_skipped_answers.sql new file mode 100644 index 00000000..26cdd6ea --- /dev/null +++ b/crates/weft-task-store/migrations/parked_fire/20261009T114508_parked_fire_skipped_answers.sql @@ -0,0 +1,3 @@ +ALTER TABLE parked_fire ADD COLUMN skipped boolean DEFAULT false NOT NULL; + +ALTER TABLE parked_fire ADD CONSTRAINT parked_fire_skipped_not_null NOT NULL skipped; diff --git a/crates/weft-task-store/migrations/project/20261010T100248_project_build_ask_order.sql b/crates/weft-task-store/migrations/project/20261010T100248_project_build_ask_order.sql new file mode 100644 index 00000000..ba93015e --- /dev/null +++ b/crates/weft-task-store/migrations/project/20261010T100248_project_build_ask_order.sql @@ -0,0 +1,7 @@ +ALTER TABLE project ADD COLUMN build_asks bigint DEFAULT 0 NOT NULL; + +ALTER TABLE project ADD COLUMN registered_ask bigint DEFAULT 0 NOT NULL; + +ALTER TABLE project ADD CONSTRAINT project_build_asks_not_null NOT NULL build_asks; + +ALTER TABLE project ADD CONSTRAINT project_registered_ask_not_null NOT NULL registered_ask; diff --git a/crates/weft-task-store/src/db.rs b/crates/weft-task-store/src/db.rs index 054003fc..a804a038 100644 --- a/crates/weft-task-store/src/db.rs +++ b/crates/weft-task-store/src/db.rs @@ -34,6 +34,23 @@ pub const RECORD_POOL_CONNECTIONS: u32 = 8; /// broker, so its batches are counted apart from other brokers'. pub const RECORD_POOL_APPLICATION: &str = "weft-records"; +/// Whether a database error means the database could not be reached or +/// is restarting, rather than that the statement itself failed: no +/// connection to be had, a connection that broke, or the server refusing +/// because it is shutting down or starting up (SQLSTATE class 08, and +/// 57P01 to 57P03). Only that is worth asking again on; anything else +/// fails the same way next time. The one reading, for every service that +/// answers a caller who can wait. +pub fn unreachable(e: &sqlx::Error) -> bool { + match e { + sqlx::Error::PoolTimedOut | sqlx::Error::PoolClosed | sqlx::Error::Io(_) | sqlx::Error::Protocol(_) => true, + sqlx::Error::Database(db) => db + .code() + .is_some_and(|code| code.starts_with("08") || matches!(&*code, "57P01" | "57P02" | "57P03")), + _ => false, + } +} + /// A pool on `url` of at most `max_connections`, each request for a /// connection waiting at most `acquire_timeout`. The database may still be /// starting: it is tried again for a minute before the error is answered. diff --git a/crates/weft-task-store/src/parked_fires.rs b/crates/weft-task-store/src/parked_fires.rs index 3f78953d..b3f7eb30 100644 --- a/crates/weft-task-store/src/parked_fires.rs +++ b/crates/weft-task-store/src/parked_fires.rs @@ -50,6 +50,9 @@ pub static GROUP: crate::SchemaGroup = crate::SchemaGroup { instance_gap JSONB, -- An answer to a waiting run, rather than an event of an entry. is_resume BOOLEAN NOT NULL, + -- An answer that is a person skipping the wait (its `/skip` + -- door), not a value: `payload` is then null and unread. + skipped BOOLEAN NOT NULL DEFAULT FALSE, -- Set on an answer already taken off its wait (the wait's -- signal is gone) for a run a worker drives: the run it is -- for. Its worker takes it; if the worker lets go of the run @@ -121,6 +124,33 @@ pub struct Waiting { /// change of values routes it again. Shown per trigger by `weft status` /// and `ctx.instances().list()`. pub instance_gap: Option, + /// An answer (`is_resume`) that is a person skipping the wait, not a + /// value: `payload` is then null and unread. Always false for an + /// entry's event. + pub skipped: bool, +} + +impl Waiting { + /// The answer this is, when it answers a waiting run. + pub fn answer(&self) -> weft_core::primitive::WaitAnswer { + stored_answer(self.payload.clone(), self.skipped) + } +} + +/// An answer as a row holds it (`payload`, `skipped`) and back: the one +/// spelling of a skip in this table. +fn answer_row(answer: &weft_core::primitive::WaitAnswer) -> (Value, bool) { + match answer { + weft_core::primitive::WaitAnswer::Given { value } => (value.clone(), false), + weft_core::primitive::WaitAnswer::Skipped => (Value::Null, true), + } +} + +fn stored_answer(payload: Value, skipped: bool) -> weft_core::primitive::WaitAnswer { + match skipped { + true => weft_core::primitive::WaitAnswer::Skipped, + false => weft_core::primitive::WaitAnswer::Given { value: payload }, + } } /// Seconds an event waits before its `attempts`-th retry: 1, 2, 4, ... @@ -145,7 +175,14 @@ pub const MAX_PARKED_ENTRY_FIRES: i64 = 1000; /// `attempts`-th failure, retried after its backoff. pub fn waiting(fire_id: uuid::Uuid, payload: Value, caller: Option, attempts: u32, instance_gap: Option) -> Waiting { let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_secs() as i64); - Waiting { fire_id, payload, caller, attempts, not_before: now + park_backoff_secs(attempts), instance_gap } + Waiting { fire_id, payload, caller, attempts, not_before: now + park_backoff_secs(attempts), instance_gap, skipped: false } +} + +/// An answer to a waiting run, as it waits until the run can take it: +/// [`waiting`] for a value, or a person's skip. +pub fn waiting_answer(fire_id: uuid::Uuid, answer: weft_core::primitive::WaitAnswer) -> Waiting { + let (payload, skipped) = answer_row(&answer); + Waiting { skipped, ..waiting(fire_id, payload, None, 0, None) } } /// THE one park. Every park site (an answer whose run cannot take it now, a @@ -188,8 +225,8 @@ pub async fn park(pool: &sqlx::PgPool, token: &str, waiting: &Waiting, held_by: return Ok(ParkAppend::Refused(ParkRefusal::QueueFull)); } let inserted = sqlx::query( - "INSERT INTO parked_fire (token, fire_id, payload, caller, attempts, not_before, instance_gap, is_resume) \ - VALUES ($1, $2, $3, $4, $5, $6, $7, $8) ON CONFLICT (fire_id) DO NOTHING", + "INSERT INTO parked_fire (token, fire_id, payload, caller, attempts, not_before, instance_gap, is_resume, skipped) \ + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) ON CONFLICT (fire_id) DO NOTHING", ) .bind(token) .bind(waiting.fire_id) @@ -199,6 +236,7 @@ pub async fn park(pool: &sqlx::PgPool, token: &str, waiting: &Waiting, held_by: .bind(waiting.not_before) .bind(waiting.instance_gap.as_ref().map(sqlx::types::Json)) .bind(is_resume) + .bind(waiting.skipped) .execute(&mut *tx) .await?; tx.commit().await?; @@ -252,7 +290,7 @@ pub struct Head { /// overtake each other. pub async fn take_head(tx: &mut sqlx::PgConnection, token: &str, now: i64) -> anyhow::Result> { let row: Option = sqlx::query_as( - "SELECT token, seq, fire_id, payload, caller, attempts, not_before, instance_gap, is_resume FROM parked_fire p \ + "SELECT token, seq, fire_id, payload, caller, attempts, not_before, instance_gap, is_resume, skipped FROM parked_fire p \ WHERE p.token = $1 AND p.seq = (SELECT min(q.seq) FROM parked_fire q WHERE q.token = $1) \ AND p.not_before <= $2 AND p.instance_gap IS NULL \ FOR UPDATE SKIP LOCKED", @@ -268,7 +306,7 @@ pub async fn take_head(tx: &mut sqlx::PgConnection, token: &str, now: i64) -> an /// again, whatever their backoff: the same as [`take_head`] but due now. pub async fn take_head_now(tx: &mut sqlx::PgConnection, token: &str) -> anyhow::Result> { let row: Option = sqlx::query_as( - "SELECT token, seq, fire_id, payload, caller, attempts, not_before, instance_gap, is_resume FROM parked_fire p \ + "SELECT token, seq, fire_id, payload, caller, attempts, not_before, instance_gap, is_resume, skipped FROM parked_fire p \ WHERE p.token = $1 AND p.seq = (SELECT min(q.seq) FROM parked_fire q WHERE q.token = $1) \ FOR UPDATE SKIP LOCKED", ) @@ -303,20 +341,22 @@ pub async fn restamp_in(tx: &mut sqlx::PgConnection, head: &Head, attempts: u32, Ok(()) } -/// Hand `value`, the answer to the wait `token` of run `execution_id`, +/// Hand `answer`, the answer to the wait `token` of run `execution_id`, /// which a worker drives, to that worker, on the caller's transaction: /// nobody but a run's worker writes the record of a run it drives. The /// wait's signal is consumed by the caller in the same transaction, so it /// is answered once. -pub async fn hand_answer_in(tx: &mut sqlx::PgConnection, token: &str, execution_id: weft_core::ExecutionId, value: &Value) -> anyhow::Result<()> { +pub async fn hand_answer_in(tx: &mut sqlx::PgConnection, token: &str, execution_id: weft_core::ExecutionId, answer: &weft_core::primitive::WaitAnswer) -> anyhow::Result<()> { + let (value, skipped) = answer_row(answer); sqlx::query( - "INSERT INTO parked_fire (token, fire_id, payload, attempts, not_before, is_resume, execution_id) \ - VALUES ($1, $2, $3, 0, 0, TRUE, $4)", + "INSERT INTO parked_fire (token, fire_id, payload, attempts, not_before, is_resume, execution_id, skipped) \ + VALUES ($1, $2, $3, 0, 0, TRUE, $4, $5)", ) .bind(token) .bind(uuid::Uuid::new_v4()) .bind(value) .bind(execution_id) + .bind(skipped) .execute(&mut *tx) .await?; Ok(()) @@ -326,11 +366,12 @@ pub async fn hand_answer_in(tx: &mut sqlx::PgConnection, token: &str, execution_ /// first: what the worker driving the run takes. They go once the run's /// record holds them (`weft_record_batch` deletes the answers a batch /// resolves). -pub async fn answers_for<'e, E: sqlx::PgExecutor<'e>>(executor: E, execution_id: weft_core::ExecutionId) -> anyhow::Result> { - Ok(sqlx::query_as("SELECT token, payload FROM parked_fire WHERE execution_id = $1 ORDER BY seq") +pub async fn answers_for<'e, E: sqlx::PgExecutor<'e>>(executor: E, execution_id: weft_core::ExecutionId) -> anyhow::Result> { + let rows: Vec<(String, Value, bool)> = sqlx::query_as("SELECT token, payload, skipped FROM parked_fire WHERE execution_id = $1 ORDER BY seq") .bind(execution_id) .fetch_all(executor) - .await?) + .await?; + Ok(rows.into_iter().map(|(token, payload, skipped)| (token, stored_answer(payload, skipped))).collect()) } #[derive(sqlx::FromRow)] @@ -344,6 +385,7 @@ struct HeadRow { not_before: i64, instance_gap: Option>, is_resume: bool, + skipped: bool, } impl HeadRow { @@ -358,6 +400,7 @@ impl HeadRow { attempts: self.attempts.max(0) as u32, not_before: self.not_before, instance_gap: self.instance_gap.map(|gap| gap.0), + skipped: self.skipped, }, is_resume: self.is_resume, } diff --git a/crates/weft-task-store/src/worker_door.rs b/crates/weft-task-store/src/worker_door.rs index 50537cbf..e249ab98 100644 --- a/crates/weft-task-store/src/worker_door.rs +++ b/crates/weft-task-store/src/worker_door.rs @@ -1,10 +1,12 @@ //! Which worker processes are alive (`worker_lease`), the image each runs, //! what each drives right now per trigger (`worker_lease.in_flight`), and what each counted //! at its door this minute (`door_count`), all written once a second by -//! every worker through the broker (`/v1/door/tick`). A worker's live -//! lease is half of what says a run is being worked on -//! ([`crate::in_flight_sql!`]), and what it drives is what a drain waits -//! for. +//! every worker that has work under way, through the broker +//! (`/v1/door/tick`). A worker with nothing under way stops ticking after +//! one last tick that says so, and its lease lapses: it owns no run then, +//! and it ticks again before it takes one. A worker's live lease is half +//! of what says a run is being worked on ([`crate::in_flight_sql!`]), and +//! what it drives is what a drain waits for. /// How long a worker's lease (`worker_lease`) holds without a tick, at /// this install's pace: a few missed ticks, so a busy second never reads diff --git a/docs/src/build/a-person-in-the-loop.md b/docs/src/build/a-person-in-the-loop.md index 475515d2..8bff1ede 100644 --- a/docs/src/build/a-person-in-the-loop.md +++ b/docs/src/build/a-person-in-the-loop.md @@ -61,9 +61,11 @@ and the buttons or boxes your other fields asked for. The primary button says **Submit** for a question inside a running program, and **Fire** for a trigger that starts one. -Beside it, **Skip** answers with nothing and lets the run carry on, and -**Cancel run** ends the whole execution. Neither shows up on a trigger, because -skipping a trigger would fire it. +Beside it, **Skip** declines the question: the step that asked ends skipped, +every output of it closes, and the steps reading them skip in turn while the +rest of the run carries on. **Cancel run** ends the whole execution. Neither +shows up on a trigger, which has no question to decline: weft refuses a skip +there. After a submit the card says `Submitted` and moves to the next one. Back in the graph, the cyan ring goes green and the rest of your program runs. diff --git a/docs/src/connections/declaring-a-service.md b/docs/src/connections/declaring-a-service.md index f60fe59e..57904ba8 100644 --- a/docs/src/connections/declaring-a-service.md +++ b/docs/src/connections/declaring-a-service.md @@ -19,7 +19,7 @@ weft::access_node!(SlackAccessNode); | `acquisition` | **required** | How a credential is obtained | | `auth` | `[]` | How a request through it is signed. Empty for something that signs nothing, like a database | | `doors` | `["own"]` | Which connect doors to offer | -| `test` | none | A call made at connect time to check it works, and where identity usually comes from | +| `test` | none | A call made at connect time to check it works, and where identity usually comes from. Whatever permission it needs must be granted on every connection (see `always` below) | | `identity` | none | The display name, assembled from stored values, like `"{team} / {user}"` | | `permissions` | `[]` | The permission catalogue people tick from | | `all_permissions_url` | none | Where the provider's full list is, when yours is a subset | @@ -84,6 +84,17 @@ of scopes, list the ones your nodes actually need and point credential can never serve it, which greys out the shared option for anyone who ticks it. +If your `test` call needs a permission of its own to learn who the account is, +mark that permission `always: true`. It is then asked for on every connection, +whatever the person ticks, and never shown to tick. Google does this with +`openid` and `userinfo.email`, so a connection made for Gmail alone still +passes its check against Google's "who is this" endpoint. Without it, a test +call that reads Drive fails every connection that did not tick Drive. Only +mark a permission `always` when the provider grants it without a review and an +app allows it without being set up for it, because every app and every +connection asks for it. An `always` permission cannot also be `default` or +`own_only`. + ## Verification Say honestly what your check can learn: diff --git a/docs/src/connections/using-one-in-a-node.md b/docs/src/connections/using-one-in-a-node.md index c9bfc903..50e8c1a9 100644 --- a/docs/src/connections/using-one-in-a-node.md +++ b/docs/src/connections/using-one-in-a-node.md @@ -98,6 +98,23 @@ would depend on which input arrives, like a post that needs one scope for a person and another for a company page, that is two nodes, each declaring its own. One node with the rule written in prose protects nobody. +If an optional input needs one more permission only when somebody uses it, +leave that permission out of `requiresScopes`, which every program using the +node is asked for, and add it in `run` before you open the connection: + +```rust +if reply_to.is_some() { + let mut needed = account.required_permissions().to_vec(); + needed.push("https://www.googleapis.com/auth/gmail.readonly".to_string()); + account = account.with_required_permissions(needed); +} +let http = ctx.client(&account).await?; +``` + +Extend the list rather than replace it: it already holds what `requiresScopes` +declared. Opening then refuses a connection that lacks the permission, naming +it, the same way it refuses one missing a declared one. + A missing stored value always refuses. A missing permission only refuses when weft actually knows it is missing: on a service that reports nothing, the call goes through and the provider decides. diff --git a/docs/src/language/how-a-program-runs.md b/docs/src/language/how-a-program-runs.md index 326255f7..70bd830c 100644 --- a/docs/src/language/how-a-program-runs.md +++ b/docs/src/language/how-a-program-runs.md @@ -37,10 +37,12 @@ For an ordinary step: | What happened | What the step does | |---|---| | A required input closed | Skips | +| A wired input its node marks required once wired (`requiredWhenWired`) closed | Skips, as for a required one. Left unwired, it is optional | | An optional input closed | Nothing. The other inputs decide | | Every input closed | Skips | | Every `@require_one_of` input closed | Skips | | `_should_flow` is `false`, or closed | Skips | +| It was waiting on a person, who skipped the question | Skips | A skipped step closes its own outputs, which can skip the next step, and the next, until the closures reach something that can carry on with what it has. diff --git a/docs/src/nodes/metadata.md b/docs/src/nodes/metadata.md index dd1bef29..c9bfd6c4 100644 --- a/docs/src/nodes/metadata.md +++ b/docs/src/nodes/metadata.md @@ -42,6 +42,7 @@ Three keys are required: `type`, `label` and `description`. | `name` | String | **required** | The port name, unique among inputs | | `type` | type string | **required** | What it takes | | `required` | Boolean | `false` | A firing with nothing here skips the node. Write it only when true | +| `requiredWhenWired` | Boolean | `false` | For an input whose absence means something (no `instance`: the program's own copy). Unwired, the node runs without it; wired, a wire that delivers nothing skips the node instead of falling back to that meaning. Never with `required` | | `accepts` | List | both | `["wire"]` refuses a written value, `["literal"]` refuses an arrow. A `Bus` or `Generator` port is wire-only whatever you say | | `widget` | Object | from the type | The editor control | | `default` | any | none | What the runtime supplies when nothing drives the input. Never written into source | @@ -181,13 +182,13 @@ one for everybody: "then": { "message": "Route '{id}' would exist once per instance, because {per_instance_reason}..." } } ``` -A node that starts an instance's copy of an infra node refuses a name that is -no such node, and `SetInstanceValues` refuses a key that is no field an -instance fills: +A node that starts an instance's copy of an infra node, when it is given an +instance, refuses a name that is no such node, and `SetInstanceValues` refuses +a key that is no field an instance fills: ```json -{ "when": { "kind": "not", "of": { "kind": "input_names", "port": "node", "names": { "node": { "role": "infra", "per_instance": true } } } }, - "then": { "message": "StartInstanceInfra '{id}' names {names} in `node`...", "port": "node" } } +{ "when": { "kind": "all", "of": [{ "kind": "input_satisfied", "port": "instance" }, { "kind": "not", "of": { "kind": "input_names", "port": "node", "names": { "node": { "role": "infra", "per_instance": true } } } }] }, + "then": { "message": "StartInfra '{id}' names {names} in `node`...", "port": "node" } } { "when": { "kind": "not", "of": { "kind": "input_names", "port": "values", "names": { "field": { "instance_filled": true } } } }, "then": { "message": "SetInstanceValues '{id}' gives {names} in `values`...", "port": "values" } } ``` diff --git a/docs/src/running/browser-extension.md b/docs/src/running/browser-extension.md index f95ab14b..9707bf4d 100644 --- a/docs/src/running/browser-extension.md +++ b/docs/src/running/browser-extension.md @@ -64,7 +64,7 @@ option buttons, approve and reject pairs, images and files. |---|---| | **Submit** | Answers a question inside a run already going | | **Fire** | Starts a new run, on a trigger | -| **Skip** | Answers with nothing and lets the run carry on. Not shown on a trigger, because skipping a trigger would fire it | +| **Skip** | Declines the question: the step that asked ends skipped, and the steps reading its answer skip too. The rest of the run carries on. Not shown on a trigger, which has no question to skip (weft refuses it) | | **Cancel run** | Ends the whole execution. Also not shown on a trigger | Approve and reject are labelled by whoever wrote the step, so they can read diff --git a/docs/src/running/cli.md b/docs/src/running/cli.md index 7eb7dda5..c1ae504d 100644 --- a/docs/src/running/cli.md +++ b/docs/src/running/cli.md @@ -124,7 +124,7 @@ answers "has my trigger fired since I changed it". | Command | What it does | |---|---| -| `weft activate` | Sets up every shared trigger and starts the listeners, and prints the project's own address, where its routes answer at the root (on your machine a local port, on a cloud install its Cloud Run address). On your machine, if you want that address on a port of your choosing, `--port ` opens it there and the project keeps it for later activates. If another program or another of your projects holds that port, the command fails and nothing is activated. Without `--port`, the project keeps the port it has, or takes a free one when it has none or another program took it. Builds and registers first if it has to. A trigger that runs once per instance is left off and named in a warning, with the way to switch it on: `--instance `, or an `ActivateInstanceTriggers` node in the program. It refuses while a connection the program needs is not picked on this install, naming each step, and while infra a trigger reads is not running (it never starts infra: `weft infra start` does). A worker still up from an older build is replaced on the way: `--running-policy cancel` (the default) cancels what it runs, `wait` lets that land first, up to `--drain-timeout`. If some triggers are on and others off (an infra stop took down the ones reading it), it turns on the ones that are off and leaves the rest as they are; if every trigger is on and you changed the source, it tells you to run `weft resync` | +| `weft activate` | Sets up every shared trigger and starts the listeners, and prints the project's own address, where its routes answer at the root (on your machine a local port, on a cloud install its Cloud Run address). On your machine, if you want that address on a port of your choosing, `--port ` opens it there and the project keeps it for later activates. If another program or another of your projects holds that port, the command fails and nothing is activated. Without `--port`, the project keeps the port it has, or takes a free one when it has none or another program took it. Builds and registers first if it has to. A trigger that runs once per instance is left off and named in a warning, with the way to switch it on: `--instance `, or an `ActivateTriggers` node in the program, given the instance. It refuses while a connection the program needs is not picked on this install, naming each step, and while infra a trigger reads is not running (it never starts infra: `weft infra start` does). A worker still up from an older build is replaced on the way: `--running-policy cancel` (the default) cancels what it runs, `wait` lets that land first, up to `--drain-timeout`. If some triggers are on and others off (an infra stop took down the ones reading it), it turns on the ones that are off and leaves the rest as they are; if every trigger is on and you changed the source, it tells you to run `weft resync` | | `weft deactivate` | Stops the program's own triggers listening, and tells you how many instances still have theirs on. `--instance ` stops one instance's, `--all-instances` stops every instance's and leaves the program's own alone | | `weft resync` | Deactivate and re-activate in one shot against your current program. With no flag it does every trigger that is on, the program's own and every instance's, and prints whose it did. It only touches triggers that are on; for ones that are off, use `weft activate` | | `weft cancel-activate` | Cancels an activate in flight | diff --git a/docs/src/running/cloud.md b/docs/src/running/cloud.md index a36102e5..3c00774a 100644 --- a/docs/src/running/cloud.md +++ b/docs/src/running/cloud.md @@ -504,7 +504,17 @@ which limit. If the program runs on several copies of its worker, each copy counts calls itself and hears the others' counts once a second, so a per-minute limit can let through about one second's worth of extra calls, and `callsAtOnce` is shared -out between the copies that are up. +out between the copies that are up. A copy with nothing to do stops listening +so the install can sleep. When a call wakes it, that call is decided at once on +what the copy heard before it went quiet, and so is every call until the copy +hears back from weft's broker: under a second normally, a few seconds when the +broker was asleep too and has to start. During that window the copy counts +itself as the only copy, so if several copies wake at the same moment, each +can let in up to the full `callsAtOnce`. The per-minute limits barely move: +they can only overshoot by the calls that reach the copy in those few +seconds. We chose this on purpose: the other way would make the first call +after every pause wait for the broker, and on a sleeping install that means +waiting for it to start. ## When something goes wrong diff --git a/docs/src/running/instances.md b/docs/src/running/instances.md index eefdff01..e1bde94a 100644 --- a/docs/src/running/instances.md +++ b/docs/src/running/instances.md @@ -471,9 +471,9 @@ the `instances` package of the standard catalog has a node for each step: | Node | What it does | |---|---| -| `StartInstanceInfra` | Starts an instance's copy of an infra node, and fires once it runs (or, with `waitUntilRunning` off, once weft accepted the start) | -| `StopInstanceInfra` | Stops it, keeping its disk | -| `TerminateInstanceInfra` | Deletes it and its disk | +| `StartInfra` | Starts an instance's copy of an infra node, and fires once it runs (or, with `waitUntilRunning` off, once weft accepted the start) | +| `StopInfra` | Stops it, keeping its disk | +| `TerminateInfra` | Deletes it and its disk | | `InstanceInfraStatus` | Reads the state of an instance's copy | | `ListInstanceInfra` | Lists every copy of an infra node, the shared one and each instance's | | `ListInstances` | Lists every instance weft holds anything for (values, connections, copies, tokens, triggers), with counts and states, and the events waiting on a field not yet filled | @@ -481,20 +481,25 @@ the `instances` package of the standard catalog has a node for each step: | `InstanceCosts` | Lists what an instance's runs cost, with the total | | `GetInstanceValues` | Reads an instance's values for the `@instance_filled` fields, keyed `node.field` | | `SetInstanceValues` | Sets or clears an instance's `@instance_filled` values, and sets up again any of its triggers that read a changed one | -| `ActivateInstanceTriggers` | Turns on an instance's triggers | -| `DeactivateInstanceTriggers` | Turns them off | +| `ActivateTriggers` | Turns on an instance's triggers | +| `DeactivateTriggers` | Turns them off | | `MintInstanceToken` | Mints an instance token to hand to a browser | | `WipeInstance` | Removes everything attached to an instance | +If you leave `instance` unwired on `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` or `DeactivateTriggers`, it acts on the program's own copies +instead: the shared copy of an infra node not marked `@per_instance`, or the +triggers every instance shares. + If your backend should create instances, put `MintInstanceToken`, -`StartInstanceInfra` and `ActivateInstanceTriggers` behind a gated route, in +`StartInfra` and `ActivateTriggers` behind a gated route, in that order. If you want the request to answer at once rather than wait -minutes for the copy, turn `waitUntilRunning` off on `StartInstanceInfra` and +minutes for the copy, turn `waitUntilRunning` off on `StartInfra` and reply with the token after it: the start is accepted by then, so a list read right after shows the copy as `provisioning`. A reply placed before the start leaves nothing for that list to show yet. If one of the instance's triggers reads a value with no fallback, activating it at creation is refused, so leave -`ActivateInstanceTriggers` for a second call your website makes once the +`ActivateTriggers` for a second call your website makes once the settings are saved. The same route can remove an instance with `WipeInstance`: weft cannot tell when an instance is no longer needed, so your backend says so. diff --git a/extension-browser/package.json b/extension-browser/package.json index bc471a68..9799fa86 100644 --- a/extension-browser/package.json +++ b/extension-browser/package.json @@ -2,7 +2,7 @@ "name": "weft-extension", "description": "Human-in-the-loop task manager for weft projects", "private": true, - "version": "1.0.78", + "version": "1.0.79", "packageManager": "pnpm@10.32.1", "type": "module", "scripts": { diff --git a/extension-browser/src/entrypoints/popup/App.svelte b/extension-browser/src/entrypoints/popup/App.svelte index 8848cec3..a3da2f24 100644 --- a/extension-browser/src/entrypoints/popup/App.svelte +++ b/extension-browser/src/entrypoints/popup/App.svelte @@ -156,8 +156,9 @@ }); const refresh = refreshFlight.join; - /// Skip = "I don't want to answer this one." Resume the lane - /// with null; the rest of the run keeps going. + /// Skip = "I don't want to answer this one." The step that asked + /// ends skipped, its outputs close and what reads them skips; the + /// rest of the run keeps going. async function handleSkipTask(task: PendingTask) { try { await skipTask(task); @@ -598,10 +599,11 @@ + runner's): a RESUME task acts on a paused run (skip = end the + asking step as skipped, cancel = kill the run), so it shows them; a + TRIGGER has no question to skip (weft refuses it) and no in-flight + run to cancel, so opening it (to submit its form and START a run) + is the only action. --> {#snippet taskCard(task: PendingTask)}
{#if !isTrigger(currentTask)} diff --git a/extension-browser/src/lib/api.ts b/extension-browser/src/lib/api.ts index 856a73c0..e6fbd3d4 100644 --- a/extension-browser/src/lib/api.ts +++ b/extension-browser/src/lib/api.ts @@ -446,11 +446,10 @@ export async function submitTask( } } -/// Skip ONE task: resume its lane with null. Sibling lanes of -/// the same execution keep going. Most upstream code patterns -/// auto-skip on null inputs (downstream null-propagation), so -/// this is the "I don't want to answer this one, do whatever" -/// action. Auth: signal token alone (knowing it = permission). +/// Skip ONE task: decline the question. The step that asked ends +/// skipped, its outputs close and what reads them skips; sibling lanes +/// of the same execution keep going. Auth: signal token alone (knowing +/// it = permission). export async function skipTask(task: PendingTask): Promise { const tokenConfig = task._tokenConfig; if (!tokenConfig) throw new Error('Task missing token configuration'); diff --git a/extension-vscode/package.json b/extension-vscode/package.json index 6735f968..d595ae9c 100644 --- a/extension-vscode/package.json +++ b/extension-vscode/package.json @@ -2,7 +2,7 @@ "name": "weft-vscode", "displayName": "Weft", "description": "Write Weft programs and watch them run: a live, editable graph of every .weft file, with diagnostics, runs and executions in the editor.", - "version": "0.2.266", + "version": "0.2.267", "publisher": "weavemind", "license": "SEE LICENSE IN LICENSE", "repository": { diff --git a/extension-vscode/src/actionBarState.test.ts b/extension-vscode/src/actionBarState.test.ts index 745eb267..1f8fab47 100644 --- a/extension-vscode/src/actionBarState.test.ts +++ b/extension-vscode/src/actionBarState.test.ts @@ -28,6 +28,7 @@ function snapshot(): ActionAvailability { orphanedInfra: false, mode: 'active', runningCount: 0, + triggersOff: [], infraRollup: 'none', infraNodes: [], preservation: { parked: 0, suspended: 0 }, diff --git a/extension-vscode/src/cli.ts b/extension-vscode/src/cli.ts index 173b100a..8d22b7d6 100644 --- a/extension-vscode/src/cli.ts +++ b/extension-vscode/src/cli.ts @@ -15,11 +15,15 @@ import type * as vscode from 'vscode'; /// side reads off it. interface CliFailure { message?: string; - /// The request never reached the daemon: it is not running, or was - /// never installed here. A FLAG rather than a sentence to match on, - /// because the sentence is wording and wording changes. - // SYNC: daemonUnreachable <-> crates/weft-cli/src/progress.rs report_plain_error + /// The request never reached this machine's daemon: it is not + /// running, or was never installed here. A FLAG rather than a sentence + /// to match on, because the sentence is wording and wording changes. + // SYNC: daemonUnreachable, installUnreachable <-> crates/weft-cli/src/progress.rs error_detail daemonUnreachable?: boolean; + /// The request never reached the install it went to, this machine's or + /// a remote one (`--on `). Only `daemonUnreachable` means + /// something here could be started to fix it. + installUnreachable?: boolean; } function jsonFailure(value: unknown): CliFailure | undefined { @@ -30,6 +34,7 @@ function jsonFailure(value: unknown): CliFailure | undefined { return { message: typeof detail?.message === 'string' ? detail.message : undefined, daemonUnreachable: detail?.daemonUnreachable === true, + installUnreachable: detail?.installUnreachable === true, }; } @@ -46,6 +51,10 @@ export class WeftCliError extends Error { /// somebody than "that project is not registered", so it travels as /// its own fact rather than as words inside `stderr`. public readonly daemonUnreachable: boolean = false, + /// True when the install the verb went to did not answer, this + /// machine's daemon or a remote install. A remote one is not + /// something to start from here: the person checks the network. + public readonly installUnreachable: boolean = false, ) { const reason = stderr.trim() ? stderr.trim() : `exited ${code}`; super(`weft ${args.join(' ')}: ${reason}`); @@ -139,6 +148,7 @@ export function runWeftJson( code, failure?.message ?? stderr, failure?.daemonUnreachable ?? false, + failure?.installUnreachable ?? false, ), ); return; diff --git a/extension-vscode/src/execFollower.ts b/extension-vscode/src/execFollower.ts index 6724d162..a9de6c14 100644 --- a/extension-vscode/src/execFollower.ts +++ b/extension-vscode/src/execFollower.ts @@ -42,6 +42,11 @@ import type { // data by bytes, since a single node result can be much larger than a status. export const MAX_REPLAY_BUFFER_BYTES = 8 * 1024 * 1024; +/** How a wait ended: its answer, a person skipping it, or the run giving + * it up (the waiting step failed with `error`). */ +// SYNC: AwaitEnd <-> crates/weft-core/src/primitive.rs AwaitEnd +export type AwaitEnd = { type: 'answered'; value: unknown } | { type: 'skipped' } | { type: 'gave_up'; error: string }; + // SYNC: DispatcherEvent <-> crates/weft-core/src/live_event.rs DispatcherEvent, weavemind/website/src/lib/graph/dispatcher-host.ts translateDispatcherEvent // SYNC: event_id <-> crates/weft-core/src/live_event.rs IdentifiedEvent export type DispatcherEvent = { event_id: string } & ( @@ -54,7 +59,11 @@ export type DispatcherEvent = { event_id: string } & ( // SYNC: input origins <-> crates/weft-core/src/live_event.rs DispatcherEvent, packages/weft-graph/src/protocol.ts NodeExecEvent, packages/weft-graph/src/webview/lib/types/index.ts NodeExecution | { kind: 'node_started'; execution_id: string; node: string; frames: Frame[]; input: unknown; closed_ports: string[]; provided_ports?: string[]; backup_ports?: string[]; inherited_ports?: Record; inherited_from?: string; project_id: string; at_unix: number } | { kind: 'node_suspended'; execution_id: string; node: string; frames: Frame[]; token: string; inherited_from?: string; project_id: string; at_unix: number } - | { kind: 'node_resumed'; execution_id: string; node: string; frames: Frame[]; token: string | null; value: unknown; inherited_from?: string; project_id: string; at_unix: number } + | { kind: 'node_resumed'; execution_id: string; node: string; frames: Frame[]; token: string | null; inherited_from?: string; project_id: string; at_unix: number } + // A wait of the firing ended, the first ending on record. A wait the run + // holds in its worker never suspends or resumes the firing, so this is + // the one event that sees every wait's ending. + | { kind: 'wait_ended'; execution_id: string; node: string; frames: Frame[]; token: string; ended: AwaitEnd; inherited_from?: string; project_id: string; at_unix: number } | { kind: 'node_cancelled'; execution_id: string; node: string; frames: Frame[]; reason: string; inherited_from?: string; project_id: string; at_unix: number } | { kind: 'node_completed'; execution_id: string; node: string; frames: Frame[]; output: unknown; inherited_from?: string; project_id: string; at_unix: number } | { kind: 'node_failed'; execution_id: string; node: string; frames: Frame[]; error: string; inherited_from?: string; project_id: string; at_unix: number } @@ -673,6 +682,10 @@ export class ExecutionFollower implements vscode.Disposable { case 'trigger_url_changed': case 'infra_config_error': break; + // How a wait ended is what `weft freeze` records; the graph paints + // the firing's own lifecycle, which the node events already carry. + case 'wait_ended': + break; default: { // Exhaustiveness: if a new DispatcherEvent variant is // added to the union without a route here, TypeScript diff --git a/extension-vscode/src/extension.ts b/extension-vscode/src/extension.ts index ec6db4d8..d8b8876d 100644 --- a/extension-vscode/src/extension.ts +++ b/extension-vscode/src/extension.ts @@ -962,6 +962,12 @@ export function activate(context: vscode.ExtensionContext) { return err instanceof WeftCliError && err.daemonUnreachable; } + /// A remote install (`--on `) did not answer: nothing here + /// starts it, so it gets its own banner, never the daemon's. + function isRemoteUnreachable(err: unknown): err is WeftCliError { + return err instanceof WeftCliError && err.installUnreachable && !err.daemonUnreachable; + } + interface StatusResult { snapshot: import('../../packages/weft-graph/src/protocol').ActionAvailability; /// Every execution running right now, per the dispatcher. The @@ -1002,7 +1008,8 @@ export function activate(context: vscode.ExtensionContext) { // not running or was never installed on this machine. Every verb // will fail the same way, so the bar keeps its buttons and says // what is wrong above them instead of quietly looking like a - // project nobody has registered. + // project nobody has registered. A remote install that does not + // answer is said the same way, without the daemon's advice. if (isDaemonUnreachable(err)) { actionBar.setError( projectId, @@ -1023,6 +1030,23 @@ export function activate(context: vscode.ExtensionContext) { ], }, ); + } else if (isRemoteUnreachable(err)) { + actionBar.setError( + projectId, + 'status', + 'The install this project targets is not answering, so nothing here can run.', + { + what: 'Reading the project status', + stage: 'dispatch', + diagnostics: [ + { + severity: 'error', + message: err.stderr.trim() || 'Nothing answered at the install.', + hint: 'Check the network, then try again.', + }, + ], + }, + ); } else { console.warn('[weft] status fetch failed; project not registered yet', err); actionBar.clearErrorIfVerb(projectId, 'status'); diff --git a/packages/weft-connect/src/core/recipe.test.ts b/packages/weft-connect/src/core/recipe.test.ts index 9a931497..7bf576a8 100644 --- a/packages/weft-connect/src/core/recipe.test.ts +++ b/packages/weft-connect/src/core/recipe.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import { canDo, displayLabel, needsConsent, ownFields, ownRegistration, permissionSummary } from './recipe'; +import { canDo, displayLabel, guideSteps, needsConsent, ownFields, ownRegistration, permissionSummary, requestedPermissions, tickablePermissions } from './recipe'; import type { AccessSpecWire, GrantSummary } from './wire'; const oauth: AccessSpecWire = { @@ -32,6 +32,19 @@ describe('recipe', () => { expect(ownFields(oauth).map((f) => f.name)).toEqual(['client_id', 'client_secret']); }); + it('asks for what the service always needs on top of what is ticked, and never offers it to tick', () => { + const whoami: AccessSpecWire = { + ...oauth, + permissions: [{ id: 'email', label: 'Your address', description: '', always: true }, ...(oauth.permissions ?? [])], + own_page: { guide: { steps: ['Allow {permissions}.'] } }, + }; + expect(requestedPermissions(whoami, ['mail'])).toEqual(['email', 'mail']); + expect(requestedPermissions(whoami, ['email', 'mail'])).toEqual(['email', 'mail']); + expect(tickablePermissions(whoami).map((p) => p.id)).toEqual(['mail', 'cal', 'drive', 'docs']); + expect(guideSteps(whoami, ['mail'])).toEqual(['Allow Your address, Read mail.']); + expect(permissionSummary(whoami, ['email', 'mail'])).toBe('Read mail'); + }); + it('reads a connection row the way the list shows it', () => { expect(permissionSummary(oauth, ['mail', 'cal', 'drive', 'docs'])).toBe('Read mail, Calendar, Drive, ...'); expect(canDo(oauth, row({ scopes: ['mail'], permissions_verified: false }))).toBe('Read mail (claimed)'); diff --git a/packages/weft-connect/src/core/recipe.ts b/packages/weft-connect/src/core/recipe.ts index 423b7314..d608bde8 100644 --- a/packages/weft-connect/src/core/recipe.ts +++ b/packages/weft-connect/src/core/recipe.ts @@ -32,8 +32,9 @@ export function ownFields(spec: AccessSpecWire): CredentialFieldWire[] { // SYNC: guideSteps <-> crates/weft-core/src/access/spec.rs AccessSpec::guide_steps export function guideSteps(spec: AccessSpecWire, ticked: string[]): string[] { const steps = spec.own_page?.guide?.steps ?? []; + const requested = requestedPermissions(spec, ticked); const labels = (spec.permissions ?? []) - .filter((p) => ticked.includes(p.id)) + .filter((p) => requested.includes(p.id)) .map((p) => p.label) .join(', '); return steps.map((s) => s.replaceAll('{permissions}', labels)); @@ -50,7 +51,7 @@ export function guideLink(spec: AccessSpecWire, ticked: string[]): string | unde // encodeURIComponent leaves !'()* bare; the Rust side encodes // everything outside RFC 3986 unreserved. Encode them too so the // two produce byte-identical URLs for every id. - const encoded = encodeURIComponent(ticked.join(',')).replace( + const encoded = encodeURIComponent(requestedPermissions(spec, ticked).join(',')).replace( /[!'()*]/g, (c) => '%' + c.charCodeAt(0).toString(16).toUpperCase(), ); @@ -68,10 +69,20 @@ export function defaultPermissions(spec: AccessSpecWire): string[] { } // The catalogue entries the consent/tick surfaces show: everything -// except own-account-only capabilities (those surface as their own -// tutorial sections instead). +// except what is asked for always, and own-account-only capabilities +// (those surface as their own tutorial sections instead). +// SYNC: tickablePermissions <-> crates/weft-core/src/access/spec.rs AccessSpec::tickable_permissions export function tickablePermissions(spec: AccessSpecWire) { - return (spec.permissions ?? []).filter((p) => !p.own_only); + return (spec.permissions ?? []).filter((p) => !p.always && !p.own_only); +} + +// What a consent asks for when `ticked` is ticked: the permissions asked +// for always, then the ticked ones, each once. +// SYNC: requestedPermissions <-> crates/weft-core/src/access/spec.rs AccessSpec::requested_permissions +export function requestedPermissions(spec: AccessSpecWire, ticked: string[]): string[] { + const out = (spec.permissions ?? []).filter((p) => p.always).map((p) => p.id); + for (const id of ticked) if (!out.includes(id)) out.push(id); + return out; } // The name a service is shown under. @@ -94,9 +105,11 @@ export function permissionLabels(spec: AccessSpecWire, ids: string[]): string[] } // A short "these permissions" summary: the first three labels, an -// ellipsis past that. +// ellipsis past that. What every connection of the service is asked for +// (`always`) says nothing about this one, so it is left out. export function permissionSummary(spec: AccessSpecWire, ids: string[]): string { - const named = permissionLabels(spec, ids); + const always = (spec.permissions ?? []).filter((p) => p.always).map((p) => p.id); + const named = permissionLabels(spec, ids.filter((id) => !always.includes(id))); return named.slice(0, 3).join(', ') + (named.length > 3 ? ', ...' : ''); } diff --git a/packages/weft-connect/src/core/wire.ts b/packages/weft-connect/src/core/wire.ts index 58277914..3d9e94ab 100644 --- a/packages/weft-connect/src/core/wire.ts +++ b/packages/weft-connect/src/core/wire.ts @@ -44,6 +44,10 @@ export interface Permission { label: string; description: string; default?: boolean; + /** Asked for on every consent, whatever is ticked, and never shown to + * tick: what the service's test call needs to learn who the account + * is. */ + always?: boolean; /** This capability creates or reads things INSIDE the credential's * own account, so a runtime-supplied (shared) credential can never * serve it; the editor greys the shared option and resolution @@ -400,6 +404,11 @@ export interface PortDefinition { export interface InputDefinition extends PortDefinition { // SYNC: InputDefinition.accepts <-> crates/weft-core/src/project.rs InputDefinition.accepts accepts?: Accepts; + /// Optional while unwired, required once wired: a wire into it that + /// arrives closed skips the node. Its wiring decides, so the editor + /// draws it required only while a wire feeds it. + // SYNC: InputDefinition.requiredWhenWired <-> crates/weft-core/src/project.rs InputDefinition.required_when_wired + requiredWhenWired?: boolean; // SYNC: InputDefinition.widget <-> crates/weft-core/src/project.rs InputDefinition.widget widget?: Widget; default?: unknown; diff --git a/packages/weft-graph/src/protocol.ts b/packages/weft-graph/src/protocol.ts index 538448cf..84514c50 100644 --- a/packages/weft-graph/src/protocol.ts +++ b/packages/weft-graph/src/protocol.ts @@ -658,6 +658,9 @@ export interface InputSpec { name: string; type: string; required?: boolean; + /// Optional while unwired, required once wired; never with `required`. + // SYNC: InputSpec.requiredWhenWired <-> crates/weft-core/src/node.rs InputSpec.required_when_wired + requiredWhenWired?: boolean; // SYNC: InputSpec.accepts <-> crates/weft-core/src/node.rs InputSpec.accepts accepts?: Accepts; widget?: Widget; @@ -845,7 +848,8 @@ export type SkipReason = | { kind: 'required_input_closed'; port: string; failure?: Failure } | { kind: 'every_input_closed'; failure?: Failure } | { kind: 'one_of_group_closed'; ports: string[]; failure?: Failure } - | { kind: 'scope_skipped'; scope: string }; + | { kind: 'scope_skipped'; scope: string } + | { kind: 'wait_skipped' }; /// A node that broke, as the closures it leaves carry it: the node spelled /// the way the program reads it, and its error. Passed on unchanged through @@ -1445,6 +1449,10 @@ export interface ActionAvailability { /// Count of running, non-suspended executions. Drives the /// deactivating-state UI: shows "draining N executions...". runningCount: number; + /// The program's own triggers that are off while the project is + /// active (an infra stop took down the ones reading it): what the + /// bar's second button turns back on, by name. + triggersOff: string[]; /// Infra rollup. infraRollup: InfraRollup; /// An infra operation is in flight that the rollup cannot see yet. @@ -1510,6 +1518,10 @@ export type CliPhase = /// The build goes on; detail `elapsedSeconds`, `images`, the ones /// still building, and `built`, the ones done during this wait. | 'build_wait' + /// The install cannot be reached to see where the build is; the CLI + /// keeps looking. Detail `elapsedSeconds` and `message`, why the last + /// look failed. The next `build_wait` or `build_image` says it answers again. + | 'build_unreachable' | 'dispatcher_call_start' | 'dispatcher_call_done' | 'infra_provision_start' @@ -1701,6 +1713,8 @@ export type BackendSnapshot = { infraBusy: boolean; /// Drain progress when status='deactivating'. runningCount: number; + /// See `ActionAvailability.triggersOff`. + triggersOff: string[]; /// Hibernate-grace deadline, when present. firesDeadlineUnix?: number; }; diff --git a/packages/weft-graph/src/run-spec.test.ts b/packages/weft-graph/src/run-spec.test.ts index 23d4b550..70314a4a 100644 --- a/packages/weft-graph/src/run-spec.test.ts +++ b/packages/weft-graph/src/run-spec.test.ts @@ -35,6 +35,23 @@ describe('run spec', () => { answers: [{ node: 'review', payload: 'yes', question: 'Continue?' }], caller: [{ text: 'hello' }] }; expect(parseRunSpec(spec)).toEqual(spec); }); + it('reads a wait ending as a payload (null included), a skip or a give-up, and refuses two or none', () => { + for (const answer of [{ node: 'review', payload: null }, { node: 'review', skipped: true }, { node: 'review', gave_up: "node 'review' gave up its wait: quiet" }]) { + const spec = { name: 'x', answers: [answer] }; + expect(parseRunSpec(spec)).toEqual(spec); + } + const two: Array<[Record, string]> = [ + [{ payload: null, skipped: true }, '(a payload and "skipped": true)'], + [{ payload: 1, gave_up: 'quiet' }, '(a payload and "gave_up")'], + [{ skipped: true, gave_up: 'quiet' }, '("skipped": true and "gave_up")'], + ]; + for (const [ending, named] of two) { + expect(() => parseRunSpec({ name: 'x', answers: [{ node: 'review', ...ending }] })).toThrow(`ended more than one way ${named}`); + } + expect(() => parseRunSpec({ name: 'x', answers: [{ node: 'review' }] })).toThrow('says nothing about how its wait ended'); + expect(() => parseRunSpec({ name: 'x', answers: [{ node: 'review', skipped: false }] })).toThrow('says nothing about how its wait ended'); + expect(() => parseRunSpec({ name: 'x', answers: [{ node: 'review', gave_up: 3 }] })).toThrow('gave_up'); + }); it('reads how long a run is kept and refuses what is no duration', () => { expect(parseRunSpec({ name: 'x', settings: { keep_for: '12h' } }).settings).toEqual({ keep_for: '12h' }); expect(parseRunSpec({ name: 'x', settings: { keep_for: 'forever' } }).settings).toEqual({ keep_for: 'forever' }); diff --git a/packages/weft-graph/src/run-spec.ts b/packages/weft-graph/src/run-spec.ts index 514f221b..171c9344 100644 --- a/packages/weft-graph/src/run-spec.ts +++ b/packages/weft-graph/src/run-spec.ts @@ -149,12 +149,23 @@ export function parseRunSpec(value: unknown): RunSpec { if (fire.length !== 2) throw new Error('fire: expected [trigger, payload]'); string(fire[0], 'fire trigger'); } + // SYNC: answer refusals <-> crates/weft-core/src/run_spec.rs Answer TryFrom if (spec.answers !== undefined) for (const value of array(spec.answers, 'answers')) { const answer = object(value, 'answer'); - fields(answer, ['node', 'frames', 'payload', 'question'], 'answer'); + fields(answer, ['node', 'frames', 'payload', 'skipped', 'gave_up', 'question'], 'answer'); string(answer.node, 'answer node'); frames(answer.frames, 'answer frames'); - if (!('payload' in answer)) throw new Error('answer: missing payload'); + if (answer.skipped !== undefined && typeof answer.skipped !== 'boolean') throw new Error('answer: skipped must be true or false'); + // A null `gave_up` is left out, as the Rust side reads it. + if (answer.gave_up != null) string(answer.gave_up, 'answer gave_up'); + // `payload: null` is a given null: only an absent payload is none. + const said = [ + ...('payload' in answer ? ['a payload'] : []), + ...(answer.skipped === true ? ['"skipped": true'] : []), + ...(answer.gave_up != null ? ['"gave_up"'] : []), + ]; + if (said.length > 1) throw new Error(`the answer to '${answer.node}' says its wait ended more than one way (${said.join(' and ')}); a wait ends once, so keep only one`); + if (said.length === 0) throw new Error(`the answer to '${answer.node}' says nothing about how its wait ended; give the value the person answered ("payload"), "skipped": true if they skipped it, or "gave_up" with the error if the run gave it up`); } if (spec.caller !== undefined) array(spec.caller, 'caller'); if (spec.frozen_from != null) { @@ -196,12 +207,20 @@ export function parseRunSpec(value: unknown): RunSpec { return normalized as unknown as RunSpec; } -export interface Answer { +/** How one wait ended, as a frozen example holds it: the value the person + * gave as `payload` (`null` included), `skipped: true` for a person + * skipping it, or `gave_up` with the error the waiting step failed with + * when the run gave the wait up. Exactly one of the three. */ +// SYNC: Answer <-> crates/weft-core/src/run_spec.rs Answer +export type Answer = { node: string; frames?: Array<{ index: number }>; - payload: JsonValue; question?: JsonValue; -} +} & ( + | { payload: JsonValue; skipped?: false; gave_up?: never } + | { skipped: true; payload?: never; gave_up?: never } + | { gave_up: string; payload?: never; skipped?: false } +); export interface FrozenFrom { version: string; diff --git a/packages/weft-graph/src/status.test.ts b/packages/weft-graph/src/status.test.ts index 7bb95556..0dd26e25 100644 --- a/packages/weft-graph/src/status.test.ts +++ b/packages/weft-graph/src/status.test.ts @@ -35,6 +35,20 @@ describe('parseStatusPayload', () => { expect(snap.preservation).toEqual({ parked: 2, suspended: 1 }); }); + it('names the program\'s own triggers that are off, never an instance\'s', () => { + const snap = parseStatusPayload({ + status: 'active', + activations: [ + { trigger: 'gmail.watch', status: 'inactive' }, + { trigger: 'daily', status: 'active' }, + { trigger: 'fresh', status: 'registered' }, + { trigger: 'gmail.watch', instance: 'ada', status: 'inactive' }, + ], + }); + expect(snap.triggersOff).toEqual(['gmail.watch', 'fresh']); + expect(parseStatusPayload({}).triggersOff).toEqual([]); + }); + it('collapses unknown enum strings to their resting value (version skew, no crash)', () => { const snap = parseStatusPayload({ status: 'some-future-status', diff --git a/packages/weft-graph/src/status.ts b/packages/weft-graph/src/status.ts index 3b6a0711..81c650a4 100644 --- a/packages/weft-graph/src/status.ts +++ b/packages/weft-graph/src/status.ts @@ -46,6 +46,7 @@ export interface RawStatusPayload { }; available_actions?: string[]; preservation?: { parked?: number; suspended?: number }; + activations?: Array<{ trigger?: string; instance?: string; status?: string }>; executions?: { last_status?: string; last_execution_id?: string; @@ -103,6 +104,7 @@ export function emptyActionAvailability(): ActionAvailability { orphanedInfra: false, mode: 'unknown', runningCount: 0, + triggersOff: [], infraRollup: 'none', infraBusy: false, infraNodes: [], @@ -150,6 +152,7 @@ export function parseStatusPayload(raw: RawStatusPayload): ActionAvailability { mode: String(raw.mode ?? 'unknown'), ...(firesDeadlineUnix !== undefined ? { firesDeadlineUnix } : {}), runningCount: Number(raw.running_count ?? 0), + triggersOff: triggersOff(raw), infraRollup, infraBusy: !!raw.infra_busy, infraNodes, @@ -160,6 +163,17 @@ export function parseStatusPayload(raw: RawStatusPayload): ActionAvailability { }; } +/// The program's own triggers that are off: registered and never +/// switched on, or taken down. An instance's copies are its own and stay +/// out of it, as the bar acts on the program's. +// SYNC: triggersOff <-> crates/weft-dispatcher/src/api/project.rs is_down +function triggersOff(raw: RawStatusPayload): string[] { + return (raw.activations ?? []) + .filter((a) => !a.instance && (a.status === 'registered' || a.status === 'inactive')) + .map((a) => a.trigger ?? '') + .filter((trigger) => trigger !== ''); +} + /// One execution running right now, and what it is for. // SYNC: RunningExecution <-> crates/weft-core/src/projects.rs RunningExecution export interface RunningExecution { @@ -202,6 +216,7 @@ export function backendFromSnapshot(snapshot: ActionAvailability): BackendSnapsh infraRollup: snapshot.infraRollup, infraBusy: snapshot.infraBusy, runningCount: snapshot.runningCount, + triggersOff: snapshot.triggersOff, ...(snapshot.firesDeadlineUnix !== undefined ? { firesDeadlineUnix: snapshot.firesDeadlineUnix } : {}), diff --git a/packages/weft-graph/src/webview/lib/components/project/ActionBar.svelte b/packages/weft-graph/src/webview/lib/components/project/ActionBar.svelte index 1ac5a489..34750d88 100644 --- a/packages/weft-graph/src/webview/lib/components/project/ActionBar.svelte +++ b/packages/weft-graph/src/webview/lib/components/project/ActionBar.svelte @@ -186,6 +186,19 @@ const infraReady = $derived(sourceInfraReady(verbInputs)); const isVerbAvailable = (v: ActionVerb): boolean => isVerbOffered(v, verbInputs); + // The triggers that are off while the others run, as the second + // button names them: two by name, then a count. + function offNames(off: string[]): string { + if (off.length === 0) return 'the triggers that are off'; + if (off.length <= 2) return off.join(', '); + return `${off.slice(0, 2).join(', ')} and ${off.length - 2} more`; + } + + function offTitle(off: string[]): string { + const which = off.length > 0 ? `Off while the others run: ${off.join(', ')}.` : 'Some triggers are off while the others run.'; + return `${which} An infra stop takes down the triggers reading it. Turn them back on.`; + } + // The build-transition axis. While not 'none', the whole bar is in // the unified transitional pattern: the slot that owns the verb // shows "Building... (cancel)" and every other verb is gated (the @@ -229,6 +242,15 @@ case 'build_image': case 'build_image_done': case 'build_wait': return 'Building...'; + case 'build_unreachable': { + // The install did not answer the last look at the build: the + // build may go on there, but nothing here can see it, so the + // bar stops saying "Building..." until it answers again. + // SYNC: the build_unreachable detail <-> crates/weft-cli/src/progress.rs (Progress::build_unreachable) + const elapsed = + typeof detail?.elapsedSeconds === 'number' ? ` (${detail.elapsedSeconds}s)` : ''; + return `Install not answering, looking again${elapsed}...`; + } case 'build_done': return 'Loading...'; case 'infra_provision_start': case 'infra_provision_done': return 'Provisioning infra...'; @@ -936,10 +958,10 @@ {/if}
diff --git a/packages/weft-graph/src/webview/lib/components/project/ProjectNode.svelte b/packages/weft-graph/src/webview/lib/components/project/ProjectNode.svelte index 2f6ba9c5..3de7d544 100644 --- a/packages/weft-graph/src/webview/lib/components/project/ProjectNode.svelte +++ b/packages/weft-graph/src/webview/lib/components/project/ProjectNode.svelte @@ -1918,12 +1918,14 @@ live in the body as fields, never on the edge rail) -->
{#each wireableInputs as input} - {@const pMarker = portMarkerStyle(input, oneOfRequiredPorts, literalFilledPorts, getPortColor(input.portType), 'input')} + {@const pMarker = portMarkerStyle(input, oneOfRequiredPorts, literalFilledPorts, getPortColor(input.portType), 'input', '', wiredInputPorts.has(input.name))} {@const inputDeleteAction = portDeleteAction(input, providedInputs, canAddInputPorts)}
g.includes(input.name)).map(g => g.join(' or ')).join('; ')}` : input.name} + title={input.requiredWhenWired + ? `${input.name}: required when wired (if its wire delivers nothing, this node skips); optional while unwired` + : !input.required && oneOfRequiredPorts.has(input.name) ? `At least one required: ${oneOfRequiredGroups.filter(g => g.includes(input.name)).map(g => g.join(' or ')).join('; ')}` : input.name} oncontextmenu={(e) => { e.preventDefault(); e.stopPropagation(); diff --git a/packages/weft-graph/src/webview/lib/utils/port-context-menu.test.ts b/packages/weft-graph/src/webview/lib/utils/port-context-menu.test.ts index 739466bd..f08421d0 100644 --- a/packages/weft-graph/src/webview/lib/utils/port-context-menu.test.ts +++ b/packages/weft-graph/src/webview/lib/utils/port-context-menu.test.ts @@ -54,6 +54,14 @@ describe('buildPortMenuItems', () => { expect(labels(items)).toContain('an input of the same name conflicts'); }); + it('an input required only once wired offers no required toggle, only why', () => { + const items = build({ port: { ...port, requiredWhenWired: true } }); + expect(labels(items)).not.toContain('☑ Make required'); + expect(labels(items)).not.toContain('☐ Make optional'); + expect(notes(items).map((n) => n.label)).toEqual(['Required when wired, optional while unwired: its wiring decides.']); + expect(labels(items)).toContain('✎ Type: String'); + }); + it('an output has no required toggle', () => { expect(labels(build({}, { side: 'output' }))).toEqual(['✎ Type: String']); }); diff --git a/packages/weft-graph/src/webview/lib/utils/port-context-menu.ts b/packages/weft-graph/src/webview/lib/utils/port-context-menu.ts index e60b13a5..ece987a2 100644 --- a/packages/weft-graph/src/webview/lib/utils/port-context-menu.ts +++ b/packages/weft-graph/src/webview/lib/utils/port-context-menu.ts @@ -209,8 +209,17 @@ export function buildPortMenuItems(opts: BuildPortMenuOptions): PortMenuItem[] { if (opts.instanceFilled) items.push(instanceFilledMenuItem(opts.instanceFilled)); // Required toggle (inputs only; outputs do not have runtime required - // semantics, and only an input caller hands over the writer). - if (opts.side === 'input') { + // semantics, and only an input caller hands over the writer). An input + // required only once wired has its requiredness decided by its wiring + // (the compiler refuses a header making it required), so it says so + // instead of offering the toggle. + if (opts.side === 'input' && port.requiredWhenWired) { + items.push({ + label: 'Required when wired, optional while unwired: its wiring decides.', + note: true, + color: '#71717a', + }); + } else if (opts.side === 'input') { const setRequired = opts.onSetRequired; items.push({ label: port.required ? '☐ Make optional' : '☑ Make required', diff --git a/packages/weft-graph/src/webview/lib/utils/port-marker.test.ts b/packages/weft-graph/src/webview/lib/utils/port-marker.test.ts index d1fd4053..56917b74 100644 --- a/packages/weft-graph/src/webview/lib/utils/port-marker.test.ts +++ b/packages/weft-graph/src/webview/lib/utils/port-marker.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import { filledMarkerStyle, portMarkerStyle } from './port-marker'; +import { filledMarkerStyle, portLooksRequired, portMarkerStyle } from './port-marker'; import type { PortDefinition } from '../types'; const port = (over: Partial = {}): PortDefinition => @@ -47,3 +47,20 @@ describe('port markers', () => { expect(out.style).toBe(filledMarkerStyle('#0af', 'port').style); }); }); + +describe('portLooksRequired', () => { + it('draws a required input required and an optional one optional, wired or not', () => { + for (const wired of [false, true]) { + expect(portLooksRequired(port({ required: true }), wired)).toBe(true); + expect(portLooksRequired(port(), wired)).toBe(false); + } + }); + + it('draws an input required only once wired as optional unwired and required wired', () => { + const p = port({ requiredWhenWired: true }); + expect(portLooksRequired(p, false)).toBe(false); + expect(portLooksRequired(p, true)).toBe(true); + expect(portMarkerStyle(p, none, none, '#0af', 'input', '', false)).toEqual(portMarkerStyle(port(), none, none, '#0af', 'input')); + expect(portMarkerStyle(p, none, none, '#0af', 'input', '', true)).toEqual(portMarkerStyle(port({ required: true }), none, none, '#0af', 'input')); + }); +}); diff --git a/packages/weft-graph/src/webview/lib/utils/port-marker.ts b/packages/weft-graph/src/webview/lib/utils/port-marker.ts index 19eae63f..1497b58d 100644 --- a/packages/weft-graph/src/webview/lib/utils/port-marker.ts +++ b/packages/weft-graph/src/webview/lib/utils/port-marker.ts @@ -11,6 +11,13 @@ import type { PortDefinition } from '../types'; */ export type PortMarkerState = 'full' | 'empty' | 'half' | 'empty-dotted'; +/** Whether an input draws as required: a required input always, and one + * required only once wired (`requiredWhenWired`) exactly while a wire + * feeds it, so its look follows connect and disconnect. */ +export function portLooksRequired(port: PortDefinition, isWired: boolean): boolean { + return port.required || (isWired && !!port.requiredWhenWired); +} + /** Pick the marker state for an input port. * A literal fill takes visual precedence over required/oneOfRequired: * if the port has a non-null body-set literal and no edge, it renders @@ -42,6 +49,7 @@ export function inputMarkerState( * - color: the port's type color * - side: 'input' (honors state) or 'output' (always full) * - extraClass: optional extra Tailwind utilities + * - isWired: whether a wire feeds this input (see `portLooksRequired`) */ export function portMarkerStyle( port: PortDefinition, @@ -50,10 +58,11 @@ export function portMarkerStyle( color: string, side: 'input' | 'output', extraClass: string = '', + isWired: boolean = false, ): { style: string; class: string } { // Outputs are always `full`, regardless of the port's `required` flag. const state: PortMarkerState = side === 'input' - ? inputMarkerState(port.required, oneOfRequiredPorts.has(port.name), literalFilledPorts.has(port.name)) + ? inputMarkerState(portLooksRequired(port, isWired), oneOfRequiredPorts.has(port.name), literalFilledPorts.has(port.name)) : 'full'; if (state === 'full') return filledMarkerStyle(color, 'port', extraClass); diff --git a/packages/weft-graph/src/webview/lib/utils/status.test.ts b/packages/weft-graph/src/webview/lib/utils/status.test.ts index 0a858dc0..f43a4eb7 100644 --- a/packages/weft-graph/src/webview/lib/utils/status.test.ts +++ b/packages/weft-graph/src/webview/lib/utils/status.test.ts @@ -41,6 +41,7 @@ describe('skipReasonText', () => { 'every input of the group (a, b) closed', ], [{ kind: 'scope_skipped', scope: 'triage' }, "the scope 'triage' it lives in did not run"], + [{ kind: 'wait_skipped' }, 'a person skipped what it was waiting for'], ]; it.each(cases)('renders %j', (reason, expected) => { diff --git a/packages/weft-graph/src/webview/lib/utils/status.ts b/packages/weft-graph/src/webview/lib/utils/status.ts index d5928af1..630002f0 100644 --- a/packages/weft-graph/src/webview/lib/utils/status.ts +++ b/packages/weft-graph/src/webview/lib/utils/status.ts @@ -66,6 +66,7 @@ export function skipReasonText(reason: SkipReason | undefined): string { case 'one_of_group_closed': return `every input of the group (${reason.ports.join(', ')}) closed${afterFailure(reason.failure)}`; case 'scope_skipped': return `the scope '${reason.scope}' it lives in did not run`; + case 'wait_skipped': return 'a person skipped what it was waiting for'; default: { // Compile-time exhaustiveness; at runtime (a dispatcher newer // than this webview sending a new kind) say so honestly diff --git a/packages/weft-graph/src/webview/lib/verb-gates.test.ts b/packages/weft-graph/src/webview/lib/verb-gates.test.ts index aee9fca8..d4d7cc84 100644 --- a/packages/weft-graph/src/webview/lib/verb-gates.test.ts +++ b/packages/weft-graph/src/webview/lib/verb-gates.test.ts @@ -133,6 +133,19 @@ describe('a registration older than the source', () => { }); }); +describe('an active project', () => { + // Every trigger on: the source has triggers, but nothing is left to + // turn on, so no second button (the bug was "Reactivate the rest" + // on every active project, which `activate` then refused). + it('offers no activate of its own, and reactivate only from the table', () => { + const on = inputs({ available: ['deactivate'], status: 'active', hasTriggers: true }); + expect(isVerbOffered('activate', on)).toBe(false); + expect(isVerbOffered('reactivate', on)).toBe(false); + const partly = inputs({ available: ['deactivate', 'reactivate'], status: 'active', hasTriggers: true }); + expect(isVerbOffered('reactivate', partly)).toBe(true); + }); +}); + describe('infra the user deleted from source', () => { it('is never offered a start, because there is no spec to start from', () => { // The live rows keep Stop and Terminate offered (the dispatcher diff --git a/packages/weft-graph/src/webview/lib/verb-gates.ts b/packages/weft-graph/src/webview/lib/verb-gates.ts index 20e3d806..7f945a6a 100644 --- a/packages/weft-graph/src/webview/lib/verb-gates.ts +++ b/packages/weft-graph/src/webview/lib/verb-gates.ts @@ -86,9 +86,13 @@ function starterGate(verb: ActionVerb, inputs: VerbInputs): boolean | undefined // the dispatcher does and the editor does not. So the source only // answers whether there is a trigger at all, and an activate with // infra down is refused by the door, naming the nodes to start. + // Only while the project is not active: once it is, which of its + // triggers are off is the dispatcher's to say (a trigger written + // since shows as drift, which resync puts live). Reactivate is + // never the source's: it brings back the work kept for triggers + // that are off, which only the dispatcher knows of. case 'activate': - case 'reactivate': - return inputs.hasTriggers; + return inputs.hasTriggers && inputs.status !== 'active'; default: return undefined; } diff --git a/tangle/claude-code/.claude/skills/weft-catalog/SKILL.md b/tangle/claude-code/.claude/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/claude-code/.claude/skills/weft-catalog/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/claude-code/.claude/skills/weft-consumers/SKILL.md b/tangle/claude-code/.claude/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/claude-code/.claude/skills/weft-consumers/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/claude-code/.claude/skills/weft-debugging/SKILL.md b/tangle/claude-code/.claude/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/claude-code/.claude/skills/weft-debugging/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/claude-code/.claude/skills/weft-editor/SKILL.md b/tangle/claude-code/.claude/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/claude-code/.claude/skills/weft-editor/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/claude-code/.claude/skills/weft-instances/SKILL.md b/tangle/claude-code/.claude/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/claude-code/.claude/skills/weft-instances/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/claude-code/.claude/skills/weft-members/SKILL.md b/tangle/claude-code/.claude/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/claude-code/.claude/skills/weft-members/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/claude-code/.claude/skills/weft-node-authoring/SKILL.md b/tangle/claude-code/.claude/skills/weft-node-authoring/SKILL.md index 8d6d984f..911d95ba 100644 --- a/tangle/claude-code/.claude/skills/weft-node-authoring/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/claude-code/.claude/skills/weft-sdp/SKILL.md b/tangle/claude-code/.claude/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/claude-code/.claude/skills/weft-sdp/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify diff --git a/tangle/cline/.cline/skills/weft-catalog/SKILL.md b/tangle/cline/.cline/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/cline/.cline/skills/weft-catalog/SKILL.md +++ b/tangle/cline/.cline/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/cline/.cline/skills/weft-consumers/SKILL.md b/tangle/cline/.cline/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/cline/.cline/skills/weft-consumers/SKILL.md +++ b/tangle/cline/.cline/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/cline/.cline/skills/weft-debugging/SKILL.md b/tangle/cline/.cline/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/cline/.cline/skills/weft-debugging/SKILL.md +++ b/tangle/cline/.cline/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/cline/.cline/skills/weft-editor/SKILL.md b/tangle/cline/.cline/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/cline/.cline/skills/weft-editor/SKILL.md +++ b/tangle/cline/.cline/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/cline/.cline/skills/weft-instances/SKILL.md b/tangle/cline/.cline/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/cline/.cline/skills/weft-instances/SKILL.md +++ b/tangle/cline/.cline/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/cline/.cline/skills/weft-members/SKILL.md b/tangle/cline/.cline/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/cline/.cline/skills/weft-members/SKILL.md +++ b/tangle/cline/.cline/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/cline/.cline/skills/weft-node-authoring/SKILL.md b/tangle/cline/.cline/skills/weft-node-authoring/SKILL.md index 8d6d984f..911d95ba 100644 --- a/tangle/cline/.cline/skills/weft-node-authoring/SKILL.md +++ b/tangle/cline/.cline/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/cline/.cline/skills/weft-sdp/SKILL.md b/tangle/cline/.cline/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/cline/.cline/skills/weft-sdp/SKILL.md +++ b/tangle/cline/.cline/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify diff --git a/tangle/codex/.agents/skills/weft-catalog/SKILL.md b/tangle/codex/.agents/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/codex/.agents/skills/weft-catalog/SKILL.md +++ b/tangle/codex/.agents/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/codex/.agents/skills/weft-consumers/SKILL.md b/tangle/codex/.agents/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/codex/.agents/skills/weft-consumers/SKILL.md +++ b/tangle/codex/.agents/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/codex/.agents/skills/weft-debugging/SKILL.md b/tangle/codex/.agents/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/codex/.agents/skills/weft-debugging/SKILL.md +++ b/tangle/codex/.agents/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/codex/.agents/skills/weft-editor/SKILL.md b/tangle/codex/.agents/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/codex/.agents/skills/weft-editor/SKILL.md +++ b/tangle/codex/.agents/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/codex/.agents/skills/weft-instances/SKILL.md b/tangle/codex/.agents/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/codex/.agents/skills/weft-instances/SKILL.md +++ b/tangle/codex/.agents/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/codex/.agents/skills/weft-members/SKILL.md b/tangle/codex/.agents/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/codex/.agents/skills/weft-members/SKILL.md +++ b/tangle/codex/.agents/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/codex/.agents/skills/weft-node-authoring/SKILL.md b/tangle/codex/.agents/skills/weft-node-authoring/SKILL.md index 8d6d984f..911d95ba 100644 --- a/tangle/codex/.agents/skills/weft-node-authoring/SKILL.md +++ b/tangle/codex/.agents/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/codex/.agents/skills/weft-sdp/SKILL.md b/tangle/codex/.agents/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/codex/.agents/skills/weft-sdp/SKILL.md +++ b/tangle/codex/.agents/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify diff --git a/tangle/cursor/.cursor/skills/weft-catalog/SKILL.md b/tangle/cursor/.cursor/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/cursor/.cursor/skills/weft-catalog/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/cursor/.cursor/skills/weft-consumers/SKILL.md b/tangle/cursor/.cursor/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/cursor/.cursor/skills/weft-consumers/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/cursor/.cursor/skills/weft-debugging/SKILL.md b/tangle/cursor/.cursor/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/cursor/.cursor/skills/weft-debugging/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/cursor/.cursor/skills/weft-editor/SKILL.md b/tangle/cursor/.cursor/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/cursor/.cursor/skills/weft-editor/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/cursor/.cursor/skills/weft-instances/SKILL.md b/tangle/cursor/.cursor/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/cursor/.cursor/skills/weft-instances/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/cursor/.cursor/skills/weft-members/SKILL.md b/tangle/cursor/.cursor/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/cursor/.cursor/skills/weft-members/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/cursor/.cursor/skills/weft-node-authoring/SKILL.md b/tangle/cursor/.cursor/skills/weft-node-authoring/SKILL.md index 8d6d984f..911d95ba 100644 --- a/tangle/cursor/.cursor/skills/weft-node-authoring/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/cursor/.cursor/skills/weft-sdp/SKILL.md b/tangle/cursor/.cursor/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/cursor/.cursor/skills/weft-sdp/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify diff --git a/tangle/devin-desktop/.windsurf/skills/weft-catalog/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-catalog/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/devin-desktop/.windsurf/skills/weft-consumers/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-consumers/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/devin-desktop/.windsurf/skills/weft-debugging/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-debugging/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/devin-desktop/.windsurf/skills/weft-editor/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-editor/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/devin-desktop/.windsurf/skills/weft-instances/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-instances/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/devin-desktop/.windsurf/skills/weft-members/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-members/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/devin-desktop/.windsurf/skills/weft-node-authoring/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-node-authoring/SKILL.md index c58baae2..51eac4ed 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-node-authoring/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/devin-desktop/.windsurf/skills/weft-sdp/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-sdp/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify diff --git a/tangle/fallback/.agents/skills/weft-catalog/SKILL.md b/tangle/fallback/.agents/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/fallback/.agents/skills/weft-catalog/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/fallback/.agents/skills/weft-consumers/SKILL.md b/tangle/fallback/.agents/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/fallback/.agents/skills/weft-consumers/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/fallback/.agents/skills/weft-debugging/SKILL.md b/tangle/fallback/.agents/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/fallback/.agents/skills/weft-debugging/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/fallback/.agents/skills/weft-editor/SKILL.md b/tangle/fallback/.agents/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/fallback/.agents/skills/weft-editor/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/fallback/.agents/skills/weft-instances/SKILL.md b/tangle/fallback/.agents/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/fallback/.agents/skills/weft-instances/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/fallback/.agents/skills/weft-members/SKILL.md b/tangle/fallback/.agents/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/fallback/.agents/skills/weft-members/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/fallback/.agents/skills/weft-node-authoring/SKILL.md b/tangle/fallback/.agents/skills/weft-node-authoring/SKILL.md index c58baae2..51eac4ed 100644 --- a/tangle/fallback/.agents/skills/weft-node-authoring/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/fallback/.agents/skills/weft-sdp/SKILL.md b/tangle/fallback/.agents/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/fallback/.agents/skills/weft-sdp/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify diff --git a/tangle/gemini-cli/.gemini/skills/weft-catalog/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-catalog/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/gemini-cli/.gemini/skills/weft-consumers/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-consumers/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/gemini-cli/.gemini/skills/weft-debugging/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-debugging/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/gemini-cli/.gemini/skills/weft-editor/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-editor/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/gemini-cli/.gemini/skills/weft-instances/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-instances/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/gemini-cli/.gemini/skills/weft-members/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-members/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/gemini-cli/.gemini/skills/weft-node-authoring/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-node-authoring/SKILL.md index c58baae2..51eac4ed 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-node-authoring/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/gemini-cli/.gemini/skills/weft-sdp/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-sdp/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify diff --git a/tangle/github-copilot/.github/skills/weft-catalog/SKILL.md b/tangle/github-copilot/.github/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/github-copilot/.github/skills/weft-catalog/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/github-copilot/.github/skills/weft-consumers/SKILL.md b/tangle/github-copilot/.github/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/github-copilot/.github/skills/weft-consumers/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/github-copilot/.github/skills/weft-debugging/SKILL.md b/tangle/github-copilot/.github/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/github-copilot/.github/skills/weft-debugging/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/github-copilot/.github/skills/weft-editor/SKILL.md b/tangle/github-copilot/.github/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/github-copilot/.github/skills/weft-editor/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/github-copilot/.github/skills/weft-instances/SKILL.md b/tangle/github-copilot/.github/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/github-copilot/.github/skills/weft-instances/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/github-copilot/.github/skills/weft-members/SKILL.md b/tangle/github-copilot/.github/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/github-copilot/.github/skills/weft-members/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/github-copilot/.github/skills/weft-node-authoring/SKILL.md b/tangle/github-copilot/.github/skills/weft-node-authoring/SKILL.md index 8d6d984f..911d95ba 100644 --- a/tangle/github-copilot/.github/skills/weft-node-authoring/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/github-copilot/.github/skills/weft-sdp/SKILL.md b/tangle/github-copilot/.github/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/github-copilot/.github/skills/weft-sdp/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify diff --git a/tangle/junie/.junie/skills/weft-catalog/SKILL.md b/tangle/junie/.junie/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/junie/.junie/skills/weft-catalog/SKILL.md +++ b/tangle/junie/.junie/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/junie/.junie/skills/weft-consumers/SKILL.md b/tangle/junie/.junie/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/junie/.junie/skills/weft-consumers/SKILL.md +++ b/tangle/junie/.junie/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/junie/.junie/skills/weft-debugging/SKILL.md b/tangle/junie/.junie/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/junie/.junie/skills/weft-debugging/SKILL.md +++ b/tangle/junie/.junie/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/junie/.junie/skills/weft-editor/SKILL.md b/tangle/junie/.junie/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/junie/.junie/skills/weft-editor/SKILL.md +++ b/tangle/junie/.junie/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/junie/.junie/skills/weft-instances/SKILL.md b/tangle/junie/.junie/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/junie/.junie/skills/weft-instances/SKILL.md +++ b/tangle/junie/.junie/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/junie/.junie/skills/weft-members/SKILL.md b/tangle/junie/.junie/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/junie/.junie/skills/weft-members/SKILL.md +++ b/tangle/junie/.junie/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/junie/.junie/skills/weft-node-authoring/SKILL.md b/tangle/junie/.junie/skills/weft-node-authoring/SKILL.md index 8d6d984f..911d95ba 100644 --- a/tangle/junie/.junie/skills/weft-node-authoring/SKILL.md +++ b/tangle/junie/.junie/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/junie/.junie/skills/weft-sdp/SKILL.md b/tangle/junie/.junie/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/junie/.junie/skills/weft-sdp/SKILL.md +++ b/tangle/junie/.junie/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify diff --git a/tangle/kilo-code/.kilo/skills/weft-catalog/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/kilo-code/.kilo/skills/weft-catalog/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/kilo-code/.kilo/skills/weft-consumers/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/kilo-code/.kilo/skills/weft-consumers/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/kilo-code/.kilo/skills/weft-debugging/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/kilo-code/.kilo/skills/weft-debugging/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/kilo-code/.kilo/skills/weft-editor/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/kilo-code/.kilo/skills/weft-editor/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/kilo-code/.kilo/skills/weft-instances/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/kilo-code/.kilo/skills/weft-instances/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/kilo-code/.kilo/skills/weft-members/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/kilo-code/.kilo/skills/weft-members/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/kilo-code/.kilo/skills/weft-node-authoring/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-node-authoring/SKILL.md index 8d6d984f..911d95ba 100644 --- a/tangle/kilo-code/.kilo/skills/weft-node-authoring/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/kilo-code/.kilo/skills/weft-sdp/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/kilo-code/.kilo/skills/weft-sdp/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify diff --git a/tangle/opencode/.opencode/skills/weft-catalog/SKILL.md b/tangle/opencode/.opencode/skills/weft-catalog/SKILL.md index fd6aaeb2..21b3006d 100644 --- a/tangle/opencode/.opencode/skills/weft-catalog/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-catalog/SKILL.md @@ -43,7 +43,7 @@ A broken node or package never breaks the catalog: it is left out, and only a pr Top-level keys: `type`, `label`, `description`, `tags`, `icon`, `color`, `inputs`, `outputs`, `types`, `requires_infra`, `images`, `publishes`, `service`, `portsFromConfig`, `features`, `display`, `validate`. -An input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` / `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). +An input entry: `name`, `type`, `required`, `requiredWhenWired` (optional unwired, required once wired), `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, and for `Access`-typed inputs `requiresScopes` and `requiresValues`. An output entry: `name`, `type`, `description`, and on an [infra node] `baked` (what that means is under the infra node shape below). `accepts` lists the drivers the port takes: `literal` (a value written in the source, in the braces or on its own line, `@file`/`@asset` included) and `wire` (a value another node produces). Absent means both; `["wire"]` means only a real node fills it (an LLM's `provider`, `params`, `historyFile`, `tools`; a consumer's `Access` handle). The list named in `portsFromConfig` is compiler-read: an inline typed value only, never a wire, never a marker. The access picker holds nothing in the source at all: its connection is picked on the install (`weft connect`). Exactly one driver per port. diff --git a/tangle/opencode/.opencode/skills/weft-consumers/SKILL.md b/tangle/opencode/.opencode/skills/weft-consumers/SKILL.md index fb3c7d7d..2390dfeb 100644 --- a/tangle/opencode/.opencode/skills/weft-consumers/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-consumers/SKILL.md @@ -65,7 +65,7 @@ any kind: they list, fire, skip, cancel and link the same way whatever the |---|---| | `GET /signal-token/signals` | the [signal]s this [api token] may see and that render a consumer payload, one [entry] each (shape below) | | `POST /signal/{signal token}` | fire one [signal] with a JSON body: answer a question, start a trigger's run, deliver a webhook's payload. No bearer: the [signal token] is the credential | -| `POST /signal/{signal token}/skip` | fire it with no payload: a parked question resumes unanswered (the node's outputs close and downstream skips). No bearer, same rule as firing | +| `POST /signal/{signal token}/skip` | decline a question a waiting run asked: the step that asked ends skipped, its outputs close and what reads them skips. Refused (409) on a trigger's [signal], which asks nothing. No bearer, same rule as firing | | `GET /signal-token/signals/{signal token}/files/{field}` | a fresh link for a stored file a [signal]'s field shows, good for an hour. You ask every time you render and never store the link | Two more, for a [consumer] that manages [signal]s: diff --git a/tangle/opencode/.opencode/skills/weft-debugging/SKILL.md b/tangle/opencode/.opencode/skills/weft-debugging/SKILL.md index db89e529..318bd5d8 100644 --- a/tangle/opencode/.opencode/skills/weft-debugging/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-debugging/SKILL.md @@ -51,7 +51,9 @@ node and the wrong value. group. `scope_skipped` names the group or loop whose `_should_flow` said no, taking this node with it: walk to that container. (A group input arriving closed is not this: it passes through to the nodes inside that - read it, and they carry their own reason.) + read it, and they carry their own reason.) `wait_skipped`: the node was + waiting on a person (a form, a question) and they skipped it, so it + ends with nothing sent on and what reads it skips: not a bug to chase. - **If a `Debug` shows `output=` empty**: correct, a `Debug` has no outputs. Its value is on its `node_started` line as `input=`, or `weft events --node `. diff --git a/tangle/opencode/.opencode/skills/weft-editor/SKILL.md b/tangle/opencode/.opencode/skills/weft-editor/SKILL.md index 01eaa635..6ba3d0cd 100644 --- a/tangle/opencode/.opencode/skills/weft-editor/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-editor/SKILL.md @@ -109,7 +109,11 @@ line) and the verb is not sent, so a build that cannot run never starts. "Running...". While following a live run it is "Stop Execution". - **Trigger slot** (when the source declares triggers): "Activate" / "Deactivate" / "Resync" (amber "Out of sync" when the project changed - since activation). An eye toggle "Show trigger subgraph". Deactivating + since activation). An eye toggle "Show trigger subgraph". When some + triggers are off while the others run (an infra stop took down the ones + reading it), a second button beside "Deactivate" turns them back on and + names them: "Activate gmail.watch", or "Reactivate ..." when runs wait on + them, two by name and then "and N more". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** (nothing dropped: everything that arrives waits and runs once the triggers are back), **Hibernate** (the same for a grace window, questions diff --git a/tangle/opencode/.opencode/skills/weft-instances/SKILL.md b/tangle/opencode/.opencode/skills/weft-instances/SKILL.md index 7bf5ea3c..8f94a568 100644 --- a/tangle/opencode/.opencode/skills/weft-instances/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-instances/SKILL.md @@ -225,17 +225,21 @@ weft never starts an instance's container by itself: the program does, or the user with `--instance` on the infra verbs. A container runs from images that `weft activate` builds and records for every node marked `@per_instance`, so activate the program before it starts any container; without that, the start -fails naming `weft activate`. The instance nodes read the names of infra -nodes and triggers you write in them at compile time (a name that is not an -infra node, or not per instance, is refused then); a name arriving on a wire -is checked only when the run reaches it. +fails naming `weft activate`. `StartInfra`, `StopInfra`, `TerminateInfra`, +`ActivateTriggers` and `DeactivateTriggers` act on the instance you give +them in `instance`; left unwired, they act on the program's own copies (the +shared container of a node not marked `@per_instance`, the shared triggers). +The instance nodes read the names of infra nodes and triggers you write in +them at compile time (a name that is not an infra node, or not the kind the +`instance` input calls for, is refused then); a name arriving on a wire is +checked only when the run reaches it. Creating an instance, from a gated route the backend calls: `MintInstanceToken` (if a browser will act inside it), then, if the program -has a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` +has a per-instance infra node, `StartInfra` with `waitUntilRunning` off (it fires `done` as soon as weft accepts the start), then `Reply` with the -token so the request ends at once, then a second `StartInstanceInfra` left on -(it fires `done` once the container runs), then `ActivateInstanceTriggers`. +token so the request ends at once, then a second `StartInfra` left on +(it fires `done` once the container runs), then `ActivateTriggers`. Never put the `Reply` before the first start: a caller that lists the instance's containers right after the answer would find nothing yet, while a start weft accepted already reads `provisioning`. A container can take minutes to come up, so the @@ -254,8 +258,8 @@ every container it ever started keeps running and costing. The shape: 1. every run for an instance writes a last-used time for it in the program's own database (one row per instance id); 2. a `Cron` in the program reads the instances idle past a limit and stops - them (`StopInstanceInfra`, which keeps the disks) or removes them - (`TerminateInstanceInfra`, or `WipeInstance` for everything); + them (`StopInfra`, which keeps the disks) or removes them + (`TerminateInfra`, or `WipeInstance` for everything); 3. the limit is the user's choice: ask them how long an idle instance lives before it is stopped, and whether it is then deleted, before you write it. @@ -309,7 +313,7 @@ instance's triggers right after it. If the container never comes up, `start()` fails with the reason, and so it does if somebody stops the container while it waits (it never starts it again behind a pause). `request_start()` returns as soon as weft accepts the start, without waiting -for the container (`StartInstanceInfra` with `waitUntilRunning` off); a start +for the container (`StartInfra` with `waitUntilRunning` off); a start refused for a passing reason, such as a build in progress, is asked again until it is accepted. Every reader of a container's state (`weft status`, `weft infra status`, @@ -347,7 +351,7 @@ started as `not started`), each instance's container under `instance infra:`, and each instance's triggers under `triggers:`, with the events a trigger holds for a missing field. A plain `weft activate` turns on the shared triggers and names the per-instance ones it left off; `weft activate ---instance ` (or `ActivateInstanceTriggers`) turns those on. `weft +--instance ` (or `ActivateTriggers` given the instance) turns those on. `weft executions --instance ` lists an instance's runs. To try the per-instance path yourself, start a container (`weft infra start --instance test-1`), turn on its triggers (`weft activate --instance test-1`), and run with `--instance diff --git a/tangle/opencode/.opencode/skills/weft-members/SKILL.md b/tangle/opencode/.opencode/skills/weft-members/SKILL.md index c0687387..39cb2322 100644 --- a/tangle/opencode/.opencode/skills/weft-members/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-members/SKILL.md @@ -67,9 +67,9 @@ into that group through the group's inputs. 2. **An admin route** the frontend's server calls, gated by `ApiKeyAuth` on the `Route`'s `auth`, its key only in that server's environment. It onboards a person: writes the row, `MintInstanceToken`, then, if there is - a per-instance infra node, `StartInstanceInfra` with `waitUntilRunning` off, + a per-instance infra node, `StartInfra` with `waitUntilRunning` off, then `Reply` with the token so the request ends at once, then a second - `StartInstanceInfra` left on and `ActivateInstanceTriggers`. A `Reply` + `StartInfra` left on and `ActivateTriggers`. A `Reply` before the start would leave the person's container out of a list read right after it. The same route (or siblings on the same gate) lists a person's instances, mints a token diff --git a/tangle/opencode/.opencode/skills/weft-node-authoring/SKILL.md b/tangle/opencode/.opencode/skills/weft-node-authoring/SKILL.md index 8d6d984f..911d95ba 100644 --- a/tangle/opencode/.opencode/skills/weft-node-authoring/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-node-authoring/SKILL.md @@ -126,8 +126,19 @@ Unknown keys are a loud parse error. Top level: | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | -Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, -`label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. +Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, +`widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, +`requiresValues`. + +If an unwired input means something of its own (no `instance`: the program's +own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; +wired, a wire that delivers nothing skips the node instead of falling back to +that meaning. If an optional input needs one more permission when it is used, +leave it out of `requiresScopes`, which every program using the node is asked +for. In `run`, when the input is used and before `ctx.client(&account)`, copy +`account.required_permissions()` (it already holds the declared ones), push the +extra one, and pass the list to `account.with_required_permissions`. Opening +then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when @@ -170,7 +181,7 @@ nodes by a feature they declare, with `with: {feature: value}`. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads -'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInstanceInfra` +'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot @@ -387,7 +398,7 @@ and end in one call: The types are in `weft::program` (`InfraCopy`, `CostRecord`, `PaidBy`, `MintedInstanceToken`). Every call is journaled, so none goes through `ctx.run`. The `instances` package already wraps most of them (`CurrentInstance`, -`StartInstanceInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before +`StartInfra`, `ListInstanceInfra`, `InstanceCosts`, `SetInstanceValues`, ...), so check it before writing one. In the `fake` tier, `rig.instance("user-42")` makes the run a run for that instance, `rig.answer_program_call("weft.infra.status", json!(..))` queues the answer to one call by its journal name (`weft.infra.copies`, @@ -401,8 +412,12 @@ plus a `service` recipe in metadata (acquisition fields with `secret: true`, auth steps, a test URL, an identity template). The macro reads the `account` input and pulses it on `access`. It has no `tests.rs`: the macro is the whole body, so there is nothing of yours to test, and the review does not ask for -one. Credentials live sealed in the runtime's access store, never in the -project. The compiler synthesizes the runtime +one. The test URL must pass whatever permissions the person ticked: when it +needs a permission of its own to learn who the account is, declare that +permission with `"always": true` and it is asked for on every connection +(Google's `openid` and `userinfo.email`; the service docs' "Permissions" +section has the rules). Credentials live sealed in the runtime's access store, +never in the project. The compiler synthesizes the runtime "no connection picked" rule from the `service` block; a hand-written one is a finding. `"connection_optional": true` inside the service block is reserved for a node that genuinely runs unconnected. That node is the one diff --git a/tangle/opencode/.opencode/skills/weft-sdp/SKILL.md b/tangle/opencode/.opencode/skills/weft-sdp/SKILL.md index 350a75ea..bfe57f3c 100644 --- a/tangle/opencode/.opencode/skills/weft-sdp/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-sdp/SKILL.md @@ -263,13 +263,30 @@ output evidence together. ## Read outside interactions -A [frozen example] records human questions and answers and incoming caller -messages for inspection; the runtime does not replay them on its own. When -a run waits on a person, you compare the current question with the -recorded question and answer, answer the current token within the user's -authority, and keep inspecting that same run. For live connections, you -send the recorded messages through a new connection and review the new -responses. +A [frozen example] records how each wait of the run ended, with the +question the node showed, and the incoming caller messages, for inspection; +the runtime does not replay them on its own. When a run waits on a person, +you compare the current question with the recorded one, end the wait the +way the recorded entry says within the user's authority, and keep +inspecting that same run. Each entry holds exactly one of these: + +- `payload`: the value the person answered with. Answer the current token + with it. +- `"skipped": true`: the person skipped the question. Skip it the same way, + `POST /signal/{token}/skip`. +- `gave_up`: nobody answered. The run could not pause, held the wait in its + worker until nothing had moved for its hold time, and gave the wait up; + the text is the error the waiting step failed with. Leave the current + one unanswered, so the new run gives up the same way when its own hold + runs out. + +The entries cover every wait that ended, including timers and nodes that +poll a job, not only questions to a person. You answer only the waits that +ask a person: timers and polls run on their own, and answering one by hand +would cut a real wait short. + +For live connections, you send the recorded messages through a new +connection and review the new responses. `weft wake ` resolves a pure time wait. A wait requiring a value must receive that value instead. Logs and inherited markers identify From 87a203700532ae8149cc50c470d8dde11b917364 Mon Sep 17 00:00:00 2001 From: Quentin Date: Sun, 11 Oct 2026 11:53:55 +0200 Subject: [PATCH 2/2] Let a run waiting on a person leave the database asleep on Google Cloud - An answer or a cancel for a running run is written with a news entry in the same transaction; a separate teller publishes it on the install's news bus (Pub/Sub on GCP, nothing locally, where the database's own signal covers it) and retries until it goes out. - Broker processes hear the news and pass it to the worker down the line it already holds, answers the same way as cancels; the held answers request and its periodic re-reads are gone. - While every run on a worker only waits, the worker stops ticking and vouches for its runs until the first wait would give up; the reaper sleeps until the earliest lease could lapse. - A worker whose runs all only wait stops itself before the broker's request limit cuts its line, handing back what can pause and ending what cannot with the limit named; a busy worker reconnects as before. - The dispatcher and the broker no longer hold their database listening busy while a request is held, so a quiet watch lets the database sleep. - CHANGELOG names this release. --- CHANGELOG.md | 17 +- Cargo.lock | 5 +- crates/weft-broker-client/src/client.rs | 103 +-- crates/weft-broker-client/src/line.rs | 112 +++- crates/weft-broker-client/src/protocol.rs | 34 +- crates/weft-broker/Cargo.toml | 2 +- crates/weft-broker/src/door.rs | 10 +- crates/weft-broker/src/handlers.rs | 15 +- crates/weft-broker/src/lib.rs | 1 + crates/weft-broker/src/line.rs | 197 ++++-- crates/weft-broker/src/news.rs | 131 ++++ crates/weft-broker/src/records.rs | 30 +- crates/weft-broker/src/state.rs | 9 +- crates/weft-core/src/deadline.rs | 26 + crates/weft-core/src/lib.rs | 2 + crates/weft-dispatcher/Cargo.toml | 2 +- crates/weft-dispatcher/src/api/mod.rs | 12 +- crates/weft-dispatcher/src/delivery.rs | 19 +- .../weft-dispatcher/src/journal/postgres.rs | 24 +- crates/weft-dispatcher/src/parked_drain.rs | 14 +- crates/weft-dispatcher/src/proxy.rs | 7 + crates/weft-dispatcher/src/reaper.rs | 73 ++- crates/weft-dispatcher/src/worker_door.rs | 7 +- crates/weft-dispatcher/tests/db_lifecycle.rs | 98 +++ crates/weft-dispatcher/tests/db_proxy.rs | 62 ++ crates/weft-engine/src/context.rs | 35 +- crates/weft-engine/src/door/limits.rs | 15 +- crates/weft-engine/src/door/mod.rs | 48 +- crates/weft-engine/src/door/ticks.rs | 152 ++++- crates/weft-engine/src/execution_driver.rs | 96 ++- .../src/execution_driver_tests/born.rs | 2 + .../src/execution_driver_tests/rig.rs | 71 ++- .../src/execution_driver_tests/unrecorded.rs | 105 +++- crates/weft-engine/src/lib.rs | 1 + crates/weft-engine/src/run_holds.rs | 150 +++++ crates/weft-engine/src/test_rig.rs | 1 + crates/weft-engine/src/worker.rs | 586 ++++++++++++++++-- crates/weft-infra-supervisor/src/lifecycle.rs | 6 +- .../tests/lifecycle_loop.rs | 2 +- crates/weft-platform-gcp/src/api.rs | 4 + crates/weft-platform-gcp/src/lib.rs | 7 +- crates/weft-platform-gcp/src/news.rs | 466 ++++++++++++++ crates/weft-platform-gcp/src/runner.rs | 9 +- crates/weft-platform-local/src/lib.rs | 3 + crates/weft-platform-local/src/news.rs | 34 + crates/weft-platform-traits/src/config.rs | 4 + crates/weft-platform-traits/src/lib.rs | 6 + crates/weft-platform-traits/src/news.rs | 140 +++++ crates/weft-runtime/Cargo.toml | 3 +- crates/weft-runtime/src/lib.rs | 25 +- crates/weft-runtime/src/platform.rs | 25 + crates/weft-runtime/src/server.rs | 42 -- crates/weft-task-store/Cargo.toml | 11 +- ...261010T121746_announcement_news_outbox.sql | 26 + ...0261010T121746_parked_fire_news_outbox.sql | 18 + crates/weft-task-store/src/announce.rs | 232 ++++++- crates/weft-task-store/src/parked_fires.rs | 27 +- crates/weft-task-store/src/pg_signal.rs | 141 ++++- crates/weft-task-store/src/runs.rs | 30 +- crates/weft-task-store/src/terminal.rs | 2 +- crates/weft-task-store/src/worker_door.rs | 38 +- crates/weft-task-store/tests/news_outbox.rs | 146 +++++ crates/weft-task-store/tests/signals.rs | 71 +++ deploy/terraform/gcp/news.tf | 55 ++ deploy/terraform/gcp/serverless.tf | 6 +- deploy/terraform/gcp/versions.tf | 1 + docs/src/running/architecture.md | 50 +- docs/src/running/cloud.md | 17 +- 68 files changed, 3453 insertions(+), 468 deletions(-) create mode 100644 crates/weft-broker/src/news.rs create mode 100644 crates/weft-core/src/deadline.rs create mode 100644 crates/weft-dispatcher/tests/db_proxy.rs create mode 100644 crates/weft-engine/src/run_holds.rs create mode 100644 crates/weft-platform-gcp/src/news.rs create mode 100644 crates/weft-platform-local/src/news.rs create mode 100644 crates/weft-platform-traits/src/news.rs create mode 100644 crates/weft-task-store/migrations/announcement/20261010T121746_announcement_news_outbox.sql create mode 100644 crates/weft-task-store/migrations/parked_fire/20261010T121746_parked_fire_news_outbox.sql create mode 100644 crates/weft-task-store/tests/news_outbox.rs create mode 100644 deploy/terraform/gcp/news.tf diff --git a/CHANGELOG.md b/CHANGELOG.md index 13adc862..567c1a01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,22 @@ What changed between versions of weft that you need to know about when you update. To update, pull and run `./setup.sh` again. -## Unreleased +## 2026-10-11 (0.2.267) + +**On Google Cloud, a run waiting on a person lets the database sleep.** + +A run that cannot pause (a route still answering its caller, a trigger with +`recorded: false`, a live stream between steps) keeps its question in its +worker while it waits. Until now that kept the database awake for the whole +wait. Now nothing touches the database until the answer or a cancel arrives: +the answer reaches the worker through a news topic weft creates in your +Google project. Upgrade as usual (merge upstream into your fork and run the +install workflow again): it creates the topic and the rights that go with it. + +A worker on Google Cloud can keep one connection to weft for at most an +hour. If every run on it is still waiting at that point, the worker stops: +runs that can pause carry on elsewhere, and a run that cannot pause ends, +saying which limit ended it. To wait longer, make the run one that can pause. **Five instance nodes lost "Instance" from their names.** diff --git a/Cargo.lock b/Cargo.lock index bbfaca55..165a40ea 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5185,7 +5185,6 @@ dependencies = [ "clap", "futures", "hex", - "http-body 1.0.1", "parking_lot", "reqwest", "serde_json", @@ -5214,17 +5213,21 @@ version = "0.2.0" dependencies = [ "anyhow", "async-trait", + "axum", "chrono", "futures", + "http-body 1.0.1", "serde", "serde_json", "sha2 0.10.9", "sqlx", "tokio", "tokio-util", + "tower", "tracing", "uuid", "weft-core", + "weft-platform-traits", ] [[package]] diff --git a/crates/weft-broker-client/src/client.rs b/crates/weft-broker-client/src/client.rs index 9cc3fc97..8dd35772 100644 --- a/crates/weft-broker-client/src/client.rs +++ b/crates/weft-broker-client/src/client.rs @@ -46,8 +46,8 @@ impl HttpCore { Self::parse_success(answer, path) } - /// A call the broker holds open for up to `hold` (never more than - /// `MAX_HOLD`, which it enforces). The hold starts once the call is + /// A call the broker holds open for up to `hold`, however long that is: + /// a line has no idle timeout. The hold starts once the call is /// written, which may take up to the usual wait to send, so the line /// waits that, the hold and [`HOLD_GRACE`], and never gives up before /// the broker's own hold ends. @@ -58,7 +58,7 @@ impl HttpCore { hold: Duration, ) -> Result { let to_send = DEFAULT_CALL_WAIT.to_send; - let wait = CallWait { to_send, to_answer: to_send + hold + HOLD_GRACE }; + let wait = CallWait { to_send, to_answer: to_send.saturating_add(hold).saturating_add(HOLD_GRACE) }; let answer = self.post_raw(path, body, wait).await?; Self::parse_success(answer, path) } @@ -171,24 +171,11 @@ impl WriteOutcome { } } -/// Wait up to `wait` in holds the broker grants (each at most -/// `MAX_HOLD`, which it enforces), asking again after a hold that ended -/// without the answer `done` looks for. Hands back the last answer. A -/// `hold` that waits out an unavailable broker ([`read_until_answered`]) -/// stretches past `wait` by as long as the broker stays down. -async fn held(wait: Duration, mut hold: F, done: impl Fn(&T) -> bool) -> Result -where - F: FnMut(Duration) -> Fut, - Fut: std::future::Future>, -{ - let deadline = tokio::time::Instant::now() + wait; - loop { - let left = deadline.saturating_duration_since(tokio::time::Instant::now()); - let answer = hold(left.min(weft_task_store::pg_signal::MAX_HOLD)).await?; - if done(&answer) || left <= weft_task_store::pg_signal::MAX_HOLD { - return Ok(answer); - } - } +/// `wait` as a held call's `wait_ms`, in milliseconds. A wait too long to +/// count in them is sent as the most they hold, which the broker reads as +/// waiting for good (`weft_core::deadline::after`). +fn wait_ms(wait: Duration) -> u64 { + u64::try_from(wait.as_millis()).unwrap_or(u64::MAX) } /// Whether the broker answered that the call did nothing at all @@ -367,17 +354,9 @@ impl TaskStoreClient for BrokerTaskStoreClient { } async fn wait_for_terminal(&self, task_id: Uuid, timeout: Duration) -> Result { - held( - timeout, - |hold| async move { - let req = TaskWaitTerminalRequest { task_id, wait_ms: hold.as_millis() as u64 }; - let resp: TaskWaitTerminalResponse = - self.http.post_held("/v1/task/wait_terminal", &req, hold).await?; - Ok(resp.into_outcome()) - }, - |outcome| outcome.status.is_terminal(), - ) - .await + let req = TaskWaitTerminalRequest { task_id, wait_ms: wait_ms(timeout) }; + let resp: TaskWaitTerminalResponse = self.http.post_held("/v1/task/wait_terminal", &req, timeout).await?; + Ok(resp.into_outcome()) } } @@ -683,19 +662,19 @@ impl BrokerRunClient { } /// The answers waiting for `execution_id`'s waits, but the waits in - /// `taken`, held up to `wait` until one is there (`RunAnswersRequest`). - pub async fn answers(&self, execution_id: ExecutionId, taken: &[String], wait: Duration) -> Result> { + /// `taken`, read once (`RunAnswersRequest`). + pub async fn answers(&self, execution_id: ExecutionId, taken: &[String]) -> Result> { let path = "/v1/run/answers"; - held( - wait, - |hold| async move { - let req = RunAnswersRequest { execution_id, wait_ms: hold.as_millis() as u64, taken: taken.to_vec() }; - let resp: RunAnswersResponse = read_until_answered(path, || self.http.post_held(path, &req, hold)).await?; - Ok(resp.answers) - }, - |answers| !answers.is_empty(), - ) - .await + let req = RunAnswersRequest { execution_id, taken: taken.to_vec() }; + let resp: RunAnswersResponse = read_until_answered(path, || self.http.post(path, &req)).await?; + Ok(resp.answers) + } + + /// What the broker pushes about the runs this worker drives: an answer + /// handed to one (`crate::line::PARKED_FIRE_CHANNEL`), and when the line + /// may have missed some (see [`BrokerLink::subscribe`]). + pub fn told(&self) -> weft_task_store::pg_signal::Subscription { + self.http.link.subscribe() } /// The cancels waiting for the runs this worker drives (`RunCancelsRequest`). @@ -870,21 +849,12 @@ impl BrokerSupervisorClient { busy_commands: &[i64], wait: Duration, ) -> Result { - held( - wait, - |hold| { - let req = SupervisorClaimCommandRequest { - claimer_replica: claimer_replica.to_string(), - busy_commands: busy_commands.to_vec(), - wait_ms: hold.as_millis() as u64, - }; - async move { - self.http.post_held("/v1/supervisor/claim_command", &req, hold).await - } - }, - |claim| !matches!(claim, SupervisorClaim::Nothing), - ) - .await + let req = SupervisorClaimCommandRequest { + claimer_replica: claimer_replica.to_string(), + busy_commands: busy_commands.to_vec(), + wait_ms: wait_ms(wait), + }; + self.http.post_held("/v1/supervisor/claim_command", &req, wait).await } pub async fn event_record( @@ -1197,19 +1167,8 @@ impl BrokerInfraStateClient { command_id: i64, wait: Duration, ) -> Result { - held( - wait, - |hold| { - let req = InfraWaitApplyRequest { - project_id, - command_id, - wait_ms: hold.as_millis() as u64, - }; - async move { self.http.post_held("/v1/infra/wait_apply", &req, hold).await } - }, - |resp: &InfraWaitApplyResponse| resp.completed, - ) - .await + let req = InfraWaitApplyRequest { project_id, command_id, wait_ms: wait_ms(wait) }; + self.http.post_held("/v1/infra/wait_apply", &req, wait).await } } diff --git a/crates/weft-broker-client/src/line.rs b/crates/weft-broker-client/src/line.rs index a26194b2..e2162006 100644 --- a/crates/weft-broker-client/src/line.rs +++ b/crates/weft-broker-client/src/line.rs @@ -26,6 +26,14 @@ //! heard, and the subscription says so ([`Heard::Lost`]) so a copy kept on //! it stops trusting itself until the line is back ([`Heard::Recheck`]). //! +//! A broker that cuts a request after a while (Cloud Run's limit on one +//! request) says so as the line opens (`Notice::Following`'s +//! `max_age_secs`), and the line tells its process when that cut comes +//! ([`BrokerLink::cut_at`]). The line is never renewed: a cut is a break +//! like any other, except for a worker whose runs only wait, which must +//! not read again and so stops before then; the process that starts in its +//! place opens a line of its own. +//! //! When the line breaks it comes back by itself. A call that had not been //! written yet goes out on the next line if that opens within the call's //! wait to send, and fails [`LineError::NotSent`] otherwise. One that had @@ -47,7 +55,8 @@ //! zero up for as long as that process lives. While it is closed nothing //! is heard, so what is kept on it is not trusted until it is back. A //! process that must hear while it asks nothing (a worker driving runs, -//! which hears their cancels) holds it open ([`BrokerLink::stay_open`]). +//! which hears their answers and cancels) holds it open +//! ([`BrokerLink::stay_open`]). //! //! Byte streams (`ctx.storage` uploads and downloads) stay ordinary //! requests: a large body on the line would hold up every call behind it. @@ -178,16 +187,26 @@ pub enum Notice { /// Sent by the caller: it stopped waiting for the call `id`, so the /// broker stops working on it. Forget { id: u64 }, - /// The broker follows for this caller; `listening` is whether its own - /// database listener hears right now. - Following { listening: bool }, + /// The broker follows for this caller. `listening` is whether its own + /// database listener hears right now; `max_age_secs` how long the + /// broker keeps one line before cutting it (a request limit), `None` + /// when it keeps it for good. + Following { listening: bool, max_age_secs: Option }, /// One notification this caller may see. Signal { channel: String, payload: String }, /// The broker's database listener lost its connection: nothing is - /// heard until a `Recheck`. + /// heard until a `Recheck` that says it listens. Lost, - /// Notifications may have been missed: look again. - Recheck, + /// Notifications may have been missed: look again. `listening` is + /// whether the broker's database listener hears right now: a recheck + /// may come while it does not (the install's news, heard while the + /// database sleeps), and then nothing kept on the line is trusted yet. + Recheck { listening: bool }, + /// The broker's database listener hears again after a quiet spell. What + /// runs wait on (their answers, their cancels) reached it meanwhile as + /// the install's news, so nothing needs looking at again; only what is + /// kept on the line may follow again. + Resumed, } /// Frame `head` and `body` for the line. @@ -211,8 +230,8 @@ pub fn unframe Deserialize<'de>>(bytes: &[u8]) -> anyhow::Result<(H, /// The notification channels a line may carry, by name: what turns a /// name the broker sent back into the `&'static str` a [`Heard`] holds. -// SYNC: LINE_CHANNELS <-> crates/weft-broker/src/line.rs (audience), crates/weft-task-store/src/runs.rs (CANCEL_CHANNEL), triggers_notify_project, crates/weft-dispatcher/src/infra_node.rs (infra_node_status_notify), crates/weft-dispatcher/src/project_store.rs (project_declared_infra_notify), crates/weft-access-store/src/lib.rs (access_notify) -pub const LINE_CHANNELS: &[&str] = &[INFRA_STATUS_CHANNEL, ACCESS_CHANNEL, CANCEL_CHANNEL, TRIGGERS_CHANNEL]; +// SYNC: LINE_CHANNELS <-> crates/weft-broker/src/line.rs (audience), crates/weft-task-store/src/runs.rs (CANCEL_CHANNEL), crates/weft-task-store/src/parked_fires.rs (PARKED_FIRE_CHANNEL), triggers_notify_project, crates/weft-dispatcher/src/infra_node.rs (infra_node_status_notify), crates/weft-dispatcher/src/project_store.rs (project_declared_infra_notify), crates/weft-access-store/src/lib.rs (access_notify) +pub const LINE_CHANNELS: &[&str] = &[INFRA_STATUS_CHANNEL, ACCESS_CHANNEL, CANCEL_CHANNEL, TRIGGERS_CHANNEL, PARKED_FIRE_CHANNEL]; /// Announced with a tenant's id when one of its routes changes: a public /// entry armed, re-armed or taken down, or how its activation stands (the @@ -244,6 +263,12 @@ pub const ACCESS_CHANNEL: &str = "weft_access"; /// (`weft_task_store::runs`). pub use weft_task_store::runs::CANCEL_CHANNEL; +/// Announced with ` ` when an answer is handed to +/// a running execution (and with a project's id alone when an event waits +/// for its trigger, which rides no line): the worker driving the run reads +/// its answers (`weft_task_store::parked_fires`). +pub use weft_task_store::parked_fires::PARKED_FIRE_CHANNEL; + /// A process's way to the broker. Every broker client of the process /// holds the same one, so the process has one line, opened on the first /// call and kept. @@ -347,6 +372,13 @@ impl BrokerLink { Subscription::with_listening(line.notices.subscribe(), line.listening.clone()) } + /// When the broker cuts the line open now, `None` while it is closed + /// or when the broker keeps it for good. Told again whenever that + /// changes, so a process can stop before the cut (see the module doc). + pub fn cut_at(&self) -> tokio::sync::watch::Receiver> { + self.line().cut.subscribe() + } + /// Hold the line open while the guard lives, calls or not, so what the /// broker pushes is heard (see the module doc). A line closed for want /// of anything to do opens again at once. @@ -423,6 +455,16 @@ struct Line { listening: Arc, /// Why the line is down, while it is. down: Mutex>, + /// When the broker cuts the open line, if it said it would. + cut: tokio::sync::watch::Sender>, +} + +/// When the broker cuts the line open now (`Notice::Following`'s +/// `max_age_secs`): `max_age` after it `opened`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Cut { + pub opened: tokio::time::Instant, + pub max_age: Duration, } impl Line { @@ -437,6 +479,7 @@ impl Line { notices, listening: Arc::new(AtomicBool::new(false)), down: Mutex::new(Some("not opened yet".into())), + cut: tokio::sync::watch::channel(None).0, }); let task = keep_open(link, Arc::downgrade(&line), ids, closed, idle_after); match runtime { @@ -467,7 +510,7 @@ impl Line { }; tokio::select! { result = &mut answered => return landed(result), - () = tokio::time::sleep_until(start + to_send) => {} + () = tokio::time::sleep_until(weft_core::deadline::after(start, to_send)) => {} } { // Under the lock the writer takes frames with, so a call is @@ -481,7 +524,7 @@ impl Line { }); } } - match tokio::time::timeout_at(start + wait.to_answer, &mut answered).await { + match tokio::time::timeout_at(weft_core::deadline::after(start, wait.to_answer), &mut answered).await { Ok(result) => landed(result), Err(_) => Err(LineError::NoAnswer(wait.to_answer)), } @@ -491,6 +534,7 @@ impl Line { /// the unwritten ones wait for the next line. fn broke(&self, why: &str) { self.listening.store(false, Ordering::Release); + self.cut.send_replace(None); let _ = self.notices.send(Heard::Lost); *self.down.lock().expect("line down") = Some(why.to_string()); let mut calls = self.calls.lock().expect("line calls"); @@ -506,6 +550,7 @@ impl Line { /// it opens again. fn idled(&self) { self.listening.store(false, Ordering::Release); + self.cut.send_replace(None); let _ = self.notices.send(Heard::Lost); *self.down.lock().expect("line down") = Some("closed while idle; it opens again on the next call".into()); } @@ -528,13 +573,31 @@ impl Line { self.calls.lock().expect("line calls").get_mut(&id).and_then(|w| w.frame.take()) } - fn heard(&self, notice: Notice) { + /// What the broker said on the line, which `opened` at that instant. + fn heard(&self, notice: Notice, opened: tokio::time::Instant) { match notice { - Notice::Following { listening: true } | Notice::Recheck => { - self.listening.store(true, Ordering::Release); + // The line follows again, or may have missed some: every waiter + // looks again (a cancel announced while it did not follow is + // found so), and copies kept on it are trusted only while the + // broker hears. Set before the recheck goes out, so a waiter + // reads it with the recheck. + Notice::Following { listening, max_age_secs } => { + self.cut.send_replace(max_age_secs.map(|secs| Cut { opened, max_age: Duration::from_secs(secs) })); + self.listening.store(listening, Ordering::Release); + let _ = self.notices.send(Heard::Recheck); + } + Notice::Recheck { listening } => { + self.listening.store(listening, Ordering::Release); let _ = self.notices.send(Heard::Recheck); } - Notice::Following { listening: false } | Notice::Lost => { + // The broker hears again after a quiet spell, in which what + // runs wait on reached it as the install's news: nothing to + // look at again, but what is kept on the line may follow again. + Notice::Resumed => { + self.listening.store(true, Ordering::Release); + let _ = self.notices.send(Heard::Resumed); + } + Notice::Lost => { self.listening.store(false, Ordering::Release); let _ = self.notices.send(Heard::Lost); } @@ -570,6 +633,9 @@ async fn keep_open( // opens: the last link let go of mid-attempt closes the line. let (base_url, token) = (inner.base_url.clone(), inner.token.clone()); drop(inner); + // The broker's limit on one request counts from no earlier than + // this. + let opening = tokio::time::Instant::now(); let opened = tokio::select! { opened = tokio::time::timeout(CONNECT_WAIT, connect(&base_url, &token)) => { opened.unwrap_or_else(|_| Err(anyhow::anyhow!("opening it took more than {CONNECT_WAIT:?}"))) @@ -580,7 +646,7 @@ async fn keep_open( Ok(socket) => { retry = RECONNECT_FIRST; *held.down.lock().expect("line down") = None; - match serve(socket, &held, first.take(), &mut outbox, &mut closed, idle_after).await { + match serve(socket, opening, &held, first.take(), &mut outbox, &mut closed, idle_after).await { Served::LetGo => return, Served::Broke(why) => { tracing::warn!(target: "weft_broker_client::line", reason = %why, "the line to the broker broke; opening it again"); @@ -663,14 +729,15 @@ fn line_url(base_url: &str) -> anyhow::Result { Ok(format!("{rest}{LINE_PATH}")) } -/// Serve one open line until it breaks, or until the process lets go of -/// its last link (`closed`). +/// Serve one open line, which `opened` at that instant, until it breaks, +/// or until the process lets go of its last link (`closed`). /// /// Every write is bounded by [`PONG_WAIT`]: a broker that vanished without /// closing leaves the socket's buffer full and a write waiting for minutes, /// which would also keep the ping's deadline from ever being looked at. async fn serve( socket: Socket, + opened: tokio::time::Instant, line: &Line, first: Option, outbox: &mut mpsc::UnboundedReceiver, @@ -740,7 +807,7 @@ async fn serve( }, Some(Ok(Message::Text(text))) => match serde_json::from_str::(&text) { Ok(notice) => { - line.heard(notice); + line.heard(notice, opened); Ok(()) } Err(e) => Err(format!("a notice that does not read ({text}): {e}")), @@ -818,10 +885,11 @@ mod tests { for notice in [ Notice::Follow, Notice::Forget { id: 7 }, - Notice::Following { listening: true }, + Notice::Following { listening: true, max_age_secs: Some(3600) }, Notice::Signal { channel: INFRA_STATUS_CHANNEL.into(), payload: "p".into() }, Notice::Lost, - Notice::Recheck, + Notice::Recheck { listening: false }, + Notice::Resumed, ] { let text = serde_json::to_string(¬ice).unwrap(); assert_eq!(serde_json::from_str::(&text).unwrap(), notice); @@ -962,7 +1030,7 @@ pub mod server { match follower.take() { Some(follow) => following.push(follow(notices)), None => { - let _ = notices.send(Notice::Following { listening: false }).await; + let _ = notices.send(Notice::Following { listening: false, max_age_secs: None }).await; } } } diff --git a/crates/weft-broker-client/src/protocol.rs b/crates/weft-broker-client/src/protocol.rs index 0b95fc34..57b284b6 100644 --- a/crates/weft-broker-client/src/protocol.rs +++ b/crates/weft-broker-client/src/protocol.rs @@ -13,11 +13,6 @@ use weft_core::access::wire::PublishedConnection; use weft_core::ExecutionId; use weft_task_store::tasks::{NewTask, TaskOutcome, TaskStatus}; -/// The longest the broker holds any request open (every `wait_ms` -/// field is capped at it). A client that wants to wait longer asks -/// again. -pub use weft_task_store::pg_signal::MAX_HOLD; - // The wire-enum helper is weft-core's (`weft_core::wire_enum!`), so an // enum can live beside the messages that carry it in any crate. use weft_core::wire_enum; @@ -163,14 +158,14 @@ pub struct RunClaimResponse { } /// `POST /v1/run/answers`: the answers waiting for the waits of a run the -/// calling worker drives, held open up to `wait_ms` (capped at -/// [`MAX_HOLD`]) until one is there. Only the worker driving a run writes -/// its record, so an answer to a run being driven waits for its worker to -/// take it (`weft_task_store::parked_fires::answers_for`). +/// calling worker drives, read once (none when none waits). Only the worker +/// driving a run writes its record, so an answer to a run being driven +/// waits for its worker to take it +/// (`weft_task_store::parked_fires::answers_for`), and the worker is told +/// when one is handed to it (`crate::line::PARKED_FIRE_CHANNEL`). #[derive(Debug, Clone, Serialize, Deserialize)] pub struct RunAnswersRequest { pub execution_id: ExecutionId, - pub wait_ms: u64, /// The waits whose answers the worker already took. An answer stays /// waiting until the run's record holds it, which an unrecorded run's /// never does, so these are left out rather than handed again. @@ -281,9 +276,8 @@ pub struct TaskEnqueueDedupResponse { pub inserted: bool, } -/// A task's outcome, held open up to `wait_ms` (the broker caps it at -/// `pg_signal::MAX_HOLD`) until it is terminal; the answer is its state -/// either way. +/// A task's outcome, held open up to `wait_ms` until it is terminal; the +/// answer is its state either way. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct TaskWaitTerminalRequest { pub task_id: Uuid, @@ -780,7 +774,7 @@ pub struct SupervisorClaimCommandRequest { /// be handed again (they stay uncompleted until they end). pub busy_commands: Vec, /// How long to hold for a command to be issued when none is - /// waiting (the broker caps it at `pg_signal::MAX_HOLD`). + /// waiting. pub wait_ms: u64, } @@ -1382,8 +1376,8 @@ pub struct InfraEnqueueApplyResponse { pub struct InfraWaitApplyRequest { pub project_id: Uuid, pub command_id: i64, - /// How long to hold for the command to complete (the broker caps it - /// at `pg_signal::MAX_HOLD`); the answer is its state either way. + /// How long to hold for the command to complete; the answer is its + /// state either way. pub wait_ms: u64, } @@ -2223,7 +2217,8 @@ pub struct DoorLetGo {} /// worker says it is alive (its `worker_lease`) and how many runs it /// drives per trigger, states what it counted at its door this minute, and /// hears what the project's other copies counted. With nothing under way -/// it sends none, and its lease lapses. +/// it sends none, and its lease lapses; with only runs that wait, its last +/// tick vouches for it until the earliest of those waits gives up. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct DoorTickRequest { /// The image it runs (the program's binary hash): a drain of the runs @@ -2242,6 +2237,11 @@ pub struct DoorTickRequest { /// The route tokens whose counts it reads back. #[serde(default)] pub tokens: Vec, + /// Until when (unix seconds) the lease holds without another tick: set + /// by the last tick before a worker whose runs only wait stops ticking + /// (`weft_task_store::worker_door::lease_until`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub vouch_until: Option, } #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] diff --git a/crates/weft-broker/Cargo.toml b/crates/weft-broker/Cargo.toml index 44d0c925..5a78ae47 100644 --- a/crates/weft-broker/Cargo.toml +++ b/crates/weft-broker/Cargo.toml @@ -8,7 +8,7 @@ repository.workspace = true [dependencies] weft-core = { workspace = true } weft-journal = { workspace = true } -weft-task-store = { workspace = true } +weft-task-store = { workspace = true, features = ["serve"] } weft-access-store = { workspace = true } weft-providers = { workspace = true } weft-broker-client = { path = "../weft-broker-client", features = ["line-server"] } diff --git a/crates/weft-broker/src/door.rs b/crates/weft-broker/src/door.rs index cb635619..df086470 100644 --- a/crates/weft-broker/src/door.rs +++ b/crates/weft-broker/src/door.rs @@ -300,8 +300,8 @@ async fn door_instance_token( /// `POST /v1/door/tick`: see [`DoorTick`]. One statement renews the -/// worker's lease, stores its counts and reads the other copies', so a -/// tick is one round trip. +/// worker's lease (as long as it vouched for, when it did), stores its +/// counts and reads the other copies', so a tick is one round trip. async fn door_tick( State(state): State>, AuthedCaller(caller): AuthedCaller, @@ -315,7 +315,11 @@ async fn door_tick( .bind(project) .bind(tenant) .bind(&req.binary_hash) - .bind(now + weft_core::time_scale::scaled_secs(weft_task_store::worker_door::WORKER_LEASE_SECS)) + .bind(weft_task_store::worker_door::lease_until( + now, + weft_core::time_scale::scaled_secs(weft_task_store::worker_door::WORKER_LEASE_SECS), + req.vouch_until, + )) .bind(sqlx::types::Json(&req.in_flight)) .bind(req.window_start) .bind(&keys) diff --git a/crates/weft-broker/src/handlers.rs b/crates/weft-broker/src/handlers.rs index c21dfcda..3b5a005d 100644 --- a/crates/weft-broker/src/handlers.rs +++ b/crates/weft-broker/src/handlers.rs @@ -173,10 +173,11 @@ fn unix_now_secs() -> u64 { .as_secs() } -/// How long a held request may actually be held: what the process asked -/// for, never more than `MAX_HOLD` (a process that wants longer asks again). -fn held(wait_ms: u64) -> Duration { - Duration::from_millis(wait_ms).min(weft_task_store::pg_signal::MAX_HOLD) +/// When a held call's wait of `wait_ms` ends: as long as the caller asks +/// (it is held on a line, which has no idle timeout), and forever for a +/// wait past what the clock holds. +pub(crate) fn held_until(wait_ms: u64) -> tokio::time::Instant { + weft_core::deadline::after(tokio::time::Instant::now(), Duration::from_millis(wait_ms)) } // ---------- Tasks ---------- @@ -427,7 +428,7 @@ pub async fn task_wait_terminal( require_task_owned_by(&state, &caller, req.task_id).await?; let outcome = state .tasks - .wait_for_terminal(req.task_id, held(req.wait_ms)) + .wait_for_terminal(req.task_id, Duration::from_millis(req.wait_ms)) .await .map_err(internal)?; Ok(Json(TaskWaitTerminalResponse::from_outcome(outcome))) @@ -1162,7 +1163,7 @@ pub async fn supervisor_claim_command( // own ticks): the answer then says so, and the process takes ownership // at once instead of on its next tick. Only after a wake, so a // supervisor woken by a command it cannot take yet does not spin. - let deadline = tokio::time::Instant::now() + held(req.wait_ms); + let deadline = held_until(req.wait_ms); let mut heard = state.signals.subscribe(); let mut woken_once = false; loop { @@ -1686,7 +1687,7 @@ pub async fn infra_wait_apply( } scope::require_project_owned_by(&state.scope_cache, &state.pool, &caller, req.project_id) .await?; - let deadline = tokio::time::Instant::now() + held(req.wait_ms); + let deadline = held_until(req.wait_ms); let mut heard = state.signals.subscribe(); loop { let answer = apply_command_state(&state, &req).await?; diff --git a/crates/weft-broker/src/lib.rs b/crates/weft-broker/src/lib.rs index 7baea3b5..32be193d 100644 --- a/crates/weft-broker/src/lib.rs +++ b/crates/weft-broker/src/lib.rs @@ -27,6 +27,7 @@ pub mod runtime_store; pub mod scope; pub mod held_signals; pub mod line; +pub mod news; pub mod notices; pub mod records; pub mod state; diff --git a/crates/weft-broker/src/line.rs b/crates/weft-broker/src/line.rs index 841adba4..a224e94e 100644 --- a/crates/weft-broker/src/line.rs +++ b/crates/weft-broker/src/line.rs @@ -27,15 +27,19 @@ use axum::Router; use tokio::sync::mpsc; use weft_broker_client::line::server::{self, Follower}; -use weft_broker_client::line::{Notice, ACCESS_CHANNEL, CANCEL_CHANNEL, INFRA_STATUS_CHANNEL, LINE_PATH, TRIGGERS_CHANNEL}; +use weft_broker_client::line::{Notice, ACCESS_CHANNEL, CANCEL_CHANNEL, INFRA_STATUS_CHANNEL, LINE_PATH, PARKED_FIRE_CHANNEL, TRIGGERS_CHANNEL}; use weft_platform_traits::identity::Principal; -use weft_task_store::pg_signal::{Heard, PgSignalWatch}; +use weft_task_store::pg_signal::{Heard, Subscription}; use crate::state::BrokerState; -/// `api` with the line beside it. +/// `api` with the line beside it. Each call on a line holds the database's +/// listening while it is answered, as a request does +/// (`weft_task_store::pg_signal::keep_listening`), and a line with no call +/// running holds nothing: a worker whose runs only wait keeps its line +/// open without keeping the database awake. pub fn routes(api: Router, state: Arc) -> Router { - let served = api.clone(); + let served = api.clone().layer(axum::middleware::from_fn_with_state(state.signals.clone(), weft_task_store::pg_signal::keep_listening)); Router::new() .route( LINE_PATH, @@ -54,15 +58,7 @@ async fn open(state: Arc, headers: HeaderMap, ws: WebSocketUpgrade, Err((status, message)) => return (status, message).into_response(), }; let follower = follower(state.lines.clone(), principal); - // A line carries calls that write, and the notices its far end waits - // on (a run's cancel), so the database is listened to while it is open. - let busy = state.signals.busy().await; - server::upgrade(ws) - .on_upgrade(move |socket| async move { - let _busy = busy; - server::serve(socket, api, Some(follower)).await - }) - .into_response() + server::upgrade(ws).on_upgrade(move |socket| server::serve(socket, api, Some(follower))).into_response() } /// How many notices a line may fall behind by before the fan-out lets go of @@ -74,12 +70,14 @@ const LINE_NOTICES: usize = 1024; /// losses and rechecks, which the caller's copies need as much as the /// notifications). A worker hears its own project and its tenant's shared /// connections; weft's own roles keep no copy that needs notices, so their -/// line follows nothing and says so. +/// line follows nothing and says so. Either way the line is told how long +/// this copy keeps one line ([`LineFanout::start`]), so the caller knows +/// when it is cut. fn follower(fanout: Arc, principal: Principal) -> Follower { Box::new(move |notices: mpsc::Sender| { tokio::spawn(async move { let Principal::Worker { tenant, project } = principal else { - let _ = notices.send(Notice::Following { listening: false }).await; + let _ = notices.send(fanout.following(false)).await; return; }; let project = project.to_string(); @@ -88,11 +86,10 @@ fn follower(fanout: Arc, principal: Principal) -> Follower { let (handed, mut heard) = mpsc::channel(LINE_NOTICES); let _following = fanout.follow(&tenant, &project, handed); let listening = fanout.listening.load(Ordering::Acquire); - let said = match (first, listening) { - (true, listening) => Notice::Following { listening }, + let said = match first { + true => fanout.following(listening), // Back after falling behind: what was missed is not known. - (false, true) => Notice::Recheck, - (false, false) => Notice::Lost, + false => Notice::Recheck { listening }, }; first = false; if notices.send(said).await.is_err() { @@ -111,14 +108,15 @@ fn follower(fanout: Arc, principal: Principal) -> Follower { /// Who a notification on `channel` is for, by its `payload`: an infra /// copy's change is its project's; a connection's is its project's, or its /// tenant's when it is shared across the tenant's projects -/// (`tenant:`); a cancel is its run's project's -/// (` `); a trigger's change is its project's. Every other -/// channel is for nobody on a line. -// SYNC: the channels and payloads <-> weft_broker_client::line::LINE_CHANNELS, crates/weft-dispatcher/src/infra_node.rs (infra_node_status_notify), crates/weft-dispatcher/src/project_store.rs (project_declared_infra_notify), crates/weft-access-store/src/lib.rs (access_notify), crates/weft-task-store/src/tasks.rs (task_ready_notify, cancel_payload), crates/weft-broker/src/caller_auth.rs (names), crates/weft-dispatcher/src/held.rs (access_changed) +/// (`tenant:`); a cancel, and an answer handed to a run, are the +/// run's project's (` `; an event parked for a trigger +/// names its project alone, and is for nobody on a line); a trigger's +/// change is its project's. Every other channel is for nobody on a line. +// SYNC: the channels and payloads <-> weft_broker_client::line::LINE_CHANNELS, crates/weft-dispatcher/src/infra_node.rs (infra_node_status_notify), crates/weft-dispatcher/src/project_store.rs (project_declared_infra_notify), crates/weft-access-store/src/lib.rs (access_notify), crates/weft-task-store/src/tasks.rs (task_ready_notify), crates/weft-task-store/src/runs.rs (run_payload), crates/weft-broker/src/caller_auth.rs (names), crates/weft-dispatcher/src/held.rs (access_changed) fn audience<'a>(channel: &str, payload: &'a str) -> Option> { match channel { INFRA_STATUS_CHANNEL => Some(Audience::Project(payload)), - CANCEL_CHANNEL => payload.split_once(' ').map(|(project, _)| Audience::Project(project)), + CANCEL_CHANNEL | PARKED_FIRE_CHANNEL => payload.split_once(' ').map(|(project, _)| Audience::Project(project)), ACCESS_CHANNEL => Some(match payload.strip_prefix("tenant:") { Some(tenant) => Audience::Tenant(tenant), None => Audience::Project(payload), @@ -141,6 +139,9 @@ pub struct LineFanout { /// Whether the broker's database listener hears right now. listening: AtomicBool, next: AtomicU64, + /// How long this copy's platform lets one request (a line) live + /// before cutting it, `None` when it never does. + max_age: Option, } #[derive(Default)] @@ -161,7 +162,7 @@ impl Lines { } /// A line's place in the fan-out, given up when it goes. -struct Following { +pub(crate) struct Following { fanout: Weak, id: u64, } @@ -175,10 +176,10 @@ impl Drop for Following { } impl LineFanout { - /// Start following `signals` for every line. - pub fn start(signals: &PgSignalWatch) -> Arc { - let mut heard = signals.subscribe(); - let fanout = Arc::new(Self { lines: Mutex::default(), listening: AtomicBool::new(heard.listening()), next: AtomicU64::new(1) }); + /// Start following `heard` (the broker's signal watch) for every line, + /// each told as it opens that it is cut after `max_age`. + pub fn start(mut heard: Subscription, max_age: Option) -> Arc { + let fanout = Arc::new(Self { lines: Mutex::default(), listening: AtomicBool::new(heard.listening()), next: AtomicU64::new(1), max_age }); let following = Arc::downgrade(&fanout); tokio::spawn(async move { loop { @@ -190,11 +191,18 @@ impl LineFanout { fanout.hand(audience, Notice::Signal { channel: channel.to_string(), payload: payload.to_string() }); } } - Ok(Heard::Recheck | Heard::Resumed) if heard.listening() => fanout.say_to_all(true), - Ok(Heard::Recheck | Heard::Resumed | Heard::Lost) => fanout.say_to_all(false), + // Looking again, whether or not the database's listener + // hears right now: a recheck may come from the install's + // news while the database sleeps (`crate::news`), and a + // cancel announced in a gap is found by looking. + Ok(Heard::Recheck) => fanout.say_to_all(Notice::Recheck { listening: heard.listening() }), + // Back from a quiet spell, in which what runs wait on + // came as the install's news: nothing to look at again. + Ok(Heard::Resumed) => fanout.say_to_all(Notice::Resumed), + Ok(Heard::Lost) => fanout.say_to_all(Notice::Lost), Err(e) => { tracing::error!(target: "weft_broker::line", error = %format!("{e:#}"), "the broker's database listener stopped; every line says so"); - fanout.say_to_all(false); + fanout.say_to_all(Notice::Lost); return; } } @@ -203,7 +211,13 @@ impl LineFanout { fanout } - fn follow(self: &Arc, tenant: &str, project: &str, line: mpsc::Sender) -> Following { + /// What a line is told as it opens: whether the listener hears + /// (`listening`), and how long it keeps one line. + fn following(&self, listening: bool) -> Notice { + Notice::Following { listening, max_age_secs: self.max_age.map(|age| age.as_secs()) } + } + + pub(crate) fn follow(self: &Arc, tenant: &str, project: &str, line: mpsc::Sender) -> Following { let id = self.next.fetch_add(1, Ordering::Relaxed); let mut lines = self.lines.lock().expect("line fanout"); lines.by_project.entry(project.to_string()).or_default().insert(id, line.clone()); @@ -226,10 +240,11 @@ impl LineFanout { } } - /// The listener hears (`true`) or stopped hearing: every line is told. - fn say_to_all(&self, listening: bool) { + /// Every line is told `notice`: look again (saying whether the + /// listener hears now), the listener hears again, or it stopped. + fn say_to_all(&self, notice: Notice) { + let listening = matches!(notice, Notice::Recheck { listening: true } | Notice::Resumed); self.listening.store(listening, Ordering::Release); - let notice = if listening { Notice::Recheck } else { Notice::Lost }; let mut lines = self.lines.lock().expect("line fanout"); let behind: Vec = lines .by_project @@ -487,7 +502,7 @@ mod tests { get(|ws: WebSocketUpgrade| async move { ws.on_upgrade(|mut socket| async move { let _follow = socket.recv().await; - for notice in [Notice::Following { listening: true }, Notice::Signal { channel: INFRA_STATUS_CHANNEL.into(), payload: "p".into() }] { + for notice in [Notice::Following { listening: true, max_age_secs: None }, Notice::Signal { channel: INFRA_STATUS_CHANNEL.into(), payload: "p".into() }] { let text = serde_json::to_string(¬ice).unwrap(); let _ = socket.send(Message::Text(text.into())).await; } @@ -509,6 +524,110 @@ mod tests { } } + // A line that opens on a copy whose database listener does not hear + // (it went quiet while the database sleeps) still looks again (a + // cancel announced while the line was down is found so) but trusts + // nothing kept on it; a recheck from the install's news does the same. + weft_core::stress_test! { + name: a_line_opening_on_a_quiet_copy_looks_again_and_trusts_nothing, + runs: 10, + worker_threads: 4, + async fn body() { + use axum::extract::ws::Message; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let app = Router::new().route( + LINE_PATH, + get(|ws: WebSocketUpgrade| async move { + ws.on_upgrade(|mut socket| async move { + let _follow = socket.recv().await; + for notice in [Notice::Following { listening: false, max_age_secs: None }, Notice::Recheck { listening: false }] { + let text = serde_json::to_string(¬ice).unwrap(); + let _ = socket.send(Message::Text(text.into())).await; + } + std::future::pending::<()>().await; + }) + }), + ); + let _server = serve(listener, app).await; + let link = link(address); + let mut heard = link.subscribe(); + async fn next(heard: &mut weft_task_store::pg_signal::Subscription) -> (Heard, bool) { + let heard_now = tokio::time::timeout(Duration::from_secs(10), heard.next()).await.expect("heard in time").expect("the line is up"); + (heard_now, heard.listening()) + } + assert_eq!(next(&mut heard).await, (Heard::Recheck, false), "opened: look again, trust nothing"); + assert_eq!(next(&mut heard).await, (Heard::Recheck, false), "news missed: look again, trust nothing"); + } + } + + /// A recheck reaches every line whether or not the database's listener + /// hears, saying which; the listener hearing again after a quiet spell + /// is said as such (nothing to look at again); a lost listener is said + /// as lost. + #[tokio::test] + async fn a_recheck_reaches_every_line_whatever_the_listener() { + let (tx, rx) = tokio::sync::broadcast::channel(8); + let listening = StdArc::new(AtomicBool::new(false)); + let fanout = LineFanout::start(weft_task_store::pg_signal::Subscription::with_listening(rx, listening.clone()), None); + let (line, mut heard) = mpsc::channel(8); + let _following = fanout.follow("t", "p", line); + async fn next(heard: &mut mpsc::Receiver) -> Notice { + tokio::time::timeout(Duration::from_secs(10), heard.recv()).await.expect("told in time").unwrap() + } + tx.send(Heard::Recheck).unwrap(); + assert_eq!(next(&mut heard).await, Notice::Recheck { listening: false }); + listening.store(true, Ordering::SeqCst); + tx.send(Heard::Resumed).unwrap(); + assert_eq!(next(&mut heard).await, Notice::Resumed); + tx.send(Heard::Lost).unwrap(); + assert_eq!(next(&mut heard).await, Notice::Lost); + } + + /// A line the broker keeps for a while tells its process when the cut + /// comes, counted from before the line opened, and stops saying so once + /// that line is gone. + #[tokio::test] + async fn a_line_says_when_the_broker_cuts_it() { + use axum::extract::ws::Message; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let hang_up = StdArc::new(tokio::sync::Notify::new()); + let told = StdArc::new(AtomicBool::new(false)); + let app = Router::new().route( + LINE_PATH, + get({ + let hang_up = hang_up.clone(); + move |ws: WebSocketUpgrade| { + let (hang_up, told) = (hang_up.clone(), told.clone()); + async move { + ws.on_upgrade(move |mut socket| async move { + // The line opened again after the hang-up is + // never followed: what it said stays cleared. + let _follow = socket.recv().await; + if !told.swap(true, Ordering::SeqCst) { + let text = serde_json::to_string(&Notice::Following { listening: true, max_age_secs: Some(3600) }).unwrap(); + let _ = socket.send(Message::Text(text.into())).await; + } + hang_up.notified().await; + }) + } + } + }), + ); + let _server = serve(listener, app).await; + let link = link(address); + let mut cut = link.cut_at(); + let before = tokio::time::Instant::now(); + let _held = link.stay_open(); + tokio::time::timeout(Duration::from_secs(10), cut.wait_for(Option::is_some)).await.expect("told in time").unwrap(); + let told = cut.borrow_and_update().expect("a cut"); + assert_eq!(told.max_age, Duration::from_secs(3600)); + assert!(told.opened >= before && told.opened <= tokio::time::Instant::now(), "counted from the opening"); + hang_up.notify_one(); + tokio::time::timeout(Duration::from_secs(10), cut.wait_for(Option::is_none)).await.expect("cleared in time").unwrap(); + } + #[test] fn a_notice_is_for_the_project_or_the_tenant_it_names() { let project = uuid::Uuid::from_u128(1).to_string(); @@ -517,6 +636,8 @@ mod tests { assert_eq!(audience(ACCESS_CHANNEL, "tenant:t"), Some(Audience::Tenant("t"))); let cancel = format!("{project} e-1"); assert_eq!(audience(CANCEL_CHANNEL, &cancel), Some(Audience::Project(&project))); + assert_eq!(audience(PARKED_FIRE_CHANNEL, &cancel), Some(Audience::Project(&project)), "an answer handed to a run"); + assert_eq!(audience(PARKED_FIRE_CHANNEL, &project), None, "an event parked for a trigger rides no line"); assert_eq!(audience("weft_task_ready", "worker:x"), None); } @@ -524,7 +645,7 @@ mod tests { /// other, and a line that cannot keep up is let go of. #[tokio::test] async fn the_fanout_hands_a_notice_to_the_lines_it_names() { - let fanout = StdArc::new(LineFanout { lines: Mutex::default(), listening: AtomicBool::new(true), next: AtomicU64::new(1) }); + let fanout = StdArc::new(LineFanout { lines: Mutex::default(), listening: AtomicBool::new(true), next: AtomicU64::new(1), max_age: None }); let (mine, mut mine_heard) = mpsc::channel(4); let (other, mut other_heard) = mpsc::channel(4); let (slow, _slow_heard) = mpsc::channel(1); diff --git a/crates/weft-broker/src/news.rs b/crates/weft-broker/src/news.rs new file mode 100644 index 00000000..dc398209 --- /dev/null +++ b/crates/weft-broker/src/news.rs @@ -0,0 +1,131 @@ +//! The install's news (`weft_platform_traits::News`), heard into the +//! broker's signal watch. +//! +//! A worker whose run only waits keeps its line to a broker copy open and +//! stops everything else (`weft_engine::run_holds`), and that copy pushes +//! the run's answer and its cancel down the line (`crate::line`) holding no +//! listening of the database, so the database can sleep. The answer and the +//! cancel are written by a dispatcher copy, +//! which also announces them as news. Each one heard here is told to the +//! copy's waiters exactly as the database would have told it, on the same +//! channel with the same payload, so nothing downstream knows the +//! difference; the database telling it too only costs a look. + +use weft_platform_traits::{Listening, NewsHeard, NewsKind, RunNews}; +use weft_task_store::pg_signal::AlsoTold; + +/// The channels news comes in on, for the watch that hears it. +pub const NEWS_CHANNELS: &[&str] = &[weft_task_store::parked_fires::PARKED_FIRE_CHANNEL, weft_task_store::runs::CANCEL_CHANNEL]; + +/// What the database announces for `news`: an answer handed to a run as a +/// parked fire naming the run, a cancel as a cancel naming it. +// SYNC: the payloads <-> crates/weft-task-store/src/parked_fires.rs (parked_fire_notify), crates/weft-dispatcher/src/journal/postgres.rs (cancel_execution) +fn as_announced(news: RunNews) -> (&'static str, String) { + let channel = match news.kind { + NewsKind::Answer => weft_task_store::parked_fires::PARKED_FIRE_CHANNEL, + NewsKind::Cancel => weft_task_store::runs::CANCEL_CHANNEL, + }; + (channel, weft_task_store::runs::run_payload(news.project, news.execution)) +} + +/// Tell `told` everything `listening` hears, for the life of the process: +/// news as its announcement, news missed as a recheck. Returns only when +/// either side stopped, which leaves the broker's waiters deaf to news, so +/// it is run supervised. +pub async fn pass_on(mut listening: Listening, told: AlsoTold) { + loop { + let heard = match listening.next().await { + Ok(heard) => heard, + Err(e) => { + tracing::error!(target: "weft_broker::news", error = %format!("{e:#}"), "the install's news stopped: runs held on this copy would hear their answers and cancels late"); + return; + } + }; + let passed = match heard { + NewsHeard::News(news) => { + let (channel, payload) = as_announced(news); + match told.signal(channel, &payload) { + Ok(passed) => passed, + Err(e) => { + tracing::error!(target: "weft_broker::news", error = %format!("{e:#}"), "news the broker's watch cannot pass on"); + return; + } + } + } + NewsHeard::MissedSome => told.recheck(), + }; + if !passed { + tracing::error!(target: "weft_broker::news", "the broker's signal watch stopped; news has nowhere to go"); + return; + } + } +} + +#[cfg(test)] +mod tests { + use std::sync::atomic::AtomicBool; + use std::sync::Arc; + use std::time::Duration; + + use tokio::sync::mpsc; + use weft_broker_client::line::Notice; + use weft_platform_traits::{FakeNews, News}; + use weft_task_store::pg_signal::{Heard, Subscription}; + + use super::*; + use crate::line::LineFanout; + + fn news(kind: NewsKind) -> RunNews { + RunNews { project: uuid::Uuid::from_u128(1), execution: uuid::Uuid::from_u128(2), kind } + } + + #[test] + fn news_is_told_as_the_database_announces_it() { + let payload = weft_task_store::runs::run_payload(uuid::Uuid::from_u128(1), uuid::Uuid::from_u128(2)); + assert_eq!(as_announced(news(NewsKind::Answer)), (weft_task_store::parked_fires::PARKED_FIRE_CHANNEL, payload.clone())); + assert_eq!(as_announced(news(NewsKind::Cancel)), (weft_task_store::runs::CANCEL_CHANNEL, payload)); + } + + async fn next(heard: &mut Subscription) -> Heard { + tokio::time::timeout(Duration::from_secs(10), heard.next()).await.expect("heard in time").unwrap() + } + + // News reaches the broker's waiters as the database's announcement + // would, and an answer and a cancel reach the line of the run's + // project; news missed is a recheck. + weft_core::stress_test! { + name: news_reaches_the_waiters_and_the_runs_line, + runs: 10, + worker_threads: 4, + async fn body() { + let fake = FakeNews::default(); + let (tx, rx) = tokio::sync::broadcast::channel(64); + let mut waiter = Subscription::from(rx); + let fanout = LineFanout::start(Subscription::with_listening(tx.subscribe(), Arc::new(AtomicBool::new(false))), None); + let (line, mut on_line) = mpsc::channel(8); + let _following = fanout.follow("t", &uuid::Uuid::from_u128(1).to_string(), line); + tokio::spawn(pass_on(fake.listen(), AlsoTold::over(&tx, NEWS_CHANNELS))); + // Told once the passing-on listens. + while fake.heard.receiver_count() < 1 { + tokio::task::yield_now().await; + } + + fake.heard.send(NewsHeard::News(news(NewsKind::Answer))).unwrap(); + let (channel, payload) = as_announced(news(NewsKind::Answer)); + assert_eq!(next(&mut waiter).await, Heard::Signal { channel, payload: payload.clone().into() }, "a waiter hears the answer handed"); + let told = tokio::time::timeout(Duration::from_secs(10), on_line.recv()).await.unwrap().unwrap(); + assert_eq!(told, Notice::Signal { channel: channel.into(), payload }, "the run's line hears its answer"); + + fake.heard.send(NewsHeard::News(news(NewsKind::Cancel))).unwrap(); + let (channel, payload) = as_announced(news(NewsKind::Cancel)); + let told = tokio::time::timeout(Duration::from_secs(10), on_line.recv()).await.unwrap().unwrap(); + assert_eq!(told, Notice::Signal { channel: channel.into(), payload }, "the run's line hears its cancel"); + + fake.heard.send(NewsHeard::MissedSome).unwrap(); + assert_eq!(next(&mut waiter).await, Heard::Signal { channel, payload: as_announced(news(NewsKind::Cancel)).1.into() }); + assert_eq!(next(&mut waiter).await, Heard::Recheck, "news missed: every waiter looks again"); + let told = tokio::time::timeout(Duration::from_secs(10), on_line.recv()).await.unwrap().unwrap(); + assert_eq!(told, Notice::Recheck { listening: false }, "and every line, though the database's listener is quiet"); + } + } +} diff --git a/crates/weft-broker/src/records.rs b/crates/weft-broker/src/records.rs index 2b183b4f..e9c07370 100644 --- a/crates/weft-broker/src/records.rs +++ b/crates/weft-broker/src/records.rs @@ -7,7 +7,6 @@ //! in the write itself, and a birth must name the worker's project. use std::sync::Arc; -use std::time::Duration; use axum::body::Bytes; use axum::extract::State; @@ -103,28 +102,17 @@ async fn run_claim(State(state): State>, AuthedCaller(caller): } /// `POST /v1/run/answers`: see [`RunAnswersRequest`]. Only the worker -/// driving the run takes its answers; the hold wakes when an answer is -/// parked for the run's project. +/// driving the run takes its answers, read once: the worker asks when its +/// run starts waiting, and again when the line tells it one was handed to +/// the run (`weft_broker_client::line::PARKED_FIRE_CHANNEL`). async fn run_answers(State(state): State>, AuthedCaller(caller): AuthedCaller, Json(req): Json) -> Resp { - let (_, project, _) = worker(&caller)?; + worker(&caller)?; crate::handlers::require_worker_drives(&state, &caller, req.execution_id).await?; - let deadline = tokio::time::Instant::now() + Duration::from_millis(req.wait_ms).min(weft_task_store::pg_signal::MAX_HOLD); - let project = project.to_string(); - // Subscribed before the first read, so an answer parked between an - // empty read and the wait still ends the wait. - let mut heard = state.signals.subscribe(); - loop { - let mut answers = weft_task_store::parked_fires::answers_for(&state.pool, req.execution_id) - .await - .map_err(|e| unavailable_or_internal(e.context("read a run's answers")))?; - answers.retain(|(token, _)| !req.taken.contains(token)); - let parked = |channel: &str, payload: &str| channel == weft_task_store::parked_fires::PARKED_FIRE_CHANNEL && payload == project; - if !answers.is_empty() - || !heard.woken_before(deadline, parked).await.map_err(|e| unavailable_or_internal(e.context("wait for a run's answers")))? - { - return Ok(Json(RunAnswersResponse { answers: answers.into_iter().map(|(token, answer)| RunAnswer { token, answer }).collect() })); - } - } + let mut answers = weft_task_store::parked_fires::answers_for(&state.pool, req.execution_id) + .await + .map_err(|e| unavailable_or_internal(e.context("read a run's answers")))?; + answers.retain(|(token, _)| !req.taken.contains(token)); + Ok(Json(RunAnswersResponse { answers: answers.into_iter().map(|(token, answer)| RunAnswer { token, answer }).collect() })) } /// `POST /v1/run/give_up`: see [`RunGiveUpRequest`]. Its ending is a diff --git a/crates/weft-broker/src/state.rs b/crates/weft-broker/src/state.rs index 266d3d5a..3017fe17 100644 --- a/crates/weft-broker/src/state.rs +++ b/crates/weft-broker/src/state.rs @@ -24,8 +24,8 @@ use crate::scope::ScopeCache; pub const BROKER_CHANNELS: &[&str] = &[ weft_task_store::tasks::TASK_READY_CHANNEL, weft_task_store::terminal::TERMINAL_CHANNEL, - // What a worker holding for the answers of a run it drives waits on - // (`crate::records`). + // An answer handed to a run a worker drives, which its line pushes + // (`crate::line`). weft_task_store::parked_fires::PARKED_FIRE_CHANNEL, weft_broker_client::lifecycle_command::INFRA_COMMAND_CHANNEL, // What a line pushes to the workers following it (`crate::line`). @@ -110,6 +110,9 @@ pub struct BrokerSettings { pub public_base_url: String, pub internet_url: Option, pub object_store_public_internet: bool, + /// How long this process's platform lets one line live before cutting + /// it, `None` when it never does (`crate::line::LineFanout::start`). + pub line_max_age: Option, } impl BrokerState { @@ -143,7 +146,7 @@ impl BrokerState { crate::caller_auth::verifiers_changed, |_| true, )?; - let lines = crate::line::LineFanout::start(&signals); + let lines = crate::line::LineFanout::start(signals.subscribe(), settings.line_max_age); let tasks: Arc = Arc::new(PostgresTaskStoreClient::new(pool.clone(), signals.clone())?); // A public infra endpoint's address is handed to whoever calls in // (a provider's webhook target), so it is built on the address the diff --git a/crates/weft-core/src/deadline.rs b/crates/weft-core/src/deadline.rs new file mode 100644 index 00000000..2eed93f4 --- /dev/null +++ b/crates/weft-core/src/deadline.rs @@ -0,0 +1,26 @@ +//! When a wait ends, for a wait its caller may make as long as it likes. + +use std::time::Duration; + +use tokio::time::Instant; + +/// `wait` from `from`, or, for a wait past what the clock holds, so far +/// ahead that it never comes. +pub fn after(from: Instant, wait: Duration) -> Instant { + from.checked_add(wait).unwrap_or_else(|| from + FOREVER) +} + +/// What [`after`] reads as never: thirty years, as tokio's own timers do. +const FOREVER: Duration = Duration::from_secs(30 * 365 * 24 * 3600); + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_wait_past_the_clock_is_forever() { + let now = Instant::now(); + assert_eq!(after(now, Duration::from_secs(5)), now + Duration::from_secs(5)); + assert_eq!(after(now, Duration::MAX), now + FOREVER); + } +} diff --git a/crates/weft-core/src/lib.rs b/crates/weft-core/src/lib.rs index b2eddb5a..7332ccbe 100644 --- a/crates/weft-core/src/lib.rs +++ b/crates/weft-core/src/lib.rs @@ -23,6 +23,8 @@ pub mod comma_list; #[cfg(feature = "runtime")] pub mod cancellation; #[cfg(feature = "runtime")] +pub mod deadline; +#[cfg(feature = "runtime")] pub mod context; pub mod deactivation; pub mod deref; diff --git a/crates/weft-dispatcher/Cargo.toml b/crates/weft-dispatcher/Cargo.toml index 24a1f85d..84b0c5eb 100644 --- a/crates/weft-dispatcher/Cargo.toml +++ b/crates/weft-dispatcher/Cargo.toml @@ -18,7 +18,7 @@ weft-compiler = { path = "../weft-compiler", features = ["build"] } # The catalog a version compiles against (`build::source`). weft-catalog = { workspace = true } weft-journal = { workspace = true } -weft-task-store = { workspace = true } +weft-task-store = { workspace = true, features = ["serve"] } weft-access-store = { workspace = true } weft-broker-client = { path = "../weft-broker-client" } weft-platform-traits = { workspace = true } diff --git a/crates/weft-dispatcher/src/api/mod.rs b/crates/weft-dispatcher/src/api/mod.rs index 83bca6b2..73621068 100644 --- a/crates/weft-dispatcher/src/api/mod.rs +++ b/crates/weft-dispatcher/src/api/mod.rs @@ -26,10 +26,14 @@ use crate::state::DispatcherState; /// webhook payloads. Trusted routes keep axum's default limit. const PUBLIC_FIRE_BODY_LIMIT: usize = 256 * 1024; +/// The longest a read of this API is held open waiting for its answer to +/// change: a client that wants longer asks again, and a hold this short +/// never meets an idle timeout on the way. +const MAX_HOLD: std::time::Duration = std::time::Duration::from_secs(25); + /// `?wait_ms=` on a read whose answer a client waits on: hold the read -/// for up to that long for the answer to change (never past `MAX_HOLD`; -/// a client that wants longer asks again). Absent, the read answers at -/// once. +/// for up to that long for the answer to change (never past [`MAX_HOLD`]). +/// Absent, the read answers at once. #[derive(serde::Deserialize, Default)] pub struct HoldQuery { #[serde(default)] @@ -198,7 +202,7 @@ fn version_note(source: Option<&weft_core::install::WeftSource>, cli_commit: &st impl HoldQuery { pub fn hold(&self) -> std::time::Duration { - std::time::Duration::from_millis(self.wait_ms).min(weft_task_store::pg_signal::MAX_HOLD) + std::time::Duration::from_millis(self.wait_ms).min(MAX_HOLD) } } diff --git a/crates/weft-dispatcher/src/delivery.rs b/crates/weft-dispatcher/src/delivery.rs index 27a0f069..efa61be0 100644 --- a/crates/weft-dispatcher/src/delivery.rs +++ b/crates/weft-dispatcher/src/delivery.rs @@ -15,6 +15,9 @@ //! on something outside it), on a task of its own so one run never holds //! up the next delivery. On a platform that stops a worker nobody is //! calling (Cloud Run), that held call is what keeps the run's worker up. +//! The database is listened to only around this side's own reads and +//! writes, never for the length of the call: a run that waits a day for a +//! person leaves the database free to sleep meanwhile. use std::time::Duration; @@ -63,13 +66,7 @@ pub fn drain_loop(state: DispatcherState) -> DrainLoop { let full = taken.len() as i64 == BATCH; for delivery in taken { let state = state.clone(); - // A run being handed over is work until it lands: the - // database is listened to meanwhile, for what it writes. - let busy = state.signals.busy().await; - tokio::spawn(async move { - let _busy = busy; - deliver(&state, delivery).await - }); + tokio::spawn(async move { deliver(&state, delivery).await }); } Ok(if full { DrainStep::More } else { DrainStep::Done }) } @@ -142,13 +139,18 @@ pub async fn worker_target( async fn deliver(state: &DispatcherState, delivery: Delivery) { let execution_id = delivery.execution_id; + // What this side reads and writes around the call is work: the + // database is listened to for it, and only for it. + let busy = state.signals.busy().await; let Some(binary_hash) = &delivery.binary_hash else { if let Err(e) = end_undeliverable(state, execution_id, "it runs no program").await { tracing::error!(target: "weft_dispatcher::delivery", %execution_id, error = %format!("{e:#}"), "could not end an undeliverable run; its delivery runs out and it is tried again"); } return; }; - let outcome = match worker_target(state, &delivery.tenant_id, delivery.project_id, binary_hash).await { + let target = worker_target(state, &delivery.tenant_id, delivery.project_id, binary_hash).await; + drop(busy); + let outcome = match target { Err(e) => Err(e), Ok(target) => run_on_worker(state, &target, execution_id).await, }; @@ -161,6 +163,7 @@ async fn deliver(state: &DispatcherState, delivery: Delivery) { ), Ok(answer) => tracing::debug!(target: "weft_dispatcher::delivery", %execution_id, ?answer, "worker answered"), Err(e) => { + let _busy = state.signals.busy().await; // Nothing claimed it: give the delivery back so the next pass // makes it again instead of waiting out the lease. A worker // still starting is the platform's bring-up going on without diff --git a/crates/weft-dispatcher/src/journal/postgres.rs b/crates/weft-dispatcher/src/journal/postgres.rs index 04486ec1..45809f81 100644 --- a/crates/weft-dispatcher/src/journal/postgres.rs +++ b/crates/weft-dispatcher/src/journal/postgres.rs @@ -123,9 +123,10 @@ impl PostgresJournal { /// record, `SuspensionResolved` at `last_seq + 1`, and is queued for a /// worker to carry on; a run a worker drives is written by that worker /// alone, so the answer is handed to it -/// (`weft_task_store::parked_fires::hand_answer_in`). The run's row is -/// locked before the signal is touched, the order every writer of a run -/// takes them in. The caller pokes the outbox once it commits. +/// (`weft_task_store::parked_fires::hand_answer_in`, which also makes the +/// news its worker hears it by). The run's row is locked before the signal +/// is touched, the order every writer of a run takes them in. The caller +/// pokes the outbox once it commits. pub(crate) async fn answer_in( tx: &mut sqlx::PgConnection, token: &str, @@ -927,9 +928,14 @@ impl Journal for PostgresJournal { .bind(sqlx::types::Json(cause)) .execute(&mut *tx) .await?; - sqlx::query("SELECT pg_notify($1, $2)") + // Announced, and the install's news for the copy holding + // the line of the run's worker (`weft_task_store::announce`). + // SYNC: 'cancel' <-> weft_task_store::announce::kind_named + sqlx::query("SELECT pg_notify($1, $2), weft_tell($3, $4, 'cancel')") .bind(weft_task_store::runs::CANCEL_CHANNEL) - .bind(weft_task_store::runs::cancel_payload(locked.project_id, execution_id)) + .bind(weft_task_store::runs::run_payload(locked.project_id, execution_id)) + .bind(locked.project_id) + .bind(execution_id) .execute(&mut *tx) .await?; write.requested = true; @@ -941,6 +947,9 @@ impl Journal for PostgresJournal { } tx.commit().await?; weft_task_store::announce::committed(&self.pool); + if write.requested { + weft_task_store::announce::news_written(&self.pool); + } Ok(write) } @@ -990,6 +999,11 @@ impl Journal for PostgresJournal { let answered = answer_in(&mut tx, token, answer, AnswerFrom::Sender).await?; tx.commit().await?; weft_task_store::announce::committed(&self.pool); + // An answer that reached its run may have been handed to the + // worker driving it, which left news for that worker. + if matches!(answered, crate::journal::Answered::Reached { .. }) { + weft_task_store::announce::news_written(&self.pool); + } Ok(answered) } diff --git a/crates/weft-dispatcher/src/parked_drain.rs b/crates/weft-dispatcher/src/parked_drain.rs index 22e15c7c..65a69d5d 100644 --- a/crates/weft-dispatcher/src/parked_drain.rs +++ b/crates/weft-dispatcher/src/parked_drain.rs @@ -204,25 +204,31 @@ async fn hand_answer( head: &Head, routing: &crate::api::signal::FireGateInfo, ) -> anyhow::Result { - let consumed = match routing.standing().arrival(crate::lease::now_unix()) { + // The wait's signal gone, and whether the answer reached its run (and + // may have left news for the worker driving it). + let (consumed, reached) = match routing.standing().arrival(crate::lease::now_unix()) { // Still parked: it waits on, and nothing behind it can go first. weft_core::arrival::Arrival::Wait => return Ok(false), weft_core::arrival::Arrival::Refused => { tracing::info!(target: "weft_dispatcher::parked_drain", token = %head.token, "a queued answer dropped: its trigger takes no work any more"); weft_task_store::parked_fires::remove_in(&mut tx, head).await?; - None + (None, false) } weft_core::arrival::Arrival::Live => match crate::journal::postgres::answer_in(&mut tx, &head.token, &head.waiting.answer(), crate::journal::postgres::AnswerFrom::Queue).await? { // The wait's signal went, and its queue with it. - crate::journal::Answered::Reached { consumed } | crate::journal::Answered::RunEnded { consumed } => Some(consumed), + crate::journal::Answered::Reached { consumed } => (Some(consumed), true), + crate::journal::Answered::RunEnded { consumed } => (Some(consumed), false), crate::journal::Answered::Gone => { weft_task_store::parked_fires::remove_in(&mut tx, head).await?; - None + (None, false) } }, }; tx.commit().await?; weft_task_store::announce::committed(&state.pg_pool); + if reached { + weft_task_store::announce::news_written(&state.pg_pool); + } if let Some(consumed) = consumed { state.listener.unregister_many(&[consumed]).await; } diff --git a/crates/weft-dispatcher/src/proxy.rs b/crates/weft-dispatcher/src/proxy.rs index 0de87c28..44fbb278 100644 --- a/crates/weft-dispatcher/src/proxy.rs +++ b/crates/weft-dispatcher/src/proxy.rs @@ -215,6 +215,13 @@ async fn forward_http( if let Some((name, value)) = &upstream.auth { call = call.header(name, value); } + // Nothing from here on touches the database, and the upstream's answer + // may take as long as the run behind it (a route held while its run + // waits on a person): the request lets go of its process's listening + // before it waits on the upstream at all. + if let Some(listening) = parts.extensions.get::() { + listening.let_go(); + } let sent = call.body(reqwest::Body::wrap_stream(body.into_data_stream())).send().await; let answer = match sent { Ok(answer) => answer, diff --git a/crates/weft-dispatcher/src/reaper.rs b/crates/weft-dispatcher/src/reaper.rs index 6ac23998..a208f36c 100644 --- a/crates/weft-dispatcher/src/reaper.rs +++ b/crates/weft-dispatcher/src/reaper.rs @@ -42,28 +42,51 @@ fn woken_reaper_safety() -> Duration { weft_core::time_scale::scaled(Duration::from_secs(60)) } -/// Whether anything in the install is in motion: a task of the -/// dispatcher's claimed, a run queued for a worker or being driven by one, -/// an activation or a build under way, a lifecycle command not finished. -/// While nothing is, no lease can lapse and no driver can die mid-way, so -/// the loops that watch for that have nothing to watch. A run parked on a -/// person keeps nothing here, so it does not keep the install awake. -pub async fn in_motion(pool: &sqlx::PgPool) -> anyhow::Result { - Ok(sqlx::query_scalar( +/// When anything in the install is next in motion: now (`Some(ZERO)`), in +/// a while, or never until a write says so (`None`). Moving now is a task +/// of the dispatcher's claimed, a run queued for a worker, an activation or +/// a build under way, a lifecycle command not finished. A run a worker +/// drives moves once its worker's lease could have lapsed for longer than +/// the lost-run sweep's margin (`crate::worker_door`): a lease nobody +/// renews any more is the only thing to watch for there, and a worker +/// whose runs only wait vouched for itself until their waits give up, so +/// the install looks once then rather than every safety interval. While +/// nothing is in motion, no lease can lapse and no driver can die mid-way, +/// so the loops that watch for that have nothing to watch. A run parked on +/// a person keeps nothing here, so it does not keep the install awake. +pub async fn in_motion(pool: &sqlx::PgPool) -> anyhow::Result> { + let (now, lapse_in): (bool, Option) = sqlx::query_as( "SELECT EXISTS (SELECT 1 FROM task WHERE status = 'claimed') \ - OR EXISTS (SELECT 1 FROM run WHERE state = 'running') \ OR EXISTS (SELECT 1 FROM run WHERE state = 'queued') \ OR EXISTS (SELECT 1 FROM trigger_activation WHERE status IN ('activating', 'deactivating')) \ OR EXISTS (SELECT 1 FROM project WHERE transition <> 'none') \ - OR EXISTS (SELECT 1 FROM infra_lifecycle_command WHERE completed_at_unix IS NULL)", + OR EXISTS (SELECT 1 FROM infra_lifecycle_command WHERE completed_at_unix IS NULL), \ + (SELECT MIN(COALESCE(l.leased_until_unix, 0)) - EXTRACT(EPOCH FROM NOW())::BIGINT \ + FROM run r LEFT JOIN worker_lease l ON l.replica = r.owner WHERE r.state = 'running')", ) .fetch_one(pool) - .await?) + .await?; + Ok(next_motion(now, lapse_in, crate::worker_door::lost_margin_secs())) } -/// `inner`, looked at again at its safety interval after it ran dry for as -/// long as anything is in motion ([`in_motion`]), and left to sleep until -/// a write wakes it once nothing is. +/// [`in_motion`]'s answer from what it read: something moves `now`, or +/// the earliest lease of a running run's worker lapses `lapse_in` seconds +/// from now (a run with no lease lapsed long ago), which the sweep judges +/// once its `margin` passed too. +fn next_motion(now: bool, lapse_in: Option, margin: i64) -> Option { + match (now, lapse_in) { + (true, _) => Some(Duration::ZERO), + (false, Some(lapse_in)) => { + let judged_in = lapse_in.saturating_add(margin); + Some(if judged_in < 0 { Duration::ZERO } else { Duration::from_secs(judged_in.unsigned_abs() + 1) }) + } + (false, None) => None, + } +} + +/// `inner`, looked at again once it ran dry when anything is next in +/// motion ([`in_motion`]), at its safety interval at the soonest, and left +/// to sleep until a write wakes it once nothing is. pub fn while_in_motion(state: &DispatcherState, inner: DrainLoop) -> DrainLoop { let state = state.clone(); let (drain, safety) = (inner.drain.clone(), inner.safety); @@ -71,7 +94,10 @@ pub fn while_in_motion(state: &DispatcherState, inner: DrainLoop) -> DrainLoop { let (state, drain) = (state.clone(), drain.clone()); async move { match drain().await? { - DrainStep::Done if in_motion(&state.pg_pool).await? => Ok(DrainStep::RetryIn(safety)), + DrainStep::Done => Ok(match in_motion(&state.pg_pool).await? { + Some(after) => DrainStep::RetryIn(after.max(safety)), + None => DrainStep::Done, + }), step => Ok(step), } } @@ -418,3 +444,20 @@ async fn sweep_tasks(state: DispatcherState) -> anyhow::Result<()> { } Ok(()) } + +#[cfg(test)] +mod tests { + use super::*; + + /// Something moving now wins; a running run's worker is looked at once + /// its lease could have lapsed past the margin (now when it already + /// did); with nothing, the loops sleep. + #[test] + fn a_running_run_is_looked_at_when_its_workers_lease_could_lapse() { + assert_eq!(next_motion(true, Some(300), 15), Some(Duration::ZERO)); + assert_eq!(next_motion(false, Some(300), 15), Some(Duration::from_secs(316)), "a vouched lease: one look, past it"); + assert_eq!(next_motion(false, Some(-10), 15), Some(Duration::from_secs(6)), "lapsed, the margin not passed yet"); + assert_eq!(next_motion(false, Some(-100), 15), Some(Duration::ZERO), "lapsed past the margin: now"); + assert_eq!(next_motion(false, None, 15), None); + } +} diff --git a/crates/weft-dispatcher/src/worker_door.rs b/crates/weft-dispatcher/src/worker_door.rs index 642f24e2..76f00054 100644 --- a/crates/weft-dispatcher/src/worker_door.rs +++ b/crates/weft-dispatcher/src/worker_door.rs @@ -36,7 +36,7 @@ pub async fn sweep(pool: &PgPool, now: i64) -> Result<()> { /// How far past its lease a worker's runs are left alone before they are /// read as lost: a worker that missed a few ticks under load is not gone. -fn lost_margin_secs() -> i64 { +pub(crate) fn lost_margin_secs() -> i64 { weft_core::time_scale::scaled_secs(weft_task_store::worker_door::WORKER_LEASE_SECS) } @@ -49,7 +49,10 @@ const LOST_BATCH: i64 = 500; /// ends (`crate::journal::Journal::let_go_of_lost`). Each run is taken /// under its own row lock, so a batch its old owner sends meanwhile and /// this sweep never write the run at once, and the epoch this raises -/// refuses whichever of that owner's batches comes after. +/// refuses whichever of that owner's batches comes after. A worker whose +/// runs only wait vouched for itself until their waits give up +/// (`weft_task_store::worker_door::lease_until`), so its lease, and its +/// runs, hold until then. pub async fn sweep_lost_runs(state: &crate::state::DispatcherState, now: i64) -> Result<()> { let lapsed_before = now - lost_margin_secs(); let lost: Vec<(weft_core::ExecutionId, String)> = sqlx::query_as( diff --git a/crates/weft-dispatcher/tests/db_lifecycle.rs b/crates/weft-dispatcher/tests/db_lifecycle.rs index 80d28118..19049545 100644 --- a/crates/weft-dispatcher/tests/db_lifecycle.rs +++ b/crates/weft-dispatcher/tests/db_lifecycle.rs @@ -689,6 +689,11 @@ async fn seed_worker(pool: &PgPool, replica: &str, project_id: Uuid, binary_hash .unwrap(); } +/// The install's news the writes left for `run`, by kind, oldest first. +async fn news_of(pool: &PgPool, run: weft_core::ExecutionId) -> Vec<(Uuid, String)> { + sqlx::query_as("SELECT project_id, kind FROM weft_news WHERE execution_id = $1 ORDER BY seq").bind(run).fetch_all(pool).await.unwrap() +} + /// Claim queued run `execution_id` of `project_id` for `owner`, the way /// a worker takes it. async fn claim(pool: &PgPool, execution_id: weft_core::ExecutionId, project_id: Uuid, owner: &str) { @@ -770,6 +775,86 @@ async fn a_lost_run_is_queued_again_when_durable_and_ended_when_fast(pool: PgPoo assert_eq!(journal.let_go_of_lost(fast, now, None).await.unwrap(), Lost::NotLost, "an ended run is not lost"); } +/// A worker whose runs only wait vouched for itself until their waits +/// give up: its lease holds that long through the broker's tick, held to +/// the longest a wait can hold; its runs are still worked on and not lost, +/// it still counts among its program's copies, and the loops that watch +/// lapsed leases look once the vouch could lapse, not before. Once the +/// vouch passed with the worker gone, its run is lost. +#[sqlx::test] +async fn a_vouching_workers_runs_are_its_own_until_its_vouch_passes(pool: PgPool) { + use std::time::Duration; + use weft_dispatcher::journal::Lost; + use weft_task_store::runs::in_flight; + use weft_task_store::worker_door::lease_until; + let (journal, projects) = setup(&pool).await; + let id = Uuid::new_v4(); + seed_project(&projects, id, "bin-A").await; + let now = weft_dispatcher::lease::now_unix(); + let run = queued_run(&journal, id).await; + claim(&pool, run, id, "worker-a").await; + let tick = |replica: &'static str, vouch: Option| { + let pool = pool.clone(); + async move { + let answer: serde_json::Value = sqlx::query_scalar("SELECT weft_door_tick($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)") + .bind(replica) + .bind(id) + .bind(TENANT) + .bind("bin-A") + .bind(lease_until(now, 15, vouch)) + .bind(json!({ "t1": 1 })) + .bind(0i64) + .bind(Vec::::new()) + .bind(Vec::::new()) + .bind(Vec::::new()) + .fetch_one(&pool) + .await + .unwrap(); + answer["copies"].as_i64().unwrap() + } + }; + assert_eq!(tick("worker-a", Some(now + 600)).await, 1); + assert_eq!(tick("worker-b", None).await, 2, "the vouching copy still counts"); + assert!(in_flight(&pool, run).await.unwrap(), "vouched: still worked on"); + assert_eq!(journal.let_go_of_lost(run, now + 500, None).await.unwrap(), Lost::NotLost, "its own until its vouch passes"); + sqlx::query("DELETE FROM worker_lease WHERE replica = 'worker-b'").execute(&pool).await.unwrap(); + let next = weft_dispatcher::reaper::in_motion(&pool).await.unwrap().expect("its vouch could lapse"); + assert!(next > Duration::from_secs(600) && next <= Duration::from_secs(620), "looked at once the vouch could lapse: {next:?}"); + assert_eq!(journal.let_go_of_lost(run, now + 700, None).await.unwrap(), Lost::Ended, "its vouch passed and its worker gone: lost"); +} + +/// A cancel asked of a run a worker drives, and an answer handed to one, +/// wake this process's news teller once they commit, which tells them. +#[sqlx::test] +async fn a_cancel_and_a_handed_answer_are_told_as_news(pool: PgPool) { + let (journal, projects) = setup(&pool).await; + let id = Uuid::new_v4(); + seed_project(&projects, id, "bin-A").await; + let bus = std::sync::Arc::new(weft_platform_traits::FakeNews::default()); + weft_task_store::announce::tell_news_through(&pool, bus.clone()); + let told = |kind: weft_platform_traits::NewsKind, run: weft_core::ExecutionId| { + let bus = bus.clone(); + async move { + tokio::time::timeout(std::time::Duration::from_secs(10), async { + while !bus.announced.lock().iter().flatten().any(|news| news.kind == kind && news.execution == run) { + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + } + }) + .await + .unwrap_or_else(|_| panic!("{kind:?} of {run} told")); + } + }; + let answered = queued_run(&journal, id).await; + claim(&pool, answered, id, "worker-a").await; + seed_wait(&journal, "tok-told", id, answered).await; + assert!(matches!(journal.answer("tok-told", &weft_core::primitive::WaitAnswer::Given { value: json!(1) }).await.unwrap(), weft_dispatcher::journal::Answered::Reached { .. })); + told(weft_platform_traits::NewsKind::Answer, answered).await; + let cancelled = queued_run(&journal, id).await; + claim(&pool, cancelled, id, "worker-a").await; + assert!(journal.cancel_execution(cancelled, None, &weft_core::exec::CancelCause::User).await.unwrap().requested); + told(weft_platform_traits::NewsKind::Cancel, cancelled).await; +} + /// A cancel ends a run nobody drives on the spot, after its last row; a /// run a worker drives is asked to stop, and its worker writes the ending. #[sqlx::test] @@ -791,6 +876,8 @@ async fn a_cancel_ends_a_waiting_run_and_asks_a_driven_one_to_stop(pool: PgPool) claim(&pool, driven, id, "worker-a").await; let written = journal.cancel_execution(driven, None, &cause).await.unwrap(); assert!(written.requested, "its worker is asked"); + assert_eq!(news_of(&pool, driven).await, vec![(id, "cancel".to_string())], "and the install's news says so, with the write"); + assert!(news_of(&pool, waiting).await.is_empty(), "a run nobody drives makes no news"); assert_eq!(written.node_cancellations, None); assert_eq!(run_column::(&pool, "state", driven).await.as_deref(), Some("running"), "its worker ends it"); let asked: Option = run_column(&pool, "cancel_requested", driven).await; @@ -1071,7 +1158,18 @@ async fn an_answer_reaches_its_run_once(pool: PgPool) { let driven = queued_run(&journal, project).await; claim(&pool, driven, project, "worker-a").await; seed_wait(&journal, "tok-driven", project, driven).await; + static CHANNELS: &[&str] = &[weft_task_store::parked_fires::PARKED_FIRE_CHANNEL]; + let watch = weft_task_store::pg_signal::PgSignalWatch::start(&pool.connect_options(), CHANNELS, weft_task_store::pg_signal::Listening::Always).await.unwrap(); + let mut heard = watch.subscribe(); assert!(matches!(journal.answer("tok-driven", &weft_core::primitive::WaitAnswer::Given { value: json!(2) }).await.unwrap(), Answered::Reached { .. })); + assert_eq!(news_of(&pool, driven).await, vec![(project, "answer".to_string())], "the install's news says it was handed, with the write"); + assert!(news_of(&pool, parked_run).await.is_empty(), "an answer written into a record makes no news"); + let named = weft_task_store::runs::run_payload(project, driven); + let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5); + assert!( + heard.woken_before(deadline, |c, p| c == weft_task_store::parked_fires::PARKED_FIRE_CHANNEL && p == named).await.unwrap(), + "the database announces the answer handed, naming its run" + ); assert_eq!(journal.events_log(driven).await.unwrap().len(), 1, "nothing written into a record its worker writes"); assert_eq!( weft_task_store::parked_fires::answers_for(&pool, driven).await.unwrap(), diff --git a/crates/weft-dispatcher/tests/db_proxy.rs b/crates/weft-dispatcher/tests/db_proxy.rs new file mode 100644 index 00000000..8abe9297 --- /dev/null +++ b/crates/weft-dispatcher/tests/db_proxy.rs @@ -0,0 +1,62 @@ +//! Layer-3 test of the door passing a caller on (`weft_dispatcher::proxy`) +//! against a REAL Postgres: a request held on its upstream (a route whose +//! run waits on a person) holds nothing that keeps the database listened +//! to, from before it waits on the upstream. +#![cfg(feature = "db-tests")] + +use std::net::SocketAddr; +use std::sync::Arc; +use std::time::Duration; + +use axum::extract::ConnectInfo; +use axum::routing::get; +use sqlx::PgPool; +use tower::ServiceExt as _; +use weft_task_store::pg_signal::{keep_listening, Listening, PgSignalWatch}; + +#[sqlx::test] +async fn a_request_waiting_on_its_upstream_holds_the_watch_no_longer(pool: PgPool) { + static CHANNELS: &[&str] = &["weft_test_proxy"]; + let watch = PgSignalWatch::start(&pool.connect_options(), CHANNELS, Listening::WhileBusy).await.unwrap(); + // An upstream that answers once released, as a held route does. + let (reached, release) = (Arc::new(tokio::sync::Notify::new()), Arc::new(tokio::sync::Notify::new())); + let upstream = axum::Router::new().route( + "/held", + get({ + let (reached, release) = (reached.clone(), release.clone()); + move || { + let (reached, release) = (reached.clone(), release.clone()); + async move { + reached.notify_one(); + release.notified().await; + "answered" + } + } + }), + ); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + tokio::spawn(async move { axum::serve(listener, upstream).await.unwrap() }); + + let http = reqwest::Client::new(); + let door = axum::Router::new() + .route( + "/held", + get(move |request: axum::extract::Request| { + let http = http.clone(); + async move { + let upstream = weft_dispatcher::proxy::Upstream { what: "the upstream", base_url: format!("http://{address}"), auth: None, hold: None }; + weft_dispatcher::proxy::forward(&http, upstream, "/held".into(), request, weft_dispatcher::proxy::Forwarding::Proxied).await + } + }), + ) + .layer(axum::middleware::from_fn_with_state(watch.clone(), keep_listening)); + let mut request = axum::http::Request::get("/held").header("host", "door").body(axum::body::Body::empty()).unwrap(); + request.extensions_mut().insert(ConnectInfo::("127.0.0.1:1".parse().unwrap())); + let call = tokio::spawn(door.oneshot(request)); + tokio::time::timeout(Duration::from_secs(10), reached.notified()).await.expect("the upstream was reached"); + assert!(!watch.worked_since(tokio::time::Instant::now()), "nothing holds the watch while the upstream answers"); + release.notify_one(); + let answer = call.await.unwrap().unwrap(); + assert_eq!(axum::body::to_bytes(answer.into_body(), usize::MAX).await.unwrap(), "answered"); +} diff --git a/crates/weft-engine/src/context.rs b/crates/weft-engine/src/context.rs index b3e48c0c..8ab4fa0f 100644 --- a/crates/weft-engine/src/context.rs +++ b/crates/weft-engine/src/context.rs @@ -135,11 +135,13 @@ impl ProcessSettings { } /// What a worker reads off its line to the broker besides answers: the -/// notices the broker pushes, and a hold that keeps the line open while -/// the worker must hear them (`weft_broker_client::line`'s module doc). +/// notices the broker pushes, a hold that keeps the line open while the +/// worker must hear them, and when the broker cuts the line open now +/// (`weft_broker_client::line`'s module doc). pub trait WorkerLine: Send + Sync { fn subscribe(&self) -> weft_task_store::pg_signal::Subscription; fn stay_open(&self) -> Box; + fn cut_at(&self) -> tokio::sync::watch::Receiver>; } impl WorkerLine for weft_broker_client::BrokerLink { @@ -150,20 +152,26 @@ impl WorkerLine for weft_broker_client::BrokerLink { fn stay_open(&self) -> Box { Box::new(weft_broker_client::BrokerLink::stay_open(self)) } + + fn cut_at(&self) -> tokio::sync::watch::Receiver> { + weft_broker_client::BrokerLink::cut_at(self) + } } -/// A line whose notices are what a test sends into [`TestLine::pushed`]. +/// A line whose notices are what a test sends into [`TestLine::pushed`], +/// and whose cut is what it sets in [`TestLine::cut`]. #[cfg(test)] pub(crate) struct TestLine { pub pushed: tokio::sync::broadcast::Sender, /// How many holds are taken right now. pub held: Arc, + pub cut: tokio::sync::watch::Sender>, } #[cfg(test)] impl TestLine { pub(crate) fn new() -> Arc { - Arc::new(Self { pushed: tokio::sync::broadcast::channel(64).0, held: Arc::default() }) + Arc::new(Self { pushed: tokio::sync::broadcast::channel(64).0, held: Arc::default(), cut: tokio::sync::watch::channel(None).0 }) } } @@ -173,6 +181,10 @@ impl WorkerLine for TestLine { self.pushed.subscribe().into() } + fn cut_at(&self) -> tokio::sync::watch::Receiver> { + self.cut.subscribe() + } + fn stay_open(&self) -> Box { struct Hold(Arc); impl Drop for Hold { @@ -268,8 +280,12 @@ pub trait RunClient: Send + Sync { /// written, for `why`. async fn let_go(&self, execution_id: ExecutionId, why: weft_broker_client::protocol::LetGo) -> anyhow::Result<()>; /// The answers waiting for `execution_id`'s waits, but the waits in - /// `taken`, held up to `wait` until one is there. - async fn answers(&self, execution_id: ExecutionId, taken: &[String], wait: Duration) -> anyhow::Result>; + /// `taken`, read once. + async fn answers(&self, execution_id: ExecutionId, taken: &[String]) -> anyhow::Result>; + /// What the broker pushes about the runs this worker drives: an answer + /// handed to one (`weft_broker_client::line::PARKED_FIRE_CHANNEL`, + /// naming the run), and a recheck when the line may have missed some. + fn told(&self) -> weft_task_store::pg_signal::Subscription; /// The cancels waiting for the runs this worker drives. async fn cancels(&self) -> anyhow::Result>; } @@ -282,8 +298,11 @@ impl RunClient for weft_broker_client::BrokerRunClient { async fn let_go(&self, execution_id: ExecutionId, why: weft_broker_client::protocol::LetGo) -> anyhow::Result<()> { weft_broker_client::BrokerRunClient::let_go(self, execution_id, why).await } - async fn answers(&self, execution_id: ExecutionId, taken: &[String], wait: Duration) -> anyhow::Result> { - weft_broker_client::BrokerRunClient::answers(self, execution_id, taken, wait).await + async fn answers(&self, execution_id: ExecutionId, taken: &[String]) -> anyhow::Result> { + weft_broker_client::BrokerRunClient::answers(self, execution_id, taken).await + } + fn told(&self) -> weft_task_store::pg_signal::Subscription { + weft_broker_client::BrokerRunClient::told(self) } async fn cancels(&self) -> anyhow::Result> { weft_broker_client::BrokerRunClient::cancels(self).await diff --git a/crates/weft-engine/src/door/limits.rs b/crates/weft-engine/src/door/limits.rs index b6058e73..fa96a672 100644 --- a/crates/weft-engine/src/door/limits.rs +++ b/crates/weft-engine/src/door/limits.rs @@ -11,8 +11,11 @@ //! exact. //! //! "At once" is held per copy: each takes its share of the limit, the limit -//! divided by the copies alive (a copy that sleeps lets its lease lapse, and -//! so is not counted). +//! divided by the copies alive. A copy that sleeps with nothing under way +//! lets its lease lapse, and so is not counted; one that sleeps because its +//! runs only wait outside vouched for its lease until the first of those +//! waits gives up, and one lease length past it (`crate::run_holds`), and +//! is counted until then, with the runs it last said were in flight. //! //! Right after a copy wakes from a sleep, and until its first tick comes //! back (under a second normally, a few seconds when the broker itself is @@ -218,11 +221,11 @@ impl Limits { counts.reported = counts.reported.max(mark); } - /// Nothing of this door is under way: no run going, and the broker - /// heard every count. - pub fn settled(&self) -> bool { + /// Whether no run of this door is going, and whether the broker heard + /// every count. + pub fn idle_and_reported(&self) -> (bool, bool) { let counts = self.state.lock().expect("door limits"); - counts.going.is_empty() && counts.changes == counts.reported + (counts.going.is_empty(), counts.changes == counts.reported) } /// The tick loop goes to sleep: from here on this copy hears nothing, diff --git a/crates/weft-engine/src/door/mod.rs b/crates/weft-engine/src/door/mod.rs index b4b78b46..01e2e2dd 100644 --- a/crates/weft-engine/src/door/mod.rs +++ b/crates/weft-engine/src/door/mod.rs @@ -48,7 +48,7 @@ use weft_core::signal::{LiveConnectionConfig, Protocol}; pub use broker::{DoorBroker, HeldDoorBroker, HeldEntry, HeldTrigger, RunFacts, Triggers}; pub use limits::{Going, Limits}; pub use permits::{Busy, RunPermit, RunPermits}; -pub use ticks::WorkUnderWay; +pub use ticks::{UnderWay, WorkUnderWay}; /// What the install allows at its public edge, as the worker holds it. #[derive(Debug, Clone, Copy, Default)] @@ -569,24 +569,26 @@ impl Door { /// While the worker has work under way, every second: state this /// worker's counts, hear the other copies and keep the worker's lease - /// (`/v1/door/tick`); with nothing under way, nothing (`ticks`). A tick - /// that fails is made again the next second; the limits hold to what - /// was last heard. `busy` is what the worker holds besides the door's - /// own runs. + /// (`/v1/door/tick`); with nothing under way, or only runs that wait + /// outside, nothing once a tick vouched for the worker (`ticks`). A + /// tick that fails is made again the next second; the limits hold to + /// what was last heard. `under_way` is what the worker says it has + /// under way. /// /// The first tick is made before this returns, so the worker is known /// alive before it takes its first call: a worker that cannot make it /// does not start, and says why. - pub async fn start_ticks(self: &Arc, busy: WorkUnderWay) -> anyhow::Result<()> { - let quiet = self.tick().await.map_err(|e| e.context("the worker's first word to the broker (its lease, and its door's counts)"))?; + pub async fn start_ticks(self: &Arc, under_way: WorkUnderWay) -> anyhow::Result<()> { + let vouch = under_way().vouch_until(); + let nothing_in_flight = self.tick(vouch).await.map_err(|e| e.context("the worker's first word to the broker (its lease, and its door's counts)"))?; let door = Arc::downgrade(self); let ticking = self.ticking.clone(); tokio::spawn(async move { let every_second = weft_core::time_scale::scaled(ticks::TICK_EVERY); let mut every = tokio::time::interval_at(tokio::time::Instant::now() + every_second, every_second); every.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); - // The last tick landed and reported nothing in flight. - let mut quiet = quiet; + // What the last tick said, when it landed. + let mut landed = Some(ticks::Landed { nothing_in_flight, vouched_until: vouch }); // The loop slept since its last tick: the next one goes at once. let mut slept = false; loop { @@ -594,7 +596,10 @@ impl Door { // The last tick said all there is: sleep until work // comes, then look again (a wake left while the loop // was ticking finds nothing new and sleeps on). - Some(strong) if quiet && strong.limits.settled() && !busy() => { + Some(strong) if { + let (door_idle, counts_reported) = strong.limits.idle_and_reported(); + ticks::may_sleep(under_way(), landed, door_idle, counts_reported) + } => { // Asleep, this copy hears nothing of the others, // whose leases may lapse meanwhile. strong.limits.asleep(); @@ -614,11 +619,12 @@ impl Door { every.tick().await; } let Some(door) = door.upgrade() else { return }; - quiet = match door.tick().await { - Ok(quiet) => quiet, + let vouch = under_way().vouch_until(); + landed = match door.tick(vouch).await { + Ok(nothing_in_flight) => Some(ticks::Landed { nothing_in_flight, vouched_until: vouch }), Err(e) => { tracing::warn!(target: "weft_engine::door", error = %format!("{e:#}"), "a tick of the door's counts failed; the next second tries again"); - false + None } }; } @@ -626,9 +632,10 @@ impl Door { Ok(()) } - /// One tick: whether it reported nothing in flight. Its counts are - /// marked heard once it lands. - async fn tick(&self) -> anyhow::Result { + /// One tick, vouching for the worker until `vouch_until` when it is + /// about to sleep with only waiting runs: whether it reported nothing + /// in flight. Its counts are marked heard once it lands. + async fn tick(&self, vouch_until: Option) -> anyhow::Result { let now = crate::now_unix() as i64; let (window_start, mine, mark) = self.limits.mine(now); let in_flight = self.limits.going(); @@ -641,6 +648,7 @@ impl Door { window_start, counts: mine.into_iter().map(|(key, hits)| DoorCount { key, hits }).collect(), tokens, + vouch_until, }; self.broker.tick(&request).await } @@ -650,10 +658,10 @@ impl Door { Ok(quiet) } - /// A run came onto this worker other than through the door (a claim of - /// a queued run), once it is counted where the tick looks: the tick - /// goes out now and renews the lease behind the run, which does not - /// wait for it (`ticks`). + /// Something the tick looks at changed (a run came onto this worker, or + /// one of its runs went quiet or moved again), once it is counted where + /// the tick looks: the tick goes out now and renews the lease behind + /// it, which does not wait for it (`ticks`). pub(crate) fn wake(&self) { self.ticking.wake(); } diff --git a/crates/weft-engine/src/door/ticks.rs b/crates/weft-engine/src/door/ticks.rs index 1a679c7e..56c37613 100644 --- a/crates/weft-engine/src/door/ticks.rs +++ b/crates/weft-engine/src/door/ticks.rs @@ -15,6 +15,17 @@ //! reported nothing in flight and every count, so the broker's picture is //! the latest one. //! +//! A worker whose every run only waits on something outside it +//! (`crate::run_holds`) sleeps too, once a tick landed with every count and +//! vouched for the worker until the earliest of those waits gives up, and a +//! lease's length past it: the broker keeps its lease that long, so those +//! runs, the door's own included, stay the worker's and the copy stays +//! counted with what it last reported, and a database that sleeps once +//! nothing queries it can sleep through a wait of hours. When that wait +//! ends its run moves and the tick goes again, vouching for the next one +//! (shorter or longer: it vouches again whenever what it would vouch for +//! changed). +//! //! The lease is what the tick keeps: a run on record as this worker's is //! read as lost once the lease lapsed and a margin as long again went by //! (`weft_dispatcher::worker_door`). Every way a run comes onto this worker @@ -27,9 +38,66 @@ use std::time::Duration; /// How often the loop ticks while something is under way (scaled). pub(super) const TICK_EVERY: Duration = Duration::from_secs(1); -/// What the worker says it still holds, besides the door's own runs: true -/// while the tick has to go on. -pub type WorkUnderWay = Box bool + Send + Sync>; +/// What the worker says it has under way, besides the door's own count. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum UnderWay { + /// Nothing: the door sleeps once its own runs are done too. + Nothing, + /// Runs, every one of them (the door's own included) only waiting on + /// something outside it, and every record written: the door sleeps + /// once a tick landed with every count and vouched for the worker until + /// `vouch_until` (unix seconds). + OnlyQuiet { vouch_until: i64 }, + /// Work: the tick goes on. + Working, +} + +impl UnderWay { + /// What a worker has under way, from whether its records are all + /// written, how many runs it drives, how many of those only wait and + /// the earliest moment one of those waits gives up, and how long its + /// lease holds without a tick (`lease_secs`). + pub fn of(records_written: bool, driving: usize, (quiet, earliest): (usize, Option), lease_secs: i64) -> Self { + match (records_written, driving, earliest) { + (false, _, _) => UnderWay::Working, + (true, 0, _) => UnderWay::Nothing, + (true, driving, Some(earliest)) if driving <= quiet => UnderWay::OnlyQuiet { vouch_until: earliest.saturating_add(lease_secs) }, + (true, _, _) => UnderWay::Working, + } + } + + /// What the tick vouches for: until when the worker's lease must hold + /// without another tick. + pub fn vouch_until(self) -> Option { + match self { + UnderWay::OnlyQuiet { vouch_until } => Some(vouch_until), + UnderWay::Nothing | UnderWay::Working => None, + } + } +} + +/// What the worker says it has under way, asked before each sleep. +pub type WorkUnderWay = Box UnderWay + Send + Sync>; + +/// What the last tick that landed said: whether nothing was in flight, +/// and until when it vouched for the worker. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) struct Landed { + pub(super) nothing_in_flight: bool, + pub(super) vouched_until: Option, +} + +/// Whether the tick loop may sleep: the worker has `under_way`, the last +/// tick `landed` (`None` when it failed), the door's own runs are all done +/// (`door_idle`) and every count it took was reported (`counts_reported`). +pub(super) fn may_sleep(under_way: UnderWay, landed: Option, door_idle: bool, counts_reported: bool) -> bool { + let Some(landed) = landed else { return false }; + match under_way { + UnderWay::Nothing => landed.nothing_in_flight && door_idle && counts_reported, + UnderWay::OnlyQuiet { vouch_until } => counts_reported && landed.vouched_until == Some(vouch_until), + UnderWay::Working => false, + } +} /// The tick's sleep and wake (see the module doc). pub struct Ticking { @@ -86,9 +154,14 @@ mod tests { /// A door serving `x`, ticking, with nothing under way but what /// `busy` says. async fn ticking(busy: impl Fn() -> bool + Send + Sync + 'static) -> (Arc, Arc) { + ticking_with(move || if busy() { UnderWay::Working } else { UnderWay::Nothing }).await + } + + /// A door serving `x`, ticking, the worker saying it has `under_way`. + async fn ticking_with(under_way: impl Fn() -> UnderWay + Send + Sync + 'static) -> (Arc, Arc) { let broker = FakeDoorBroker::new(vec![route("t1", "route", "x")]); let door = door(broker.clone()); - door.start_ticks(Box::new(busy)).await.unwrap(); + door.start_ticks(Box::new(under_way)).await.unwrap(); (door, broker) } @@ -234,7 +307,7 @@ mod tests { armed(&mut trigger).spec.limits = weft_core::signal::EntryLimits { per_minute: Some(1), ..Default::default() }; let broker = FakeDoorBroker::new(vec![trigger]); let door = door(broker.clone()); - door.start_ticks(Box::new(|| false)).await.unwrap(); + door.start_ticks(Box::new(|| UnderWay::Nothing)).await.unwrap(); sleep(secs(20.0)).await; assert_eq!(ticks(&broker).len(), 1, "asleep since the first tick"); @@ -270,6 +343,75 @@ mod tests { assert_eq!(asked.elapsed(), Duration::ZERO, "no wait while ticking"); } + /// A run of the door that only waits outside lets the door sleep after + /// a tick that vouches for the worker until the earliest wait gives up, + /// even though the run is still going (and still counted); a later + /// wait, or a shorter one, vouches again before the door sleeps on; once + /// the run moves, the door ticks at once and vouches no more. + #[tokio::test(start_paused = true)] + async fn a_door_whose_only_run_waits_vouches_until_its_wait_ends_and_sleeps() { + let quiet = Arc::new(std::sync::Mutex::new(None::)); + let (door, broker) = ticking_with({ + let quiet = quiet.clone(); + move || match *quiet.lock().unwrap() { + Some(vouch_until) => UnderWay::OnlyQuiet { vouch_until }, + None => UnderWay::Working, + } + }) + .await; + let Arrived::Run(_admitted, _) = arrive(&door, &HeaderMap::new()).await else { panic!("admitted") }; + sleep(secs(3.5)).await; + *quiet.lock().unwrap() = Some(5_000); + sleep(secs(30.0)).await; + let asleep = ticks(&broker); + let last = asleep.last().unwrap(); + assert_eq!(last.vouch_until, Some(5_000), "the last tick vouched until the wait ends"); + assert_eq!(last.in_flight.get("t1"), Some(&1), "the waiting run is still going, and counted"); + assert_eq!(asleep.iter().filter(|tick| tick.vouch_until.is_some()).count(), 1, "one vouching tick, then sleep"); + + *quiet.lock().unwrap() = Some(9_000); + door.wake(); + sleep(secs(30.0)).await; + let again = ticks(&broker); + assert_eq!(again.len(), asleep.len() + 1, "a later wait vouches once more, then sleep"); + assert_eq!(again.last().unwrap().vouch_until, Some(9_000)); + + *quiet.lock().unwrap() = Some(7_000); + door.wake(); + sleep(secs(30.0)).await; + let shorter = ticks(&broker); + assert_eq!(shorter.len(), again.len() + 1, "a shorter wait vouches again too"); + assert_eq!(shorter.last().unwrap().vouch_until, Some(7_000)); + let again = shorter; + + *quiet.lock().unwrap() = None; + door.wake(); + sleep(secs(0.5)).await; + let moved = ticks(&broker); + assert_eq!(moved.len(), again.len() + 1, "the run moved: the door ticks at once"); + assert_eq!(moved.last().unwrap().vouch_until, None); + } + + #[test] + fn what_a_worker_has_under_way_decides_whether_its_door_sleeps() { + assert_eq!(UnderWay::of(false, 0, (0, None), 15), UnderWay::Working, "a record still to write"); + assert_eq!(UnderWay::of(true, 0, (0, None), 15), UnderWay::Nothing); + assert_eq!(UnderWay::of(true, 2, (2, Some(100)), 15), UnderWay::OnlyQuiet { vouch_until: 115 }, "the earliest wait, and a lease past it"); + assert_eq!(UnderWay::of(true, 2, (1, Some(100)), 15), UnderWay::Working, "one run still moves"); + assert_eq!(UnderWay::of(true, 1, (2, Some(100)), 15), UnderWay::OnlyQuiet { vouch_until: 115 }, "a quiet run a moment after it left"); + let landed = |vouched_until| Some(Landed { nothing_in_flight: false, vouched_until }); + let idle = Some(Landed { nothing_in_flight: true, vouched_until: None }); + assert!(may_sleep(UnderWay::Nothing, idle, true, true)); + assert!(!may_sleep(UnderWay::Nothing, idle, false, true), "a run of the door still going"); + assert!(may_sleep(UnderWay::OnlyQuiet { vouch_until: 115 }, landed(Some(115)), false, true), "the door's own runs are the quiet ones"); + assert!(!may_sleep(UnderWay::OnlyQuiet { vouch_until: 115 }, landed(None), false, true), "no tick vouched yet"); + assert!(!may_sleep(UnderWay::OnlyQuiet { vouch_until: 200 }, landed(Some(115)), false, true), "a later wait is not vouched for yet"); + assert!(!may_sleep(UnderWay::OnlyQuiet { vouch_until: 100 }, landed(Some(115)), false, true), "nor a shorter one: a dead worker must not keep its lease past it"); + assert!(!may_sleep(UnderWay::OnlyQuiet { vouch_until: 115 }, None, false, true), "the last tick failed"); + assert!(!may_sleep(UnderWay::OnlyQuiet { vouch_until: 115 }, landed(Some(115)), false, false), "a count not reported yet"); + assert!(!may_sleep(UnderWay::Working, idle, true, true)); + } + /// A wake that comes before the loop starts waiting is kept for it: the /// sleep that follows returns at once. #[tokio::test] diff --git a/crates/weft-engine/src/execution_driver.rs b/crates/weft-engine/src/execution_driver.rs index f2b472fc..2ee3a012 100644 --- a/crates/weft-engine/src/execution_driver.rs +++ b/crates/weft-engine/src/execution_driver.rs @@ -201,6 +201,10 @@ pub(crate) struct RunDrive<'a> { /// How the run is asked to let go of this worker. `None` for a run /// nobody asks that of. pub(crate) hand_back: Option<&'a HandBack>, + /// How the run says it only waits on something outside it, so its + /// worker may stop ticking (`crate::run_holds`). `None` for a run no + /// worker's tick looks at. + pub(crate) hold: Option, } /// Run one execution to a terminal state or a pause. `replica` stamps @@ -242,7 +246,7 @@ pub(crate) async fn run_one_execution_observed( replica: &str, tenant_id: &str, ) -> anyhow::Result { - let RunDrive { execution_id, journal, program, starts_from, cancellation, exchange, hand_back } = run; + let RunDrive { execution_id, journal, program, starts_from, cancellation, exchange, hand_back, hold } = run; // The run's spend still being worked out, which goes on its record // before its ending (`crate::metering`). let record = RunRecord { journal: journal.clone(), costs: crate::metering::PendingCostRecords::new() }; @@ -284,6 +288,7 @@ pub(crate) async fn run_one_execution_observed( waits, bus_coordinator.clone(), hand_back, + hold.as_ref(), ))) .await { @@ -583,6 +588,7 @@ async fn drive_execution_id( waits: Arc, bus_coordinator: Arc, hand_back: Option<&HandBack>, + hold: Option<&crate::run_holds::RunHold>, ) -> anyhow::Result { let project_arc = program.project.clone(); let project = &*project_arc; @@ -683,6 +689,7 @@ async fn drive_execution_id( settings, &mut fold, hand_back, + hold, ) .await?; @@ -1295,6 +1302,8 @@ async fn drive( fold: &mut RunFold, // See `run_one_execution`. hand_back: Option<&HandBack>, + // See `RunDrive::hold`. + hold: Option<&crate::run_holds::RunHold>, ) -> anyhow::Result { let project: &ProjectDefinition = &program.project; let program_idx = &program.index; @@ -1356,16 +1365,20 @@ async fn drive( // The answers the run waits for while this worker keeps it: a wait // held in a node's call (`crate::held_waits`), and a suspended step // while a bus keeps the worker (drive() never returns while a bus is - // open, so without this its answer would never reach it). The ask is - // one held read (it ends when an answer comes, or the hold runs out), - // kept across turns of the loop: a chatty bus turns the loop many - // times a second, and an ask started afresh on every turn would ask - // the broker as often. `taken` is every wait whose answer this drive - // took, which the broker leaves out. + // open, so without this its answer would never reach it). The ask + // ([`answers_once_told`]) reads once as the run starts waiting and + // again only when the line says an answer was handed to it, kept + // across turns of the loop: a chatty bus turns the loop many times a + // second, and an ask started afresh on every turn would read as often. + // `taken` is every wait whose answer this drive took, which the broker + // leaves out. let mut taken: Vec = Vec::new(); // When the run went quiet: every running step waiting, and one of // those waits ended only from outside (see the stuck-check below). let mut quiet_since: Option = None; + // When the run's hold gives up (unix seconds), while it is quiet + // (`crate::run_holds`): set once as it goes quiet. + let mut quiet_until: Option = None; let mut resume_wait: Option>>> = None; loop { // Poison checkpoint: a journal write failed somewhere since @@ -2674,20 +2687,23 @@ async fn drive( // otherwise, so the common path never asks. if waits.held().pending() || waiting_count(executions) > 0 { if resume_wait.is_none() { - let (runs, taken) = (clients.runs.clone(), taken.clone()); - resume_wait = Some(Box::pin(async move { - runs.answers(execution_id, &taken, weft_task_store::pg_signal::MAX_HOLD).await - })); + resume_wait = Some(Box::pin(answers_once_told(clients.runs.clone(), execution_id, taken.clone()))); } } else { resume_wait = None; } // The hold runs out `holdSecs` after the run went quiet. - let hold = std::time::Duration::from_secs(settings.hold_secs().into()); + let hold_for = std::time::Duration::from_secs(settings.hold_secs().into()); let quiet_for = quiet_since.map(|since| clients.clock.now().saturating_duration_since(since)); + // While quiet, its worker may stop ticking, vouching for itself + // until the run's hold gives up. + quiet_until = quiet_for.map(|quiet_for| quiet_until.unwrap_or_else(|| hold_ends_unix(crate::now_unix(), hold_for.saturating_sub(quiet_for)))); + if let Some(hold) = hold { + hold.quiet(quiet_until.map(|until| crate::run_holds::Quiet { until, can_pause: unsuspendable(caller, bus_coordinator, settings).is_none() })); + } let held_out = async { match quiet_for { - Some(quiet_for) => clients.clock.sleep(hold.saturating_sub(quiet_for)).await, + Some(quiet_for) => clients.clock.sleep(hold_for.saturating_sub(quiet_for)).await, None => std::future::pending().await, } }; @@ -2735,9 +2751,8 @@ async fn drive( // resumes it in place (`resume_answered`). Any other answer // is a held wait's: it goes to the node's call holding it, // which records it, or is kept for the step that holds or - // suspends on it next (`HeldWaits::answer`). An empty - // answer is the broker's hold running out: nothing moved, - // and the next turn asks again. + // suspends on it next (`HeldWaits::answer`). The ask answers + // only once it found some, so something moved. resume_wait = None; let answers = answers?; moved = !answers.is_empty(); @@ -2851,9 +2866,58 @@ async fn drive( if moved { quiet_since = None; } + // Moved, or its hold gave up: quiet no longer, before it goes on. + if quiet_since.is_none() && quiet_until.take().is_some() { + if let Some(hold) = hold { + hold.quiet(None); + } + } } } +/// The answers waiting for run `execution_id`'s waits but the ones +/// `taken`, once there are some: read now (the run starts waiting, or just +/// took an answer), and again each time the line says one was handed to +/// this run, or may have missed some. Nothing else reads: a run that waits a +/// day leaves the database asleep all that time. +async fn answers_once_told( + runs: Arc, + execution_id: ExecutionId, + taken: Vec, +) -> anyhow::Result> { + // Subscribed before the first read, so an answer handed between an + // empty read and the wait still wakes it. + let mut told = runs.told(); + loop { + let answers = runs.answers(execution_id, &taken).await?; + if !answers.is_empty() { + return Ok(answers); + } + loop { + match told.next().await? { + weft_task_store::pg_signal::Heard::Signal { channel, payload } + if channel == weft_task_store::parked_fires::PARKED_FIRE_CHANNEL + && weft_task_store::runs::parse_run_payload(&payload).is_some_and(|(_, run)| run == execution_id) => + { + break + } + weft_task_store::pg_signal::Heard::Recheck => break, + // Another run's news, the line lost (a recheck follows once + // it is back), the broker hearing again after a quiet spell: + // nothing missed here. + _ => {} + } + } + } +} + +/// When the hold of a run that went quiet gives up (unix seconds), `now` +/// being the moment and `left` what is left of it, rounded up. +fn hold_ends_unix(now: u64, left: std::time::Duration) -> i64 { + let secs = now.saturating_add(left.as_secs()).saturating_add(u64::from(left.subsec_nanos() > 0)); + i64::try_from(secs).unwrap_or(i64::MAX) +} + /// Every running step waits on another and nothing outside will end it: /// unwind, gentlest first. Closing the buses may unwind everything /// (bodies return, producers terminate and close their streams diff --git a/crates/weft-engine/src/execution_driver_tests/born.rs b/crates/weft-engine/src/execution_driver_tests/born.rs index 1f18b482..835317d8 100644 --- a/crates/weft-engine/src/execution_driver_tests/born.rs +++ b/crates/weft-engine/src/execution_driver_tests/born.rs @@ -291,6 +291,7 @@ cancellation: weft_core::cancellation::CancellationFlag::new_arc(), exchange: None, hand_back: None, + hold: None, }; let catalog = catalog(vec![("Trig", Box::new(Answer)), ("Step", Box::new(Pass))]); let drove = crate::execution_driver::run_one_execution_observed(catalog, &clients, drive, "instance-test", "tenant-test") @@ -398,6 +399,7 @@ cancellation: weft_core::cancellation::CancellationFlag::new_arc(), exchange: Some(crate::execution_driver::Exchange { conn: conn.clone(), live: Some(conn.clone()), sink }), hand_back: None, + hold: None, }; let drove = tokio::time::timeout( std::time::Duration::from_secs(30), diff --git a/crates/weft-engine/src/execution_driver_tests/rig.rs b/crates/weft-engine/src/execution_driver_tests/rig.rs index 575a28f1..d97f8024 100644 --- a/crates/weft-engine/src/execution_driver_tests/rig.rs +++ b/crates/weft-engine/src/execution_driver_tests/rig.rs @@ -213,23 +213,40 @@ } } /// The broker's side of the runs a rig drives: the answers a test - /// gives, handed to whoever asks next (held until one comes, or the - /// hold runs out), and the runs let go of. - #[derive(Default)] + /// gives, handed to whoever reads next, each one told on the run's line + /// (as the broker pushes it); every read counted; the runs let go of. pub(crate) struct Answers { waiting: StdMutex>, - came: tokio::sync::Notify, + /// The run's line, as the broker pushes onto it. + pub(crate) line: tokio::sync::broadcast::Sender, + /// How many times the answers were read. + pub(crate) reads: std::sync::atomic::AtomicUsize, pub(crate) let_go: StdMutex>, } + impl Default for Answers { + fn default() -> Self { + Self { + waiting: StdMutex::default(), + line: tokio::sync::broadcast::channel(64).0, + reads: std::sync::atomic::AtomicUsize::new(0), + let_go: StdMutex::default(), + } + } + } impl Answers { - /// An answer to the wait `token`, as the install parks one for the - /// run's worker to take. + /// An answer to the wait `token`, as the install hands one to the + /// run's worker: kept for it, and told on its line (a recheck stands + /// for the run's own notice, which needs the run's id). pub(crate) fn answer(&self, token: impl Into, value: Value) { self.waiting.lock().unwrap().push(weft_broker_client::protocol::RunAnswer { token: token.into(), answer: weft_core::primitive::WaitAnswer::Given { value }, }); - self.came.notify_waiters(); + let _ = self.line.send(weft_task_store::pg_signal::Heard::Recheck); + } + + pub(crate) fn reads(&self) -> usize { + self.reads.load(std::sync::atomic::Ordering::SeqCst) } } #[async_trait] @@ -241,17 +258,12 @@ self.let_go.lock().unwrap().push((execution_id, why)); Ok(()) } - async fn answers(&self, _execution_id: ExecutionId, _taken: &[String], wait: std::time::Duration) -> anyhow::Result> { - let deadline = tokio::time::Instant::now() + wait; - loop { - let came = self.came.notified(); - tokio::pin!(came); - came.as_mut().enable(); - let answers = std::mem::take(&mut *self.waiting.lock().unwrap()); - if !answers.is_empty() || tokio::time::timeout_at(deadline, came).await.is_err() { - return Ok(answers); - } - } + async fn answers(&self, _execution_id: ExecutionId, _taken: &[String]) -> anyhow::Result> { + self.reads.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + Ok(std::mem::take(&mut *self.waiting.lock().unwrap())) + } + fn told(&self) -> weft_task_store::pg_signal::Subscription { + self.line.subscribe().into() } async fn cancels(&self) -> anyhow::Result> { Ok(Vec::new()) @@ -724,6 +736,26 @@ cancellation: Arc, caller: Option>, hand_back: Option<&HandBack>, + ) -> anyhow::Result { + // Quiet runs counted the way a worker counts them, with no door to + // tell. + let holds = crate::run_holds::Holds::new(Box::new(|| {})); + run_on_holding(project, catalog, clients, execution_id, journal, first, cancellation, caller, hand_back, &holds).await + } + + /// [`run_on`], the run's quiet spells counted in `holds`. + #[allow(clippy::too_many_arguments)] + pub(crate) async fn run_on_holding( + project: Arc, + catalog: Arc, + clients: &EngineClients, + execution_id: ExecutionId, + journal: Arc, + first: Vec, + cancellation: Arc, + caller: Option>, + hand_back: Option<&HandBack>, + holds: &Arc, ) -> anyhow::Result { // The tables a worker's claim builds for the run: the program and // the part of it its birth names. @@ -733,7 +765,8 @@ }); let program = Arc::new(crate::plan::ProgramTables::new(project, selection)); let starts_from = crate::execution_driver::StartsFrom::Record(first); - let run = RunDrive { execution_id, journal, program, starts_from, cancellation, exchange: caller.map(exchange), hand_back }; + let hold = Some(holds.of(execution_id)); + let run = RunDrive { execution_id, journal, program, starts_from, cancellation, exchange: caller.map(exchange), hand_back, hold }; tokio::time::timeout( std::time::Duration::from_secs(60), run_one_execution_observed(catalog, clients, run, "instance-test", "tenant-test"), diff --git a/crates/weft-engine/src/execution_driver_tests/unrecorded.rs b/crates/weft-engine/src/execution_driver_tests/unrecorded.rs index 7d367a5d..ad33611b 100644 --- a/crates/weft-engine/src/execution_driver_tests/unrecorded.rs +++ b/crates/weft-engine/src/execution_driver_tests/unrecorded.rs @@ -5,7 +5,7 @@ //! nowhere to park it. use super::*; - use super::engine_test_rig::{born, catalog, clients, run_on, test_manifest, Answers, AwaitTasks, MemJournal}; + use super::engine_test_rig::{born, catalog, clients, run_on_holding, test_manifest, Answers, AwaitTasks, MemJournal}; use async_trait::async_trait; use serde_json::json; use weft_core::error::WeftResult; @@ -57,14 +57,33 @@ /// Drive one unrecorded run of `node` the way the worker does. Answers /// the outcome and what the record holds of it once its writer is done. async fn drive_unrecorded(node: Box) -> (ExecutionOutcome, Vec) { - drive_holding(node, weft_core::run_settings::DEFAULT_HOLD_SECS, AwaitTasks::new(), Arc::new(Answers::default())).await + let holds = crate::run_holds::Holds::new(Box::new(|| {})); + drive_holding(node, weft_core::run_settings::DEFAULT_HOLD_SECS, AwaitTasks::new(), Arc::new(Answers::default()), &holds).await } /// [`drive_unrecorded`], the run holding its waits `hold_secs`, its - /// waits registered with `tasks` and answered through `answers`. - async fn drive_holding(node: Box, hold_secs: u32, tasks: Arc, answers: Arc) -> (ExecutionOutcome, Vec) { + /// waits registered with `tasks` and answered through `answers`, its + /// quiet spells counted in `holds`. + async fn drive_holding( + node: Box, + hold_secs: u32, + tasks: Arc, + answers: Arc, + holds: &Arc, + ) -> (ExecutionOutcome, Vec) { + drive_run(weft_core::new_execution_id(), node, hold_secs, tasks, answers, holds).await + } + + /// [`drive_holding`], of run `execution_id`. + async fn drive_run( + execution_id: weft_core::ExecutionId, + node: Box, + hold_secs: u32, + tasks: Arc, + answers: Arc, + holds: &Arc, + ) -> (ExecutionOutcome, Vec) { let project = project(); - let execution_id = weft_core::new_execution_id(); let birth = vec![ ExecEvent::ExecutionStarted { execution_id, @@ -91,7 +110,7 @@ let journal = Arc::new(MemJournal::default()); let run_clients = EngineClients { tasks, runs: answers, ..clients(journal.clone()) }; let run = born(&run_clients, execution_id, &birth).await; - let drove = run_on(Arc::new(project), catalog(vec![("Answer", node)]), &run_clients, execution_id, run, birth, CancellationFlag::new_arc(), None, None) + let drove = run_on_holding(Arc::new(project), catalog(vec![("Answer", node)]), &run_clients, execution_id, run, birth, CancellationFlag::new_arc(), None, None, holds) .await .expect("the drive ends"); run_clients.writer.written().await; @@ -118,19 +137,79 @@ } /// A wait holds in its node's call, and its answer carries the run on - /// in the same worker: nothing is written of it. + /// in the same worker: nothing is written of it. While it only waited, + /// the run counted as quiet (so its worker may stop ticking) until its + /// hold would give up, and it moved again once answered. #[tokio::test] async fn a_wait_holds_and_takes_its_answer() { let (tasks, answers) = (AwaitTasks::new(), Arc::new(Answers::default())); + let holds = crate::run_holds::Holds::new(Box::new(|| {})); + // Answered once the run is quiet, as a person answers a run that + // waited, after reading how long it waits. let answering = tokio::spawn({ - let (tasks, answers) = (tasks.clone(), answers.clone()); - async move { answers.answer(tasks.await_token().await, json!("now")) } + let (tasks, answers, holds) = (tasks.clone(), answers.clone(), holds.clone()); + async move { + let token = tasks.await_token().await; + let latest = loop { + if let (1, Some(latest)) = holds.quiet() { + break latest; + } + tokio::task::yield_now().await; + }; + answers.answer(token, json!("now")); + latest + } }); - let (outcome, written) = drive_holding(Box::new(Waits), 60, tasks.clone(), answers).await; - answering.await.unwrap(); + let asked_at = crate::now_unix() as i64; + let (outcome, written) = drive_holding(Box::new(Waits), 60, tasks.clone(), answers, &holds).await; + let quiet_until = answering.await.unwrap(); assert!(matches!(outcome, ExecutionOutcome::Completed), "{outcome:?}"); assert!(written.is_empty(), "nothing was written: {written:?}"); assert!(tasks.withdrawn.lock().unwrap().is_empty(), "an answered wait is not withdrawn"); + assert!((asked_at + 60..=crate::now_unix() as i64 + 61).contains(&quiet_until), "quiet until its hold gives up: {quiet_until}"); + assert_eq!(holds.quiet(), (0, None), "it moved again"); + } + + /// A waiting run reads its answers once as it starts waiting, and then + /// only when its line tells it one was handed to it or may have missed + /// some: another run's answer, the broker hearing again after a quiet + /// spell, and the line lost (with no recheck yet) read nothing. + #[tokio::test] + async fn a_waiting_run_reads_its_answers_only_when_told() { + let (tasks, answers) = (AwaitTasks::new(), Arc::new(Answers::default())); + let holds = crate::run_holds::Holds::new(Box::new(|| {})); + let run = weft_core::new_execution_id(); + let telling = tokio::spawn({ + let (tasks, answers, holds) = (tasks.clone(), answers.clone(), holds.clone()); + async move { + use weft_task_store::pg_signal::Heard; + let token = tasks.await_token().await; + while holds.quiet().0 == 0 { + tokio::task::yield_now().await; + } + let settled = || async { + for _ in 0..50 { + tokio::task::yield_now().await; + } + }; + settled().await; + assert_eq!(answers.reads(), 1, "read once as it started waiting"); + let named = |run| Heard::Signal { channel: weft_task_store::parked_fires::PARKED_FIRE_CHANNEL, payload: weft_task_store::runs::run_payload(uuid::Uuid::from_u128(9), run).into() }; + answers.line.send(named(weft_core::new_execution_id())).unwrap(); + answers.line.send(Heard::Resumed).unwrap(); + answers.line.send(Heard::Lost).unwrap(); + settled().await; + assert_eq!(answers.reads(), 1, "nothing for this run: nothing read"); + answers.line.send(named(run)).unwrap(); + settled().await; + assert_eq!(answers.reads(), 2, "told one was handed to it: one read"); + answers.answer(token, json!("now")); + } + }); + let (outcome, _) = drive_run(run, Box::new(Waits), 60, tasks, answers.clone(), &holds).await; + telling.await.unwrap(); + assert!(matches!(outcome, ExecutionOutcome::Completed), "{outcome:?}"); + assert_eq!(answers.reads(), 3, "the recheck its answer came with read it"); } /// A wait nothing answers is given up once the run was quiet for its @@ -138,7 +217,7 @@ #[tokio::test] async fn a_wait_nothing_answers_is_given_up_after_its_hold() { let tasks = AwaitTasks::new(); - let (outcome, written) = drive_holding(Box::new(Waits), 1, tasks.clone(), Arc::new(Answers::default())).await; + let (outcome, written) = drive_holding(Box::new(Waits), 1, tasks.clone(), Arc::new(Answers::default()), &crate::run_holds::Holds::new(Box::new(|| {}))).await; assert!(matches!(outcome, ExecutionOutcome::Failed { .. }), "{outcome:?}"); let error = written .iter() @@ -157,7 +236,7 @@ #[tokio::test] async fn a_hold_of_zero_fails_the_wait_at_once() { let tasks = AwaitTasks::new(); - let (outcome, _) = drive_holding(Box::new(Waits), 0, tasks.clone(), Arc::new(Answers::default())).await; + let (outcome, _) = drive_holding(Box::new(Waits), 0, tasks.clone(), Arc::new(Answers::default()), &crate::run_holds::Holds::new(Box::new(|| {}))).await; assert!(matches!(&outcome, ExecutionOutcome::Failed { error } if error.contains("is 0")), "{outcome:?}"); assert_eq!(*tasks.withdrawn.lock().unwrap(), vec![tasks.minted().expect("the wait was registered")]); } diff --git a/crates/weft-engine/src/lib.rs b/crates/weft-engine/src/lib.rs index 02a6388a..137adda5 100644 --- a/crates/weft-engine/src/lib.rs +++ b/crates/weft-engine/src/lib.rs @@ -18,6 +18,7 @@ pub(crate) mod metering; pub(crate) mod plan; pub mod profile; pub(crate) mod record_first; +pub(crate) mod run_holds; pub(crate) mod socket; pub(crate) mod stream_runtime; pub(crate) mod wait_tracker; diff --git a/crates/weft-engine/src/run_holds.rs b/crates/weft-engine/src/run_holds.rs new file mode 100644 index 00000000..85e66e21 --- /dev/null +++ b/crates/weft-engine/src/run_holds.rs @@ -0,0 +1,150 @@ +//! The runs a worker drives that only wait on something outside them (a +//! person's answer, a cancel), and until when each one waits at most. +//! +//! A worker ticks the broker every second while it has work under way +//! (`crate::door::ticks`), and each tick writes its lease: a database that +//! sleeps once nothing queries it would never sleep under a run that waits +//! a day for a person. So while every run a worker drives only waits +//! outside, its last tick vouches for it until the earliest moment one of +//! those waits gives up, and a lease's length past it, and then it stops +//! ticking: its lease holds until then, which keeps its runs its own and +//! keeps it counted among its program's copies. A run that moves again (its +//! wait answered, or given up at that earliest moment), or a new run +//! coming, wakes the tick, which vouches again for the next earliest. +//! +//! If the worker dies during the wait, its runs are judged lost once the +//! vouched lease passed: about when the first of their waits would have +//! given up, and the others died with the worker. + +use std::collections::HashMap; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; + +use weft_core::ExecutionId; + +/// How a run that only waits outside waits. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct Quiet { + /// When its wait gives up at most (unix seconds). + pub(crate) until: i64, + /// Whether it could pause right now (steps waiting on each other + /// beside a suspended one), so a stopping worker hands it back, rather + /// than holding its waits in the worker + /// (`crate::held_waits::Unsuspendable`). + pub(crate) can_pause: bool, +} + +/// Every run of this worker that only waits outside, by how it waits, and +/// how the door hears of a change. +pub(crate) struct Holds { + quiet: Mutex>, + /// Told whenever a run goes quiet or moves again, so the door's tick + /// looks again at whether, and for how long, it may sleep + /// (`crate::door::Door::wake`). + changed: Box, +} + +impl Holds { + pub(crate) fn new(changed: Box) -> Arc { + Arc::new(Self { quiet: Mutex::default(), changed }) + } + + /// How many runs only wait, and the earliest moment one of those waits + /// gives up. + pub(crate) fn quiet(&self) -> (usize, Option) { + let quiet = self.quiet.lock().expect("run holds"); + (quiet.len(), quiet.values().map(|quiet| quiet.until).min()) + } + + /// The runs that only wait, by how each waits, held still: no run goes + /// quiet or moves again until the guard goes. + pub(crate) fn held_still(&self) -> std::sync::MutexGuard<'_, HashMap> { + self.quiet.lock().expect("run holds") + } + + /// Run `execution_id`'s drive says how it waits ([`RunHold`]). Nothing + /// is kept for it until it first goes quiet. + pub(crate) fn of(self: &Arc, execution_id: ExecutionId) -> RunHold { + RunHold { holds: self.clone(), execution_id, quiet: AtomicBool::new(false) } + } + + fn set(&self, execution_id: ExecutionId, how: Option) { + let changed = { + let mut quiet = self.quiet.lock().expect("run holds"); + match how { + Some(how) => quiet.insert(execution_id, how) != Some(how), + None => quiet.remove(&execution_id).is_some(), + } + }; + if changed { + (self.changed)(); + } + } +} + +/// One run's place among the [`Holds`], as its drive says it; it moves +/// again (or ends) when this is dropped. A run that never goes quiet never +/// touches the [`Holds`]. +pub(crate) struct RunHold { + holds: Arc, + execution_id: ExecutionId, + /// Whether the run is among the quiet ones. + quiet: AtomicBool, +} + +impl RunHold { + /// The run only waits outside, as `how` says, or it moves again + /// (`None`). Saying the same again changes nothing. + pub(crate) fn quiet(&self, how: Option) { + if how.is_none() && !self.quiet.load(Ordering::Acquire) { + return; + } + self.quiet.store(how.is_some(), Ordering::Release); + self.holds.set(self.execution_id, how); + } +} + +impl Drop for RunHold { + fn drop(&mut self) { + self.quiet(None); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::AtomicUsize; + + /// A run counts as quiet from the moment its drive says so until it + /// moves or its drive ends; the earliest wait is what the door vouches + /// for; only a change tells the door. + #[test] + fn a_run_is_quiet_until_it_moves_or_ends() { + let changes = Arc::new(AtomicUsize::new(0)); + let holds = Holds::new(Box::new({ + let changes = changes.clone(); + move || { + changes.fetch_add(1, Ordering::SeqCst); + } + })); + let until = |until| Some(Quiet { until, can_pause: false }); + let (a, b) = (holds.of(uuid::Uuid::from_u128(1)), holds.of(uuid::Uuid::from_u128(2))); + assert_eq!(holds.quiet(), (0, None), "nothing kept before a run goes quiet"); + a.quiet(until(100)); + b.quiet(until(300)); + a.quiet(until(100)); + assert_eq!(holds.quiet(), (2, Some(100))); + assert_eq!(changes.load(Ordering::SeqCst), 2, "saying the same again tells nobody"); + a.quiet(None); + assert_eq!(holds.quiet(), (1, Some(300)), "the first wait gave up: the next earliest"); + a.quiet(until(100)); + b.quiet(None); + assert_eq!(holds.quiet(), (1, Some(100))); + a.quiet(Some(Quiet { until: 100, can_pause: true })); + assert_eq!(changes.load(Ordering::SeqCst), 6, "a run that can pause now is a change"); + drop(a); + assert_eq!(holds.quiet(), (0, None)); + drop(b); + assert_eq!(changes.load(Ordering::SeqCst), 7, "a run that moved and then ended told the door once"); + } +} diff --git a/crates/weft-engine/src/test_rig.rs b/crates/weft-engine/src/test_rig.rs index 6c092ef6..b0a71332 100644 --- a/crates/weft-engine/src/test_rig.rs +++ b/crates/weft-engine/src/test_rig.rs @@ -348,6 +348,7 @@ async fn keep_lease(broker: Arc) { window_start: weft_core::signal::limits::window(crate::now_unix() as i64).0, counts: Vec::new(), tokens: Vec::new(), + vouch_until: None, }; if let Err(e) = broker.tick(&tick).await { tracing::warn!(target: "weft_engine::test_rig", error = %format!("{e:#}"), "the test's lease could not be renewed; the next second tries again"); diff --git a/crates/weft-engine/src/worker.rs b/crates/weft-engine/src/worker.rs index bff6d4f3..55c4b851 100644 --- a/crates/weft-engine/src/worker.rs +++ b/crates/weft-engine/src/worker.rs @@ -143,14 +143,19 @@ struct Cancels { /// Told whenever a run leaves, for a claim waiting on its run's last /// drive here to end ([`Self::claim_once_left`]). left: tokio::sync::Notify, + /// Told whenever a run comes, once it is registered: the door's tick + /// (`crate::door::Door::wake`), which must keep the worker's lease for + /// it however the door slept. + came: Box, } impl Cancels { - fn new() -> Arc { + fn new(came: Box) -> Arc { Arc::new(Self { shards: std::array::from_fn(|_| std::sync::Mutex::new(HashMap::new())), changed: tokio::sync::Notify::new(), left: tokio::sync::Notify::new(), + came, }) } @@ -175,6 +180,7 @@ impl Cancels { shard.insert(execution_id, flag.clone()); } self.changed.notify_one(); + (self.came)(); Some(Registered { cancels: self.clone(), execution_id, flag }) } @@ -211,6 +217,12 @@ impl Cancels { } } + /// Every run driven here with its flag, held still: no run comes or + /// goes until the guards go. + fn held_still(&self) -> Vec>>> { + self.shards.iter().map(|shard| shard.lock().expect("cancel registry")).collect() + } + /// How many runs this worker drives. fn count(&self) -> usize { self.shards.iter().map(|shard| shard.lock().expect("cancel registry").len()).sum() @@ -251,6 +263,10 @@ struct Worker { replica: String, tenant_id: String, cancels: Arc, + /// The runs it holds while they only wait outside (`crate::run_holds`). + holds: Arc, + /// Told whenever one of [`Self::holds`] goes quiet or moves again. + quiet_changed: Arc, /// What this worker keeps of its programs and of the runs its triggers /// start (`crate::plan`). plans: Arc, @@ -360,6 +376,41 @@ struct RunStart { } impl Worker { + /// What this worker has under way, its lease holding `lease_secs` + /// without a tick: what its door ticks or sleeps on. + fn under_way(&self, lease_secs: i64) -> crate::door::UnderWay { + crate::door::UnderWay::of(self.clients.writer.settled(), self.cancels.count(), self.holds.quiet(), lease_secs) + } + + /// Stop for the cut of its line (`cut`), in one step with the look + /// that decides it, if at this instant it drives runs, every one of + /// them only waits outside and every record is written (the state its + /// door sleeps in). With no run able to come, go, go quiet or move + /// again meanwhile, it takes no new run from here on, which also hands + /// back each of those runs that can pause, and ends the others, which + /// hold their waits in this worker, at the cut ([`Limit::LineCut`]). + /// Otherwise nothing is done, and the answer says what stood in the + /// way. + fn stopped_for_cut(&self, cut: weft_broker_client::line::Cut) -> CutLook { + let quiet = self.holds.held_still(); + let driven = self.cancels.held_still(); + let runs = || driven.iter().flat_map(|shard| shard.iter()); + if runs().next().is_none() || runs().any(|(execution_id, _)| !quiet.contains_key(execution_id)) { + return CutLook::Moving; + } + if !self.clients.writer.settled() { + return CutLook::RecordsPending; + } + self.stopping.cancel(); + let left = (cut.opened + cut.max_age).saturating_duration_since(tokio::time::Instant::now()); + for (execution_id, flag) in runs() { + if quiet.get(execution_id).is_some_and(|how| !how.can_pause) { + Limit::LineCut { cut: cut.max_age, left }.end(flag, *execution_id); + } + } + CutLook::Stopped + } + /// Claim queued run `execution_id` and carry it on to its end, on a /// task of its own so whoever handed it here can go away without /// stopping it (a run stops itself before the platform's cap, see @@ -374,10 +425,14 @@ impl Worker { () = self.stopping.cancelled() => return Ok(None), registered = self.cancels.claim_once_left(execution_id) => registered, }; + // Stopped as it registered (a stop decides with every registration + // held still, `Worker::stopped_for_cut`): it goes no further. + if self.stopping.is_cancelled() { + return Ok(None); + } // The claim makes the run this worker's on record, which its lease - // keeps: the registration counts it where the door's tick looks, so - // the tick goes out now, and the claim does not wait for it. - self.door.wake(); + // keeps: the registration woke the door's tick, and the claim does + // not wait for it. let Some(claimed) = self.clients.runs.claim(execution_id).await.context("claim the run")? else { return Ok(Some(RunAnswer::NothingToRun)); }; @@ -465,7 +520,16 @@ impl Worker { let run = run_one_execution( self.catalog.clone(), &self.clients, - crate::execution_driver::RunDrive { execution_id, journal, program, starts_from, cancellation: cancellation.clone(), exchange, hand_back: Some(&hand_back) }, + crate::execution_driver::RunDrive { + execution_id, + journal, + program, + starts_from, + cancellation: cancellation.clone(), + exchange, + hand_back: Some(&hand_back), + hold: Some(self.holds.of(execution_id)), + }, &self.replica, &self.tenant_id, ); @@ -579,10 +643,7 @@ impl Worker { outcome = &mut run => return outcome, () = &mut capped, if !cap_fired => { cap_fired = true; - if !cancellation.is_cancelled() { - let cap = cap.expect("armed only with a cap"); - cancellation.cancel_because(weft_core::exec::CancelCause::Runtime { detail: run_cap_reached(execution_id, cap, live) }); - } + Limit::RunCap { cap: cap.expect("armed only with a cap"), live }.end(cancellation, execution_id); } () = &mut caller_left, if !left => { left = true; @@ -948,7 +1009,7 @@ fn spawn_cancel_watch(worker: Worker) { }; match next { Ok(weft_task_store::pg_signal::Heard::Signal { channel, payload }) if channel == weft_task_store::runs::CANCEL_CHANNEL => { - let Some((project, execution_id)) = weft_task_store::runs::parse_cancel_payload(&payload) else { + let Some((project, execution_id)) = weft_task_store::runs::parse_run_payload(&payload) else { tracing::error!(target: "weft_engine::worker", %payload, "a cancel announcement that names no run"); continue; }; @@ -956,7 +1017,10 @@ fn spawn_cancel_watch(worker: Worker) { fire_cancels(&worker).await; } } - Ok(weft_task_store::pg_signal::Heard::Recheck | weft_task_store::pg_signal::Heard::Resumed) if driving => fire_cancels(&worker).await, + // The line may have missed some (it was lost, or its copy + // missed news). The broker hearing again after a quiet spell + // (`Resumed`) says nothing missed, and reads nothing. + Ok(weft_task_store::pg_signal::Heard::Recheck) if driving => fire_cancels(&worker).await, Ok(_) => {} Err(e) => { tracing::error!(target: "weft_engine::worker", error = %format!("{e:#}"), "the cancel watch stopped hearing the line"); @@ -1000,33 +1064,95 @@ async fn fire_cancels_once(worker: &Worker) -> Result<()> { Ok(()) } -/// Why a run was stopped at the platform's cap, and what to do. -fn run_cap_reached(execution_id: ExecutionId, cap: std::time::Duration, live: bool) -> String { - let minutes = cap.as_secs() / 60; - if live { - format!( - "execution {execution_id} stopped a minute before the {minutes} minute limit this platform puts on one \ - connection. A conversation meant to last longer keeps its state outside the run and \ - has its client reconnect, each connection its own run", - ) - } else { - format!( - "execution {execution_id} stopped a minute before the {minutes} minute limit this platform puts on one \ - stretch of a run. A run that pauses whole (every branch waiting on a timer or a form) picks \ - back up with a fresh {minutes} minutes, so split long work with a pause", - ) +/// A platform limit a run reaches, which ends it before the limit with a +/// cause that says how early and what to do ([`Limit::end`]): the one way +/// a limit ends a run, whichever limit it is. +#[derive(Debug, Clone, Copy)] +enum Limit { + /// The cap on one stretch of a run (`WorkerConfig::run_cap`), reached + /// [`RUN_CAP_MARGIN`] early; `live` when a caller is on the line, whose + /// connection the cap bounds. + RunCap { cap: std::time::Duration, live: bool }, + /// The cut of the worker's line to the broker after `cut`, reached by + /// a run that waits without being able to pause, `left` before it + /// (see [`stop_before_line_cut`]). + LineCut { cut: std::time::Duration, left: std::time::Duration }, +} + +impl Limit { + /// End run `execution_id` through its cancel `flag`, unless something + /// already did. + fn end(self, flag: &CancellationFlag, execution_id: ExecutionId) { + if !flag.is_cancelled() { + tracing::info!(target: "weft_engine::worker", %execution_id, limit = ?self, "the run reached a platform limit"); + flag.cancel_because(self.cause(execution_id)); + } + } + + /// What run `execution_id` is ended with: the limit, how early, and + /// the way out that fits it. + fn cause(self, execution_id: ExecutionId) -> weft_core::exec::CancelCause { + let detail = match self { + Limit::RunCap { cap, live: true } => format!( + "execution {execution_id} stopped {} before the {} minute limit this platform puts on one \ + connection. A conversation meant to last longer keeps its state outside the run and \ + has its client reconnect, each connection its own run", + seconds(RUN_CAP_MARGIN), + cap.as_secs() / 60, + ), + Limit::RunCap { cap, live: false } => format!( + "execution {execution_id} stopped {} before the {minutes} minute limit this platform puts on one \ + stretch of a run. A run that pauses whole (every branch waiting on a timer or a form) picks \ + back up with a fresh {minutes} minutes, so split long work with a pause", + seconds(RUN_CAP_MARGIN), + minutes = cap.as_secs() / 60, + ), + Limit::LineCut { cut, left } => format!( + "execution {execution_id} stopped {} before the {} minute limit this platform puts on the \ + connection between a worker and weft, which a run that waits without being able to pause keeps open. \ + A run that can pause (recorded, not tied to its caller, no bus open) lets go of its worker while it \ + waits, and may wait as long as it needs", + seconds(left), + cut.as_secs() / 60, + ), + }; + weft_core::exec::CancelCause::Runtime { detail } + } +} + +/// `d` in whole seconds, said the way a person reads it. +fn seconds(d: std::time::Duration) -> String { + match d.as_secs() { + 1 => "1 second".into(), + secs => format!("{secs} seconds"), } } fn new_worker(catalog: Arc, clients: EngineClients, config: &WorkerConfig, door: Arc) -> Worker { + // A run going quiet or moving again wakes the door's tick, and the + // watch on the line's cut (`stop_before_line_cut`). + let quiet_changed = Arc::new(tokio::sync::Notify::new()); + let holds = crate::run_holds::Holds::new({ + let (door, quiet_changed) = (door.clone(), quiet_changed.clone()); + Box::new(move || { + door.wake(); + quiet_changed.notify_one(); + }) + }); + let cancels = Cancels::new({ + let door = door.clone(); + Box::new(move || door.wake()) + }); Worker { + holds, + quiet_changed, door, project_id: config.project_id, catalog, clients, replica: config.replica.clone(), tenant_id: config.tenant_id.clone(), - cancels: Cancels::new(), + cancels, plans: Arc::new(crate::plan::Plans::new(config.project_id)), background: weft_core::in_flight::InFlight::new("worker execution"), stopping: tokio_util::sync::CancellationToken::new(), @@ -1118,9 +1244,12 @@ pub async fn serve(catalog: Arc, clients: EngineClients, config // the broker until it is back (`weft_task_store::held_copy`). let worker = new_worker(catalog, clients, &config, door.clone()); // The door ticks while this worker owns anything on record: a run it - // drives, or a record still on its way. - let (cancels, writer) = (worker.cancels.clone(), worker.clients.writer.clone()); - door.start_ticks(Box::new(move || cancels.count() > 0 || !writer.settled())).await?; + // drives that does more than wait outside, or a record still on its + // way. With only waiting runs, it vouches for the worker until the + // earliest of their waits gives up, and sleeps. + let lease = weft_core::time_scale::scaled_secs(weft_task_store::worker_door::WORKER_LEASE_SECS); + let under_way = worker.clone(); + door.start_ticks(Box::new(move || under_way.under_way(lease))).await?; spawn_cancel_watch(worker.clone()); // What the runs share goes once nothing used it for the idle window. worker.clients.shared.sweep_every(std::time::Duration::from_secs(30)); @@ -1138,9 +1267,11 @@ pub async fn serve(catalog: Arc, clients: EngineClients, config starter: Arc::new(LiveStarter { worker: worker.clone() }), })); let app = app.layer(axum::middleware::from_fn_with_state(crate::busy::MemoryGuard::sampling(), crate::busy::refuse_when_busy)); - // Told to stop, the worker hands its recorded runs back and takes no - // new call, while the calls it holds go on to their end. + // Told to stop, or nearing the cut of its line while its runs only + // wait, the worker hands its recorded runs back and takes no new call, + // while the calls it holds go on to their end. stop_on_signal(&worker); + stop_before_line_cut(&worker); let stopping = worker.stopping.clone(); crate::caller_conn::serve( app.layer(axum::middleware::map_response(mark_worker_answer)), @@ -1186,16 +1317,100 @@ async fn wind_down(worker: &Worker) { worker.clients.writer.written().await; } -/// Fire [`Worker::stopping`] once the platform says to stop, and -/// [`Worker::stopping_overdue`] [`HAND_BACK_GRACE`] later. +/// What a look at stopping before the line's cut found +/// ([`Worker::stopped_for_cut`]). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum CutLook { + /// Every run only waited, every record was written: it stopped. + Stopped, + /// No run, or one that does more than wait. + Moving, + /// Every run only waits, but records are still on their way. + RecordsPending, +} + +/// Once [`Worker::stopping`] fired, which takes no new run and hands back +/// every run that can be suspended once its running steps end: fire +/// [`Worker::stopping_overdue`] [`HAND_BACK_GRACE`] later, which stops +/// the steps still running. +async fn overdue_after_grace(worker: &Worker) { + tokio::time::sleep(HAND_BACK_GRACE).await; + worker.stopping_overdue.cancel(); +} + +/// Stop the worker once the platform says to. A run that cannot be +/// suspended carries on until the platform ends the process. fn stop_on_signal(worker: &Worker) { - let (stopping, overdue) = (worker.stopping.clone(), worker.stopping_overdue.clone()); + let worker = worker.clone(); tokio::spawn(async move { shutdown_signal().await; tracing::info!(target: "weft_engine::worker", "told to stop: every run that can be suspended is handed back once its running steps end; the others carry on until the process goes"); - stopping.cancel(); - tokio::time::sleep(HAND_BACK_GRACE).await; - overdue.cancel(); + worker.stopping.cancel(); + overdue_after_grace(&worker).await; + }); +} + +/// Stop the worker once its line is within [`RUN_CAP_MARGIN`] of the cut +/// the broker said it makes, if every run it drives only waits (the state +/// its door sleeps in): what those runs wait on is pushed down that line, +/// and a worker whose runs only wait never opens another, since the reads +/// after a new line would wake the database. Decided as the margin is +/// reached, then again whenever that state may have been entered (a run +/// going quiet, a run leaving, the records written) until the line goes, +/// each time in one step with the stop ([`Worker::stopped_for_cut`]). A +/// busy worker lets its line be cut and opens another: its work keeps the +/// database awake anyway. A line that closed for want of anything to do +/// and opened again is a new line, counted afresh. Nothing ends the +/// process from outside then: it winds down, and the platform starts a +/// fresh worker on the next request. +fn stop_before_line_cut(worker: &Worker) { + let mut cut = worker.clients.line.cut_at(); + let worker = worker.clone(); + tokio::spawn(async move { + 'lines: loop { + let Some(told) = *cut.borrow_and_update() else { + if cut.changed().await.is_err() { + return; + } + continue; + }; + let within_margin = told.opened + told.max_age.saturating_sub(RUN_CAP_MARGIN); + tokio::select! { + changed = cut.changed() => { + if changed.is_err() { + return; + } + continue 'lines; + } + () = tokio::time::sleep_until(within_margin) => {} + } + loop { + // Armed before the look, so a change between the look and + // the wait still wakes it. + let left = worker.cancels.left.notified(); + tokio::pin!(left); + left.as_mut().enable(); + let looked = worker.stopped_for_cut(told); + if looked == CutLook::Stopped { + tracing::info!(target: "weft_engine::worker", line_cut_secs = told.max_age.as_secs(), "every run only waits and weft cuts this worker's line soon, on which they hear their answers: the ones that can pause are handed back, the others end, and the next request starts a fresh worker"); + overdue_after_grace(&worker).await; + return; + } + // `written` answers at once if the records went out since + // the look. + tokio::select! { + changed = cut.changed() => { + if changed.is_err() { + return; + } + continue 'lines; + } + () = worker.quiet_changed.notified() => {} + () = &mut left => {} + () = worker.clients.writer.written(), if looked == CutLook::RecordsPending => {} + } + } + } }); } @@ -1540,16 +1755,31 @@ mod tests { use super::*; use crate::execution_driver::engine_test_rig::{clients, MemJournal}; - /// A run stopped at the platform's cap is told the way out that fits - /// it: a pause for one nobody is talking to, and for one with a caller - /// a reconnect over kept state. + /// Every platform limit ends a run through the same cancel, told the + /// way out that fits it: a pause for a run nobody is talking to, a + /// reconnect over kept state for one with a caller, a run that can + /// pause for one that waited on the worker's line; each says how early + /// it stopped: the margin for the cap, what was left for the line. #[test] - fn the_cap_names_the_way_out_that_fits_the_run() { - let cap = std::time::Duration::from_secs(3600); - let live = run_cap_reached(ExecutionId::nil(), cap, true); - assert!(live.contains("60 minute limit") && live.contains("reconnect"), "{live}"); - let plain = run_cap_reached(ExecutionId::nil(), cap, false); - assert!(plain.contains("60 minute limit") && plain.contains("pause"), "{plain}"); + fn a_limit_ends_a_run_naming_the_way_out_that_fits_it() { + let hour = std::time::Duration::from_secs(3600); + let margin = RUN_CAP_MARGIN.as_secs(); + for (limit, early, way_out) in [ + (Limit::RunCap { cap: hour, live: true }, margin, "reconnect"), + (Limit::RunCap { cap: hour, live: false }, margin, "split long work with a pause"), + (Limit::LineCut { cut: hour, left: std::time::Duration::from_secs(37) }, 37, "A run that can pause"), + ] { + let early = format!("{early} seconds before the 60 minute limit"); + let flag = CancellationFlag::new_arc(); + limit.end(&flag, ExecutionId::nil()); + let detail = match flag.cause() { + Some(weft_core::exec::CancelCause::Runtime { detail }) => detail, + other => panic!("{limit:?} ended the run with {other:?}"), + }; + assert!(detail.contains(&early) && detail.contains(way_out), "{detail}"); + Limit::RunCap { cap: hour, live: false }.end(&flag, ExecutionId::nil()); + assert!(matches!(flag.cause(), Some(weft_core::exec::CancelCause::Runtime { detail: again }) if again == detail), "the first ending stands"); + } } #[test] @@ -1586,7 +1816,10 @@ mod tests { async fn let_go(&self, _execution_id: ExecutionId, _why: weft_broker_client::protocol::LetGo) -> anyhow::Result<()> { unreachable!("nothing is driven here") } - async fn answers(&self, _execution_id: ExecutionId, _taken: &[String], _wait: std::time::Duration) -> anyhow::Result> { + async fn answers(&self, _execution_id: ExecutionId, _taken: &[String]) -> anyhow::Result> { + unreachable!("nothing is driven here") + } + fn told(&self) -> weft_task_store::pg_signal::Subscription { unreachable!("nothing is driven here") } async fn cancels(&self) -> anyhow::Result> { @@ -1744,7 +1977,7 @@ mod tests { fn cancel_heard(execution_id: ExecutionId) -> weft_task_store::pg_signal::Heard { weft_task_store::pg_signal::Heard::Signal { channel: weft_task_store::runs::CANCEL_CHANNEL, - payload: weft_task_store::runs::cancel_payload(uuid::Uuid::from_u128(1), execution_id).into(), + payload: weft_task_store::runs::run_payload(uuid::Uuid::from_u128(1), execution_id).into(), } } @@ -1769,9 +2002,11 @@ mod tests { } // A cancel the line announces for a run driven here reaches its flag; - // one for a run driven elsewhere asks nothing; the line is held open - // exactly while something is driven; and a line that opens again asks - // for the cancels of every run driven. + // one for a run driven elsewhere asks nothing; the broker hearing again + // after a quiet spell, and a line lost, ask nothing; the line is held + // open exactly while something is driven; and a line that opens again + // (or may have missed some) asks once for the cancels of every run + // driven. weft_core::stress_test! { name: a_heard_cancel_fires_the_flag_of_the_run_driven, runs: 20, @@ -1791,6 +2026,8 @@ mod tests { line.pushed.send(cancel_heard(driven)).unwrap(); cancelled(&flag).await; assert_eq!(runs.asked.load(std::sync::atomic::Ordering::SeqCst), 1, "a cancel of a run driven elsewhere asks nothing"); + line.pushed.send(weft_task_store::pg_signal::Heard::Resumed).unwrap(); + line.pushed.send(weft_task_store::pg_signal::Heard::Lost).unwrap(); let missed = weft_core::new_execution_id(); let _missed = worker.cancels.claim(missed).expect("a fresh run"); @@ -1798,6 +2035,7 @@ mod tests { runs.cancels.lock().unwrap().push(missed); line.pushed.send(weft_task_store::pg_signal::Heard::Recheck).unwrap(); cancelled(&flag).await; + assert_eq!(runs.asked.load(std::sync::atomic::Ordering::SeqCst), 2, "only the recheck asked again"); drop(registered); drop(_missed); @@ -1805,11 +2043,253 @@ mod tests { } } + /// The race a sleeping door must never lose: the door sleeps because + /// its only run waits outside, having vouched for the worker; a caller + /// is admitted and its run registered while it sleeps. The registration + /// itself wakes the tick, which then keeps the lease for the new run. + #[tokio::test(start_paused = true)] + async fn a_run_registered_while_the_door_sleeps_on_waiting_runs_wakes_it() { + let (worker, broker) = worker_firing(|_, _| {}); + let (cancels, holds) = (worker.cancels.clone(), worker.holds.clone()); + worker.door.start_ticks(Box::new(move || crate::door::UnderWay::of(true, cancels.count(), holds.quiet(), 15))).await.unwrap(); + let waiting = worker.cancels.claim(uuid::Uuid::from_u128(1)).expect("a fresh run"); + let quiet = worker.holds.of(uuid::Uuid::from_u128(1)); + quiet.quiet(Some(crate::run_holds::Quiet { until: 5_000, can_pause: false })); + tokio::time::sleep(std::time::Duration::from_secs(30)).await; + let asleep = broker.ticks.lock().unwrap().clone(); + assert_eq!(asleep.last().unwrap().vouch_until, Some(5_015), "vouched until the wait ends, and a lease past it"); + + let _caller = worker.cancels.claim(uuid::Uuid::from_u128(2)).expect("a fresh run"); + tokio::time::sleep(std::time::Duration::from_millis(500)).await; + let woken = broker.ticks.lock().unwrap().clone(); + assert_eq!(woken.len(), asleep.len() + 1, "the new run woke the tick at once"); + assert_eq!(woken.last().unwrap().vouch_until, None, "a run that moves: one lease length at a time again"); + drop((waiting, quiet)); + } + + const HOUR: std::time::Duration = std::time::Duration::from_secs(3600); + + /// A worker watching its line's cut, the line opened now and cut after + /// an hour. + fn watching_its_line() -> (Worker, Arc) { + let line = crate::context::TestLine::new(); + let worker = worker_over(Arc::new(Runs::default()), line.clone()); + stop_before_line_cut(&worker); + line.cut.send_replace(Some(weft_broker_client::line::Cut { opened: tokio::time::Instant::now(), max_age: HOUR })); + (worker, line) + } + + /// A run holding its waits in the worker until its hold, far off. + fn holding() -> Option { + Some(crate::run_holds::Quiet { until: i64::MAX, can_pause: false }) + } + + /// Whether `flag` was ended at the line's cut, `left` before it. + fn ended_at_the_cut(flag: &CancellationFlag, left: &str) -> bool { + matches!( + flag.cause(), + Some(weft_core::exec::CancelCause::Runtime { detail }) + if detail.contains("between a worker and weft") && detail.contains(&format!("stopped {left} before")) + ) + } + + /// Let the watch on the line's cut act on what just changed, no time + /// passing. + async fn settled() { + for _ in 0..50 { + tokio::task::yield_now().await; + } + } + + /// A worker whose only run waits stops as its line comes within the + /// margin of its cut, not before, and that run ends naming the time + /// that was left; the hand-back is overdue after the grace. A line + /// closed and opened again counts afresh. + #[tokio::test(start_paused = true)] + async fn a_worker_whose_runs_only_wait_stops_before_its_line_is_cut() { + let (worker, line) = watching_its_line(); + let waiting = worker.cancels.claim(uuid::Uuid::from_u128(1)).expect("a fresh run"); + let quiet = worker.holds.of(uuid::Uuid::from_u128(1)); + quiet.quiet(holding()); + tokio::time::sleep(HOUR / 2).await; + // Closed while idle, opened again: a new line, counted from now. + line.cut.send_replace(None); + line.cut.send_replace(Some(weft_broker_client::line::Cut { opened: tokio::time::Instant::now(), max_age: HOUR })); + tokio::time::sleep(HOUR - RUN_CAP_MARGIN - std::time::Duration::from_secs(1)).await; + assert!(!worker.stopping.is_cancelled(), "the new line still has time"); + tokio::time::sleep(std::time::Duration::from_secs(2)).await; + assert!(worker.stopping.is_cancelled(), "stopping within the margin of the cut"); + let left = format!("{} seconds", RUN_CAP_MARGIN.as_secs()); + assert!(ended_at_the_cut(&waiting.flag, &left), "the waiting run ends naming the time left: {:?}", waiting.flag.cause()); + tokio::time::sleep(HAND_BACK_GRACE).await; + assert!(worker.stopping_overdue.is_cancelled(), "steps still running are stopped after the grace"); + drop((waiting, quiet)); + } + + /// A busy worker lets its line be cut: it does not stop within the + /// margin while a run moves, and stops the moment that run comes to + /// only wait, even in the last second before the cut. + #[tokio::test(start_paused = true)] + async fn a_busy_worker_stops_the_moment_its_runs_only_wait() { + let (worker, _line) = watching_its_line(); + let moving = worker.cancels.claim(uuid::Uuid::from_u128(1)).expect("a fresh run"); + tokio::time::sleep(HOUR - std::time::Duration::from_secs(1)).await; + assert!(!worker.stopping.is_cancelled(), "a busy worker carries on into the margin"); + assert!(!moving.flag.is_cancelled()); + let quiet = worker.holds.of(uuid::Uuid::from_u128(1)); + quiet.quiet(holding()); + settled().await; + assert!(worker.stopping.is_cancelled(), "only waiting in the last second: it stops at once"); + assert!(ended_at_the_cut(&moving.flag, "1 second"), "{:?}", moving.flag.cause()); + drop((moving, quiet)); + } + + /// The look and the stop are one step: a run that moves again (its + /// answer came) as the margin arrives keeps the worker running and + /// watching, as does another run that moves; once that one leaves and + /// the first waits again, the worker stops then. + #[tokio::test(start_paused = true)] + async fn a_run_that_moves_as_the_margin_arrives_keeps_the_worker_watching() { + let (worker, _line) = watching_its_line(); + let waiting = worker.cancels.claim(uuid::Uuid::from_u128(1)).expect("a fresh run"); + let quiet = worker.holds.of(uuid::Uuid::from_u128(1)); + quiet.quiet(holding()); + let other = worker.cancels.claim(uuid::Uuid::from_u128(2)).expect("a fresh run"); + tokio::time::sleep(HOUR - RUN_CAP_MARGIN - std::time::Duration::from_millis(1)).await; + // Answered as the margin arrives. + quiet.quiet(None); + drop(other); + tokio::time::sleep(std::time::Duration::from_secs(10)).await; + assert!(!worker.stopping.is_cancelled(), "a run moves: no stop"); + assert!(!waiting.flag.is_cancelled()); + quiet.quiet(holding()); + settled().await; + assert!(worker.stopping.is_cancelled(), "every run waits again within the margin: it stops then"); + assert!(ended_at_the_cut(&waiting.flag, "50 seconds"), "{:?}", waiting.flag.cause()); + drop((waiting, quiet)); + } + + /// Another run that moves keeps a worker whose first run waits from + /// stopping; once it leaves, the worker stops at once. + #[tokio::test(start_paused = true)] + async fn a_worker_stops_once_its_last_moving_run_leaves() { + let (worker, _line) = watching_its_line(); + let waiting = worker.cancels.claim(uuid::Uuid::from_u128(1)).expect("a fresh run"); + let quiet = worker.holds.of(uuid::Uuid::from_u128(1)); + quiet.quiet(holding()); + let moving = worker.cancels.claim(uuid::Uuid::from_u128(2)).expect("a fresh run"); + tokio::time::sleep(HOUR - std::time::Duration::from_secs(30)).await; + assert!(!worker.stopping.is_cancelled(), "a run moves: no stop"); + drop(moving); + settled().await; + assert!(worker.stopping.is_cancelled(), "the last moving run left: it stops then"); + assert!(ended_at_the_cut(&waiting.flag, "30 seconds"), "{:?}", waiting.flag.cause()); + drop((waiting, quiet)); + } + + /// In one stop, a waiting run that can pause is left to the stopping + /// path, which hands it back, and one that holds its waits in the + /// worker ends at the cut. + #[tokio::test(start_paused = true)] + async fn a_stop_hands_back_what_can_pause_and_ends_what_cannot() { + let (worker, _line) = watching_its_line(); + let pausable = worker.cancels.claim(uuid::Uuid::from_u128(1)).expect("a fresh run"); + let pausable_quiet = worker.holds.of(uuid::Uuid::from_u128(1)); + pausable_quiet.quiet(Some(crate::run_holds::Quiet { until: i64::MAX, can_pause: true })); + let held = worker.cancels.claim(uuid::Uuid::from_u128(2)).expect("a fresh run"); + let held_quiet = worker.holds.of(uuid::Uuid::from_u128(2)); + held_quiet.quiet(holding()); + let asked = worker.stopping.child_token(); + tokio::time::sleep(HOUR - RUN_CAP_MARGIN + std::time::Duration::from_secs(1)).await; + assert!(asked.is_cancelled(), "asked to leave: the run that can pause is handed back"); + assert!(!pausable.flag.is_cancelled(), "a run that can pause is never ended at the cut"); + let left = format!("{} seconds", RUN_CAP_MARGIN.as_secs()); + assert!(ended_at_the_cut(&held.flag, &left), "{:?}", held.flag.cause()); + drop((pausable, pausable_quiet, held, held_quiet)); + } + + /// A record that writes only when let through. + struct GatedRecord { + record: Arc, + through: tokio::sync::Semaphore, + } + + #[async_trait::async_trait] + impl weft_journal::RecordClient for GatedRecord { + async fn record_batch(&self, batch: Vec) -> Result { + self.through.acquire().await.expect("the gate stays").forget(); + self.record.record_batch(batch).await + } + async fn record_of(&self, execution_id: ExecutionId) -> anyhow::Result { + self.record.record_of(execution_id).await + } + async fn give_up(&self, execution_id: ExecutionId, why: String) -> Result<(), weft_journal::BatchError> { + self.record.give_up(execution_id, why).await + } + } + + /// Every run only waits but its records are still on their way at the + /// margin: no stop yet; once they are written, the worker stops then, + /// however soon after the look they landed. + #[tokio::test(start_paused = true)] + async fn a_worker_stops_once_the_records_of_its_waiting_runs_are_written() { + let gate = Arc::new(GatedRecord { record: Arc::new(MemJournal::default()), through: tokio::sync::Semaphore::new(0) }); + let mut clients = clients(Arc::new(MemJournal::default())); + clients.writer = crate::journal_writer::WorkerJournal::start(gate.clone(), Default::default(), &crate::context::io_runtime().expect("the io runtime starts")); + let line = crate::context::TestLine::new(); + clients.line = line.clone(); + let worker = worker_with(clients); + stop_before_line_cut(&worker); + line.cut.send_replace(Some(weft_broker_client::line::Cut { opened: tokio::time::Instant::now(), max_age: HOUR })); + + let execution_id = uuid::Uuid::from_u128(1); + let waiting = worker.cancels.claim(execution_id).expect("a fresh run"); + let quiet = worker.holds.of(execution_id); + quiet.quiet(holding()); + let journal = worker.clients.writer.run(crate::journal_writer::RunSpec { + execution_id, + settings: Default::default(), + keep_for: weft_core::run_settings::KeepFor::WEFT_DEFAULT, + epoch: 1, + next_seq: 0, + redaction: Default::default(), + }); + journal.hand(&ExecEvent::ExecutionFailed { execution_id, error: "on its way".into(), at_unix: 0 }, None).unwrap(); + tokio::time::sleep(HOUR - RUN_CAP_MARGIN + std::time::Duration::from_secs(1)).await; + assert!(!worker.clients.writer.settled(), "the record is held at the gate"); + assert!(!worker.stopping.is_cancelled(), "records on their way: no stop yet"); + + gate.through.add_permits(1_000); + for _ in 0..500 { + if worker.stopping.is_cancelled() { + break; + } + // The writer runs on its own runtime, in real time. + tokio::task::spawn_blocking(|| std::thread::sleep(std::time::Duration::from_millis(10))).await.unwrap(); + } + assert!(worker.stopping.is_cancelled(), "the records went out: it stops then"); + assert!(waiting.flag.is_cancelled()); + drop((waiting, quiet, journal)); + } + + /// A line the broker keeps for good never stops the worker. + #[tokio::test(start_paused = true)] + async fn a_line_kept_for_good_never_stops_the_worker() { + let (worker, line) = watching_its_line(); + line.cut.send_replace(None); + let waiting = worker.cancels.claim(uuid::Uuid::from_u128(1)).expect("a fresh run"); + let quiet = worker.holds.of(uuid::Uuid::from_u128(1)); + quiet.quiet(holding()); + tokio::time::sleep(HOUR * 24).await; + assert!(!worker.stopping.is_cancelled()); + drop((waiting, quiet)); + } + /// A delivery of a run whose drive here is still leaving claims it /// once that drive left. #[tokio::test] async fn a_claim_waits_for_the_runs_last_drive_here_to_leave() { - let cancels = Cancels::new(); + let cancels = Cancels::new(Box::new(|| {})); let execution_id = weft_core::new_execution_id(); let leaving = cancels.claim(execution_id).expect("a fresh run"); let mut claiming = tokio::spawn({ diff --git a/crates/weft-infra-supervisor/src/lifecycle.rs b/crates/weft-infra-supervisor/src/lifecycle.rs index 681cbc9b..e4c3499f 100644 --- a/crates/weft-infra-supervisor/src/lifecycle.rs +++ b/crates/weft-infra-supervisor/src/lifecycle.rs @@ -227,6 +227,10 @@ pub async fn drain( work(state, changes, Until::Idle).await } +/// How long one claim sleeps on the broker while commands run, before +/// [`work`] asks again. +const CLAIM_HOLD: Duration = Duration::from_secs(25); + /// How long [`work`] goes on. #[derive(Clone, Copy, PartialEq, Eq)] enum Until { @@ -251,7 +255,7 @@ async fn work( // them. A pass with nothing running asks once and ends on nothing. let hold = match (until, busy.is_empty()) { (Until::Idle, true) => Duration::ZERO, - _ => weft_broker_client::protocol::MAX_HOLD, + _ => CLAIM_HOLD, }; let busy_commands: Vec = busy.values().map(|(_, command, _)| *command).collect(); // Ownership first: a claim can only hand out a project this supervisor diff --git a/crates/weft-infra-supervisor/tests/lifecycle_loop.rs b/crates/weft-infra-supervisor/tests/lifecycle_loop.rs index c9cf9841..c0e6b430 100644 --- a/crates/weft-infra-supervisor/tests/lifecycle_loop.rs +++ b/crates/weft-infra-supervisor/tests/lifecycle_loop.rs @@ -724,7 +724,7 @@ async fn command_left_by_displaced_owner_completes_once_new_owner_runs_it() { // A `wait`-policy stop on a project whose running count is gated stays // mid-drain until the test opens the gate, which is how a command is // kept running while the loop's other behavior is observed. The claim -// holds up to `MAX_HOLD` (25s) when nothing is waiting, so every wait +// holds up to 25s (`CLAIM_HOLD`) when nothing is waiting, so every wait // below is bounded well under it: a loop that failed to ask again would // hold past the bound and fail the test. diff --git a/crates/weft-platform-gcp/src/api.rs b/crates/weft-platform-gcp/src/api.rs index b596326c..2959d81d 100644 --- a/crates/weft-platform-gcp/src/api.rs +++ b/crates/weft-platform-gcp/src/api.rs @@ -107,6 +107,10 @@ impl Google { self.send(self.http.post(url).json(body)).await.with_context(|| format!("POST {url}")) } + pub async fn put(&self, url: &str, body: &Value) -> anyhow::Result { + self.send(self.http.put(url).json(body)).await.with_context(|| format!("PUT {url}")) + } + pub async fn patch(&self, url: &str, body: &Value) -> anyhow::Result { self.send(self.http.patch(url).json(body)).await.with_context(|| format!("PATCH {url}")) } diff --git a/crates/weft-platform-gcp/src/lib.rs b/crates/weft-platform-gcp/src/lib.rs index bce36985..949c67be 100644 --- a/crates/weft-platform-gcp/src/lib.rs +++ b/crates/weft-platform-gcp/src/lib.rs @@ -9,7 +9,8 @@ //! verified against Google's keys (`identity`), the object store on //! Cloud Storage (`storage`), frontends on Cloud Run (`frontends`), the //! door in front of the install's domains on a load balancer -//! (`domains`), the holders on a worker pool (`holders`). `api` is the one REST +//! (`domains`), the holders on a worker pool (`holders`), what happened +//! to a run told across copies on Pub/Sub (`news`). `api` is the one REST //! client they share, `names` what they call things, `accounts` the //! project's own service account. @@ -36,6 +37,8 @@ pub mod infra_host; #[cfg(feature = "control")] pub mod names; #[cfg(feature = "control")] +pub mod news; +#[cfg(feature = "control")] pub mod runner; #[cfg(feature = "control")] pub mod storage; @@ -44,6 +47,6 @@ pub use metadata::{identity_from_env, MetadataTokens}; #[cfg(feature = "control")] pub use { alarm::CloudTasksAlarm, api::Google, domains::LoadBalancerDomains, frontends::CloudRunFrontends, holders::WorkerPoolHolders, - identity::GoogleIdentity, images::CloudBuildImages, infra_host::ComputeInfraHost, runner::CloudRunRunner, + identity::GoogleIdentity, images::CloudBuildImages, infra_host::ComputeInfraHost, news::PubSubNews, runner::{CloudRunRunner, REQUEST_CAP}, storage::GcsObjectStore, }; diff --git a/crates/weft-platform-gcp/src/news.rs b/crates/weft-platform-gcp/src/news.rs new file mode 100644 index 00000000..9ab7cf65 --- /dev/null +++ b/crates/weft-platform-gcp/src/news.rs @@ -0,0 +1,466 @@ +//! News on Google Cloud: one Pub/Sub topic per install, and a +//! subscription of its own for every process that listens. +//! +//! A piece of news is one message, its run and kind as attributes. A +//! process that listens makes its own subscription to the topic the first +//! time it is asked to (in the background, never on a request's path) and +//! pulls from it for as long as it lives, each pull held by Pub/Sub until a +//! message comes. Nobody can address one Cloud Run instance from outside, +//! so the instance reaches out instead. +//! +//! A subscription Pub/Sub dropped (a day without a pull, Pub/Sub's +//! shortest expiry, which only an instance frozen that long reaches) is +//! made again under the same name; news announced meanwhile was missed, so +//! [`NewsHeard::MissedSome`] goes out each time a subscription is up. The +//! process deletes its subscription as it stops; one that died before it +//! could is dropped by its expiry. + +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::Arc; +use std::time::Duration; + +use async_trait::async_trait; +use serde_json::{json, Value}; +use tokio::sync::broadcast; +use weft_platform_traits::config::GcpPlatform; +use weft_platform_traits::news::LISTEN_CAPACITY; +use weft_platform_traits::{Listening, News, NewsHeard, NewsKind, RunNews}; + +use crate::api::{is_status, Google}; + +/// How long Pub/Sub holds one pull for a message before it answers an +/// empty one: the deadline this side sets on it (`X-Server-Timeout`, which +/// Google's REST APIs take as the time the server has to finish), since +/// Pub/Sub states no bound of its own. +const SERVER_HOLD: Duration = Duration::from_secs(60); + +/// How long this side waits for a pull's answer: well past +/// [`SERVER_HOLD`], so a pull is never given up while Pub/Sub may still +/// answer it with messages it took for it (which would then wait out their +/// acknowledgement deadline before coming again). +const PULL_WAIT: Duration = Duration::from_secs(90); + +/// How long a message waits to be acknowledged before Pub/Sub sends it +/// again; it is acknowledged as soon as it is handed on. +const ACK_DEADLINE_SECS: u32 = 10; + +/// How long a message nobody pulled is kept: Pub/Sub's shortest. A +/// listener that missed one is told so once it is live again. +const RETENTION: &str = "600s"; + +/// How long a subscription nobody pulls from lives: Pub/Sub's shortest. +const EXPIRY: &str = "86400s"; + +/// How many messages one pull takes at most. +const PULL_AT_MOST: u32 = 1000; + +/// The pause after a call that failed, at first and at most. +const RETRY_FIRST: Duration = Duration::from_secs(1); +const RETRY_LONGEST: Duration = Duration::from_secs(60); + +/// The calls news makes to Pub/Sub, a trait so its loop is tested against +/// a fake. +#[async_trait] +pub trait PubSubCalls: Send + Sync { + async fn publish(&self, topic: &str, body: &Value) -> anyhow::Result<()>; + async fn create_subscription(&self, subscription: &str, body: &Value) -> anyhow::Result<()>; + /// Held until a message comes or [`SERVER_HOLD`] passed, answering an + /// empty pull then. + async fn pull(&self, subscription: &str, body: &Value) -> anyhow::Result; + async fn acknowledge(&self, subscription: &str, body: &Value) -> anyhow::Result<()>; + async fn delete_subscription(&self, subscription: &str) -> anyhow::Result<()>; +} + +const PUBSUB: &str = "https://pubsub.googleapis.com/v1"; + +/// Pub/Sub's REST API, as the runtime's own account. +pub struct GooglePubSub { + google: Google, +} + +#[async_trait] +impl PubSubCalls for GooglePubSub { + async fn publish(&self, topic: &str, body: &Value) -> anyhow::Result<()> { + self.google.post(&format!("{PUBSUB}/{topic}:publish"), body).await.map(|_| ()) + } + + async fn create_subscription(&self, subscription: &str, body: &Value) -> anyhow::Result<()> { + self.google.put(&format!("{PUBSUB}/{subscription}"), body).await.map(|_| ()) + } + + async fn pull(&self, subscription: &str, body: &Value) -> anyhow::Result { + let url = format!("{PUBSUB}/{subscription}:pull"); + let pull = self.google.http().post(&url).json(body).header("X-Server-Timeout", SERVER_HOLD.as_secs().to_string()).timeout(PULL_WAIT); + let resp = self.google.send_raw(pull).await.map_err(|e| e.context(format!("POST {url}")))?; + let resp = Google::success(resp).await.map_err(|e| e.context(format!("POST {url}")))?; + Ok(resp.json().await?) + } + + async fn acknowledge(&self, subscription: &str, body: &Value) -> anyhow::Result<()> { + self.google.post(&format!("{PUBSUB}/{subscription}:acknowledge"), body).await.map(|_| ()) + } + + async fn delete_subscription(&self, subscription: &str) -> anyhow::Result<()> { + self.google.delete(&format!("{PUBSUB}/{subscription}")).await.map(|_| ()) + } +} + +pub struct PubSubNews { + calls: Arc, + /// `projects/

/topics/`. + topic: String, + /// `projects/

/subscriptions/`: this process's own. + subscription: String, + heard: broadcast::Sender, + /// The loop pulling from the subscription, once something listens. + pulling: parking_lot::Mutex>>, + /// Whether the subscription was made: only one that was is deleted. + made: Arc, +} + +impl PubSubNews { + pub fn new(google: Google, gcp: &GcpPlatform) -> Self { + Self::over(Arc::new(GooglePubSub { google }), &gcp.project, &gcp.news_topic) + } + + /// News on topic `topic` of `project`, through `calls`, this process's + /// subscription named after the topic and a name of its own. + pub fn over(calls: Arc, project: &str, topic: &str) -> Self { + let own = uuid::Uuid::new_v4().simple().to_string(); + Self { + calls, + topic: format!("projects/{project}/topics/{topic}"), + subscription: format!("projects/{project}/subscriptions/{topic}-{own}"), + heard: broadcast::channel(LISTEN_CAPACITY).0, + pulling: parking_lot::Mutex::new(None), + made: Arc::new(AtomicBool::new(false)), + } + } +} + +/// The attribute values a message carries its kind in. +// SYNC: kind_name <-> kind_named +fn kind_name(kind: NewsKind) -> &'static str { + match kind { + NewsKind::Answer => "answer", + NewsKind::Cancel => "cancel", + } +} + +// SYNC: kind_named <-> kind_name +fn kind_named(name: &str) -> Option { + match name { + "answer" => Some(NewsKind::Answer), + "cancel" => Some(NewsKind::Cancel), + _ => None, + } +} + +/// One publish announcing every piece of `news`, each a message of +/// nothing but attributes. +fn publish_body(news: &[RunNews]) -> Value { + let messages: Vec = news + .iter() + .map(|news| { + json!({ + "attributes": { + "project": news.project.to_string(), + "execution": news.execution.to_string(), + "kind": kind_name(news.kind), + }, + }) + }) + .collect(); + json!({ "messages": messages }) +} + +/// The news a message's attributes say, if they are news. +fn news_in(message: &Value) -> Option { + let attribute = |name: &str| message.get("attributes")?.get(name)?.as_str(); + Some(RunNews { + project: attribute("project")?.parse().ok()?, + execution: attribute("execution")?.parse().ok()?, + kind: kind_named(attribute("kind")?)?, + }) +} + +/// A subscription of `topic` that keeps nothing past what news needs. +fn subscription_body(topic: &str) -> Value { + json!({ + "topic": topic, + "ackDeadlineSeconds": ACK_DEADLINE_SECS, + "messageRetentionDuration": RETENTION, + "expirationPolicy": { "ttl": EXPIRY }, + }) +} + +/// What a pull answered: each message's ack id, and the news it is +/// (`None` for a message that is not, acknowledged and dropped loudly: +/// nothing but weft publishes to the topic). +fn pulled(answer: &Value) -> Vec<(String, Option)> { + let Some(received) = answer.get("receivedMessages").and_then(Value::as_array) else { return Vec::new() }; + received + .iter() + .filter_map(|r| Some((r.get("ackId").and_then(Value::as_str)?.to_string(), r.get("message").and_then(news_in)))) + .collect() +} + +/// Make the subscription and pull from it, handing on what it hears, for +/// as long as the process lives (the task is stopped by [`News::close`]). +/// [`NewsHeard::MissedSome`] goes out as soon as a subscription is made: +/// Pub/Sub keeps for it every message published from then on, so whatever +/// came before is all that was missed. A call that fails, a subscription +/// that keeps going away included, waits a growing pause before the next, +/// so listeners are told to look again at that pace at most. +async fn pull_forever(calls: Arc, topic: String, subscription: String, heard: broadcast::Sender, made: Arc) { + let mut retry = RETRY_FIRST; + let pull = json!({ "maxMessages": PULL_AT_MOST }); + let pause = |retry: &mut Duration| { + let now = *retry; + *retry = (*retry * 2).min(RETRY_LONGEST); + tokio::time::sleep(now) + }; + loop { + match calls.create_subscription(&subscription, &subscription_body(&topic)).await { + // Already there: an earlier make whose answer was lost. + Ok(()) => {} + Err(e) if is_status(&e, 409) => {} + Err(e) => { + tracing::warn!(target: "weft_platform_gcp::news", %subscription, error = %format!("{e:#}"), "could not subscribe to the install's news; trying again"); + pause(&mut retry).await; + continue; + } + } + made.store(true, Ordering::Release); + tracing::info!(target: "weft_platform_gcp::news", %subscription, "listening to the install's news"); + // Nobody listening yet is fine: a listener comes later. + let _ = heard.send(NewsHeard::MissedSome); + loop { + let answer = match calls.pull(&subscription, &pull).await { + Ok(answer) => answer, + Err(e) if is_status(&e, 404) => { + tracing::warn!(target: "weft_platform_gcp::news", %subscription, "the news subscription is gone; making it again"); + pause(&mut retry).await; + break; + } + Err(e) => { + tracing::warn!(target: "weft_platform_gcp::news", %subscription, error = %format!("{e:#}"), "could not pull the install's news; trying again"); + pause(&mut retry).await; + continue; + } + }; + retry = RETRY_FIRST; + let messages = pulled(&answer); + if messages.is_empty() { + continue; + } + let mut acks = Vec::with_capacity(messages.len()); + for (ack, news) in messages { + match news { + Some(news) => { + let _ = heard.send(NewsHeard::News(news)); + } + None => tracing::error!(target: "weft_platform_gcp::news", %subscription, "a message on the news topic that is not news; dropped"), + } + acks.push(ack); + } + // A message not acknowledged comes again, which costs a look + // and nothing else. + if let Err(e) = calls.acknowledge(&subscription, &json!({ "ackIds": acks })).await { + tracing::warn!(target: "weft_platform_gcp::news", %subscription, error = %format!("{e:#}"), "could not acknowledge the news heard; it comes again"); + } + } + } +} + +#[async_trait] +impl News for PubSubNews { + async fn announce(&self, news: &[RunNews]) -> anyhow::Result<()> { + self.calls.publish(&self.topic, &publish_body(news)).await + } + + fn listen(&self) -> Listening { + let listening = self.heard.subscribe(); + let mut pulling = self.pulling.lock(); + if pulling.is_none() { + *pulling = Some(tokio::spawn(pull_forever( + self.calls.clone(), + self.topic.clone(), + self.subscription.clone(), + self.heard.clone(), + self.made.clone(), + ))); + } + listening.into() + } + + async fn close(&self) -> anyhow::Result<()> { + let Some(pulling) = self.pulling.lock().take() else { return Ok(()) }; + pulling.abort(); + if !self.made.load(Ordering::Acquire) { + return Ok(()); + } + self.calls.delete_subscription(&self.subscription).await + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::api::ApiError; + + fn news(kind: NewsKind) -> RunNews { + RunNews { project: uuid::Uuid::from_u128(1), execution: uuid::Uuid::from_u128(2), kind } + } + + #[test] + fn an_announcement_reads_back_as_the_news_it_was() { + for kind in [NewsKind::Answer, NewsKind::Cancel] { + let body = publish_body(&[news(kind)]); + let answer = json!({ "receivedMessages": [{ "ackId": "a1", "message": body["messages"][0] }] }); + assert_eq!(pulled(&answer), vec![("a1".to_string(), Some(news(kind)))]); + } + } + + #[test] + fn a_message_that_is_not_news_is_still_acknowledged() { + let answer = json!({ "receivedMessages": [{ "ackId": "a1", "message": { "data": "eA==" } }] }); + assert_eq!(pulled(&answer), vec![("a1".to_string(), None)]); + assert!(pulled(&json!({})).is_empty(), "an empty pull hears nothing"); + } + + #[test] + fn a_subscription_keeps_nothing_past_what_news_needs() { + let body = subscription_body("projects/p/topics/t"); + assert_eq!(body["topic"], "projects/p/topics/t"); + assert_eq!(body["expirationPolicy"]["ttl"], "86400s"); + assert_eq!(body["messageRetentionDuration"], "600s"); + } + + /// Every call made, in order, what each pull answers (none left: a + /// pull that never answers, or the last answer again when `repeat`), + /// and whether making a subscription fails. + #[derive(Default)] + struct FakePubSub { + calls: parking_lot::Mutex>, + pulls: parking_lot::Mutex>>, + repeat_gone: bool, + create_fails: bool, + } + + impl FakePubSub { + fn log(&self, call: &'static str, on: &str, body: Value) { + self.calls.lock().push((call, on.to_string(), body)); + } + + fn made(&self) -> Vec<&'static str> { + self.calls.lock().iter().map(|(call, _, _)| *call).collect() + } + } + + #[async_trait] + impl PubSubCalls for FakePubSub { + async fn publish(&self, topic: &str, body: &Value) -> anyhow::Result<()> { + self.log("publish", topic, body.clone()); + Ok(()) + } + async fn create_subscription(&self, subscription: &str, body: &Value) -> anyhow::Result<()> { + self.log("create", subscription, body.clone()); + match self.create_fails { + true => Err(anyhow::Error::new(ApiError { status: 403, body: "denied".into() })), + false => Ok(()), + } + } + async fn pull(&self, subscription: &str, body: &Value) -> anyhow::Result { + self.log("pull", subscription, body.clone()); + if self.repeat_gone { + return Err(anyhow::Error::new(ApiError { status: 404, body: "gone".into() })); + } + let next = self.pulls.lock().pop_front(); + match next { + Some(answer) => answer, + None => std::future::pending().await, + } + } + async fn acknowledge(&self, subscription: &str, body: &Value) -> anyhow::Result<()> { + self.log("ack", subscription, body.clone()); + Ok(()) + } + async fn delete_subscription(&self, subscription: &str) -> anyhow::Result<()> { + self.log("delete", subscription, Value::Null); + Ok(()) + } + } + + async fn next(listening: &mut Listening) -> NewsHeard { + tokio::time::timeout(Duration::from_secs(10), listening.next()).await.expect("heard in time").expect("still listening") + } + + /// The first listener makes the subscription (with its expiry), is + /// told as soon as it is made, hears each message pulled with each one + /// acknowledged; a subscription Pub/Sub dropped is made again after a + /// pause and said live again, and closing deletes it. + #[tokio::test(start_paused = true)] + async fn news_is_pulled_acknowledged_and_its_subscription_kept() { + let fake = Arc::new(FakePubSub::default()); + let message = publish_body(&[news(NewsKind::Answer)])["messages"][0].clone(); + fake.pulls.lock().extend([ + Ok(json!({ "receivedMessages": [{ "ackId": "a1", "message": message }] })), + Err(anyhow::Error::new(ApiError { status: 404, body: "gone".into() })), + Ok(json!({})), + ]); + let pubsub = PubSubNews::over(fake.clone(), "proj", "weft-news"); + pubsub.announce(&[news(NewsKind::Cancel), news(NewsKind::Answer)]).await.unwrap(); + assert_eq!(fake.made(), vec!["publish"], "announcing listens to nothing, and a batch is one publish"); + assert_eq!(fake.calls.lock()[0].2["messages"].as_array().unwrap().len(), 2); + assert_eq!(fake.calls.lock()[0].1, "projects/proj/topics/weft-news"); + + let mut listening = pubsub.listen(); + assert_eq!(next(&mut listening).await, NewsHeard::MissedSome); + assert_eq!(next(&mut listening).await, NewsHeard::News(news(NewsKind::Answer))); + assert_eq!(next(&mut listening).await, NewsHeard::MissedSome, "the dropped subscription was made again"); + let calls = fake.calls.lock().clone(); + assert_eq!(calls.iter().map(|(call, _, _)| *call).take(6).collect::>(), ["publish", "create", "pull", "ack", "pull", "create"]); + assert!(calls[1].1.starts_with("projects/proj/subscriptions/weft-news-"), "{}", calls[1].1); + assert_eq!(calls[1].2["expirationPolicy"]["ttl"], EXPIRY); + assert_eq!(calls[3].2, json!({ "ackIds": ["a1"] })); + assert_eq!(calls[5].1, calls[1].1, "made again under its own name"); + + let _second = pubsub.listen(); + pubsub.close().await.unwrap(); + let made = fake.made(); + assert_eq!(made.iter().filter(|c| **c == "create").count(), 2, "a second listener shares the one subscription"); + assert_eq!(made.last(), Some(&"delete")); + } + + /// A subscription that is gone again at every pull is made again only + /// after a growing pause, so listeners are told to look again at that + /// pace, not in a loop. + #[tokio::test(start_paused = true)] + async fn a_subscription_that_keeps_going_away_is_made_again_slowly() { + let fake = Arc::new(FakePubSub { repeat_gone: true, ..FakePubSub::default() }); + let pubsub = PubSubNews::over(fake.clone(), "proj", "weft-news"); + let mut listening = pubsub.listen(); + tokio::time::sleep(Duration::from_secs(60)).await; + let creates = fake.made().iter().filter(|c| **c == "create").count(); + assert!((2..=8).contains(&creates), "made again after a growing pause, not in a loop: {creates}"); + let mut told = 0; + while let Ok(Ok(heard)) = tokio::time::timeout(Duration::from_millis(1), listening.next()).await { + assert_eq!(heard, NewsHeard::MissedSome); + told += 1; + } + assert_eq!(told, creates, "told once per subscription made"); + } + + /// A subscription that was never made is not deleted as the process + /// stops. + #[tokio::test(start_paused = true)] + async fn only_a_subscription_that_was_made_is_deleted() { + let fake = Arc::new(FakePubSub { create_fails: true, ..FakePubSub::default() }); + let pubsub = PubSubNews::over(fake.clone(), "proj", "weft-news"); + let _listening = pubsub.listen(); + tokio::time::sleep(Duration::from_secs(5)).await; + pubsub.close().await.unwrap(); + assert!(!fake.made().contains(&"delete"), "{:?}", fake.made()); + } +} diff --git a/crates/weft-platform-gcp/src/runner.rs b/crates/weft-platform-gcp/src/runner.rs index 8635de54..1b05bfa9 100644 --- a/crates/weft-platform-gcp/src/runner.rs +++ b/crates/weft-platform-gcp/src/runner.rs @@ -30,8 +30,12 @@ use weft_platform_traits::{Patience, Runner, WorkerEndpoint, WorkerSettings, Wor use crate::api::{is_status, Google}; use crate::names; -/// The longest one request may run on Cloud Run. -const REQUEST_CAP: Duration = Duration::from_secs(3600); +/// The longest one request may run on the Cloud Run services weft makes: +/// a project's workers here, and weft's own roles (its Terraform). The +/// broker tells every line to it this limit, and a worker whose runs only +/// wait stops before its line reaches it (`weft_broker_client::line`). +// SYNC: REQUEST_CAP <-> deploy/terraform/gcp/serverless.tf (google_cloud_run_v2_service.role timeout) +pub const REQUEST_CAP: Duration = Duration::from_secs(3600); /// How long a bring-up waits for a deploy in flight to settle, and how /// often it looks. Cloud Run settles a revision in minutes; past this the @@ -786,6 +790,7 @@ mod tests { build_bucket: "acme-weft-builds".into(), builder_service_account: "weft-builder@acme.iam.gserviceaccount.com".into(), tasks_queue: "weft-wakes".into(), + news_topic: "weft-news".into(), core_service_account: "weft-core@acme.iam.gserviceaccount.com".into(), holder_pool: "projects/acme/locations/us-central1/workerPools/weft-holder".into(), dispatcher_service: "weft-role-dispatcher".into(), diff --git a/crates/weft-platform-local/src/lib.rs b/crates/weft-platform-local/src/lib.rs index 30f2f944..75dbc0c6 100644 --- a/crates/weft-platform-local/src/lib.rs +++ b/crates/weft-platform-local/src/lib.rs @@ -9,6 +9,7 @@ //! - `images`: `docker build`, into the daemon's own store. //! - `alarm`: wakes kept in Postgres, delivered by the runtime. //! - `identity`: identities the install signs with its own key. +//! - `news`: nothing to carry, the database being listened to for good. pub mod alarm; pub mod docker; @@ -18,6 +19,7 @@ pub mod holders; pub mod identity; pub mod images; pub mod infra_host; +pub mod news; pub mod runner; pub use alarm::{Deliver, HttpDeliver, LocalAlarm, NotTaken}; @@ -28,4 +30,5 @@ pub use holders::OneProcessHolds; pub use identity::LocalIdentity; pub use images::{bound_build_cache, DockerImageBuilder}; pub use infra_host::{AgentBroker, AgentIdentity, DiskBacking, GpuAccess, LocalInfraHost, LocalInfraHostConfig, Publish}; +pub use news::DatabaseNews; pub use runner::{LocalRunner, LocalRunnerConfig}; diff --git a/crates/weft-platform-local/src/news.rs b/crates/weft-platform-local/src/news.rs new file mode 100644 index 00000000..800221dc --- /dev/null +++ b/crates/weft-platform-local/src/news.rs @@ -0,0 +1,34 @@ +//! News on a local install: its database never sleeps, and its one +//! process listens to it for good, hearing every write that hands a run an +//! answer or asks it to stop (`weft_task_store::parked_fires`, +//! `weft_task_store::runs`). News carries nothing on top. + +use async_trait::async_trait; +use tokio::sync::broadcast; +use weft_platform_traits::{Listening, News, NewsHeard, RunNews}; + +pub struct DatabaseNews { + /// Kept so every listener stays open, hearing nothing. + heard: broadcast::Sender, +} + +impl Default for DatabaseNews { + fn default() -> Self { + Self { heard: broadcast::channel(1).0 } + } +} + +#[async_trait] +impl News for DatabaseNews { + async fn announce(&self, _news: &[RunNews]) -> anyhow::Result<()> { + Ok(()) + } + + fn listen(&self) -> Listening { + self.heard.subscribe().into() + } + + async fn close(&self) -> anyhow::Result<()> { + Ok(()) + } +} diff --git a/crates/weft-platform-traits/src/config.rs b/crates/weft-platform-traits/src/config.rs index d5d553e0..1935d830 100644 --- a/crates/weft-platform-traits/src/config.rs +++ b/crates/weft-platform-traits/src/config.rs @@ -147,6 +147,9 @@ pub struct GcpPlatform { /// The Cloud Tasks queue wakes go through. #[serde(rename = "tasksQueue")] pub tasks_queue: String, + /// The Pub/Sub topic news goes through (`crate::news`). + #[serde(rename = "newsTopic")] + pub news_topic: String, /// The service account weft's own roles run as. #[serde(rename = "coreServiceAccount")] pub core_service_account: String, @@ -569,6 +572,7 @@ pub(crate) mod tests { build_bucket: "b".into(), builder_service_account: "builder@p".into(), tasks_queue: "q".into(), + news_topic: "weft-news".into(), core_service_account: "core@p".into(), holder_pool: "projects/p/locations/r/workerPools/weft-holder".into(), dispatcher_service: "weft-role-dispatcher".into(), diff --git a/crates/weft-platform-traits/src/lib.rs b/crates/weft-platform-traits/src/lib.rs index 1fdcbc86..47d1228d 100644 --- a/crates/weft-platform-traits/src/lib.rs +++ b/crates/weft-platform-traits/src/lib.rs @@ -19,6 +19,8 @@ //! - `frontends`: hosting a project's frontend on the install's cloud. //! - `domains`: the door in front of the install's own domains. //! - `holder_pool`: how many holders keep held connections open. +//! - `news`: telling the install's other copies what happened to a run, +//! without the database. //! - `clock`: time, abstracted so tests advance it deterministically. //! - `unit_agent`: what weft asks the agent beside an infra unit. //! @@ -34,6 +36,7 @@ pub mod holder_pool; pub mod identity; pub mod images; pub mod infra_host; +pub mod news; pub mod object_store; pub mod roles; pub mod runner; @@ -48,6 +51,7 @@ pub use frontends::{FrontendHosting, FrontendSite, HostedFrontend}; pub use holder_pool::HolderPool; pub use images::{BuildHandle, BuildRequest, BuildStatus, ImageBuilder, ImageDeleted, Staging}; pub use infra_host::{EndpointAt, InfraHost, UnitObservation, UnitRunState}; +pub use news::{Listening, News, NewsHeard, NewsKind, RunNews}; pub use object_store::{ object_store_for, ObjectEntry, ObjectStore, ObjectStoreConfig, PresignAudience, S3ObjectStore, SharedObjectStore, @@ -67,6 +71,8 @@ pub use holder_pool::fake::FakeHolderPool; #[cfg(any(test, feature = "test-helpers"))] pub use images::FakeImageBuilder; #[cfg(any(test, feature = "test-helpers"))] +pub use news::fake::FakeNews; +#[cfg(any(test, feature = "test-helpers"))] pub use object_store::fake::{FakeCall as ObjectStoreFakeCall, FakeObjectStore}; #[cfg(any(test, feature = "test-helpers"))] pub use runner::fake::{FakeRunner, RunnerCall}; diff --git a/crates/weft-platform-traits/src/news.rs b/crates/weft-platform-traits/src/news.rs new file mode 100644 index 00000000..a93e8578 --- /dev/null +++ b/crates/weft-platform-traits/src/news.rs @@ -0,0 +1,140 @@ +//! Telling the install's other copies that something happened to a run, +//! without the database. +//! +//! A run's worker keeps its line open to one broker copy while it drives +//! the run, and that copy has to pass on what happened to the run +//! elsewhere: a person answered what it waits on, or somebody cancelled it. +//! The copy that wrote it is a dispatcher copy, and a serverless database +//! that sleeps once nothing queries it cannot carry the word: listening to +//! it means holding a connection open, which keeps it awake. News carries +//! it instead, and only the word: which run, and what kind of thing +//! happened. Whoever hears it reads what happened through weft, as on any +//! announcement. +//! +//! The contract is a doorbell: news may be lost, heard twice or out of +//! order. A listener that may have missed some is told so +//! ([`NewsHeard::MissedSome`]) and looks at everything it waits on again. + +use async_trait::async_trait; +use tokio::sync::broadcast; + +/// What happened to a run, as news says it. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub enum NewsKind { + /// An answer to one of its waits was handed to its worker. + Answer, + /// It was asked to stop. + Cancel, +} + +/// One piece of news: run `execution` of `project` had something happen. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct RunNews { + pub project: uuid::Uuid, + pub execution: weft_core::ExecutionId, + pub kind: NewsKind, +} + +/// What a listener hears. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum NewsHeard { + News(RunNews), + /// The listening started, or came back after a gap: news may have been + /// missed, so whatever waits on some looks again. + MissedSome, +} + +#[async_trait] +pub trait News: Send + Sync { + /// Tell every copy listening each piece of `news`, in one call. Called + /// once the writes it announces committed; an error means none of it + /// went out for sure, and the caller announces it again. + async fn announce(&self, news: &[RunNews]) -> anyhow::Result<()>; + + /// Everything heard from now on. The first call starts the process's + /// listening, in the background (never on its caller's path), and + /// [`NewsHeard::MissedSome`] says when it is up. + fn listen(&self) -> Listening; + + /// The process is stopping: what its listening holds goes. + async fn close(&self) -> anyhow::Result<()>; +} + +/// How many pieces of news a slow listener may fall behind by before it is +/// told it missed some. +pub const LISTEN_CAPACITY: usize = 1024; + +/// One listener's view of the news. +pub struct Listening { + heard: broadcast::Receiver, +} + +impl From> for Listening { + fn from(heard: broadcast::Receiver) -> Self { + Self { heard } + } +} + +impl Listening { + /// The next thing heard. Falling behind is missing some; the news + /// stopping is an error, since nothing would be heard again. + pub async fn next(&mut self) -> anyhow::Result { + match self.heard.recv().await { + Ok(heard) => Ok(heard), + Err(broadcast::error::RecvError::Lagged(_)) => Ok(NewsHeard::MissedSome), + Err(broadcast::error::RecvError::Closed) => anyhow::bail!("the install's news stopped"), + } + } +} + +/// Records every announcement, and hands every listener what a test sends. +#[cfg(any(test, feature = "test-helpers"))] +pub mod fake { + use super::*; + + pub struct FakeNews { + /// Every call, each the batch it announced. + pub announced: parking_lot::Mutex>>, + pub heard: broadcast::Sender, + } + + impl Default for FakeNews { + fn default() -> Self { + Self { announced: parking_lot::Mutex::default(), heard: broadcast::channel(LISTEN_CAPACITY).0 } + } + } + + #[async_trait] + impl News for FakeNews { + async fn announce(&self, news: &[RunNews]) -> anyhow::Result<()> { + self.announced.lock().push(news.to_vec()); + Ok(()) + } + + fn listen(&self) -> Listening { + self.heard.subscribe().into() + } + + async fn close(&self) -> anyhow::Result<()> { + Ok(()) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn a_listener_that_fell_behind_is_told_it_missed_some() { + let (tx, rx) = broadcast::channel(1); + let mut listening = Listening::from(rx); + let news = RunNews { project: uuid::Uuid::from_u128(1), execution: uuid::Uuid::from_u128(2), kind: NewsKind::Answer }; + tx.send(NewsHeard::News(news)).unwrap(); + tx.send(NewsHeard::News(news)).unwrap(); + assert_eq!(listening.next().await.unwrap(), NewsHeard::MissedSome); + assert_eq!(listening.next().await.unwrap(), NewsHeard::News(news)); + drop(tx); + assert!(listening.next().await.is_err(), "news that stopped is loud"); + } +} diff --git a/crates/weft-runtime/Cargo.toml b/crates/weft-runtime/Cargo.toml index 8a7b42ef..7825d086 100644 --- a/crates/weft-runtime/Cargo.toml +++ b/crates/weft-runtime/Cargo.toml @@ -22,7 +22,7 @@ weft-broker = { path = "../weft-broker" } weft-broker-client = { path = "../weft-broker-client" } weft-listener = { path = "../weft-listener" } weft-infra-supervisor = { path = "../weft-infra-supervisor" } -weft-task-store = { workspace = true } +weft-task-store = { workspace = true, features = ["serve"] } weft-platform-traits = { workspace = true } weft-platform-local = { workspace = true } weft-platform-gcp = { workspace = true, features = ["control"] } @@ -41,7 +41,6 @@ tracing = { workspace = true } tracing-subscriber = { workspace = true } uuid = { workspace = true } parking_lot = { workspace = true } -http-body = "1" [dev-dependencies] weft-platform-traits = { workspace = true, features = ["test-helpers"] } diff --git a/crates/weft-runtime/src/lib.rs b/crates/weft-runtime/src/lib.rs index 09949d27..2723f9a4 100644 --- a/crates/weft-runtime/src/lib.rs +++ b/crates/weft-runtime/src/lib.rs @@ -180,6 +180,21 @@ pub async fn serve(config: InstallConfig, only: Option) -> anyhow::Res _ => None, }; + // What happened to a run, told across copies: a write leaves it in the + // outbox, which this process's flusher tells once it is given the + // news (`weft_task_store::announce`), and the broker, which holds the + // line of the run's worker, hears it (`weft_broker::news`). Every + // process that writes tells it, so news a dead one left goes out with + // the next write of any. + let news = match &pool { + Some((_, pool)) if writes => { + let news = platform::news(&config, &parts)?; + weft_task_store::announce::tell_news_through(pool, news.clone()); + Some(news) + } + _ => None, + }; + let mut internal = Router::new(); let mut public = Router::new(); let mut outside: Option = None; @@ -212,9 +227,12 @@ pub async fn serve(config: InstallConfig, only: Option) -> anyhow::Res public_base_url: config.public_url.clone(), internet_url: config.internet_url.clone(), object_store_public_internet: config.object_store.public_internet(), + line_max_age: platform::line_max_age(&config, serverless), }, ) .await?; + let heard = news.as_ref().expect("the broker's process hears news").listen(); + weft_dispatcher::app::spawn_supervised("news", weft_broker::news::pass_on(heard, state.signals.also_told())); loops.push((CoreRole::Broker, Loops::Drain(weft_broker::drain_loops(state.clone())))); internal = mount(internal, CoreRole::Broker, weft_broker::router(state)); } @@ -456,7 +474,7 @@ pub async fn serve(config: InstallConfig, only: Option) -> anyhow::Res let addr: std::net::SocketAddr = ([0, 0, 0, 0], port).into(); let mut app = public.merge(internal); if let Some(signals) = &signals { - app = app.layer(axum::middleware::from_fn_with_state(signals.clone(), server::keep_listening)); + app = app.layer(axum::middleware::from_fn_with_state(signals.clone(), weft_task_store::pg_signal::keep_listening)); for t in first_passes { let signals = signals.clone(); tokio::spawn(async move { @@ -516,5 +534,10 @@ pub async fn serve(config: InstallConfig, only: Option) -> anyhow::Res if let Some(waker) = waker_at_exit { waker.settle(role_waker::RING_GRACE).await; } + if let Some(news) = news { + if let Err(e) = news.close().await { + tracing::warn!(target: "weft_runtime", error = %format!("{e:#}"), "could not let go of this process's news; the platform drops it on its own later"); + } + } served? } diff --git a/crates/weft-runtime/src/platform.rs b/crates/weft-runtime/src/platform.rs index 80f70072..745ef91c 100644 --- a/crates/weft-runtime/src/platform.rs +++ b/crates/weft-runtime/src/platform.rs @@ -163,6 +163,31 @@ pub async fn build(config: &InstallConfig, pool: Option<&sqlx::PgPool>) -> anyho } } +/// How this process tells the install's other copies what happened to a +/// run, and hears it (`weft_platform_traits::news`): on a machine, nothing +/// (its one process listens to the database for good); on Google Cloud, the +/// install's Pub/Sub topic. +pub fn news(config: &InstallConfig, parts: &Parts) -> anyhow::Result> { + Ok(match &config.platform { + PlatformConfig::Local(_) => Arc::new(weft_platform_local::DatabaseNews::default()), + PlatformConfig::Gcp(gcp) => { + let google = parts.google.clone().context("a process on Google Cloud holds its Google client")?; + Arc::new(weft_platform_gcp::PubSubNews::new(google, gcp)) + } + }) +} + +/// How long this process's platform lets one request live before cutting +/// it, which is how long a line to it lasts: a role placed serverless on +/// Google Cloud is a Cloud Run service; anything else keeps a request for +/// good. +pub fn line_max_age(config: &InstallConfig, serverless: bool) -> Option { + match (&config.platform, serverless) { + (PlatformConfig::Gcp(_), true) => Some(weft_platform_gcp::REQUEST_CAP), + _ => None, + } +} + /// The object store the install config names. A Cloud Storage bucket is /// reached through the platform's own Google client, so only on GCP. pub async fn object_store(settings: &ObjectStoreSettings, parts: &Parts) -> anyhow::Result { diff --git a/crates/weft-runtime/src/server.rs b/crates/weft-runtime/src/server.rs index d8879760..bba1bc8b 100644 --- a/crates/weft-runtime/src/server.rs +++ b/crates/weft-runtime/src/server.rs @@ -24,7 +24,6 @@ use axum::routing::post; use axum::Router; use weft_platform_traits::roles::TICK_PATH; use weft_platform_traits::{Alarm, CoreRole, Wake}; -use weft_task_store::pg_signal::{Busy, PgSignalWatch}; /// A role's own tick, for a role that scales to zero: drain its loops, /// then set its next wake, if any loop wants one. `run` takes the loops @@ -82,47 +81,6 @@ async fn tick(State(t): State, axum::extract::RawQuery(query): axum::extra } } -/// Keep `watch` listening while a request is answered, to the end of its -/// answer's body: a stream of events is listened for until it closes. A -/// process that scales to zero lets its watch go quiet only between -/// requests (`weft_task_store::pg_signal::Listening::WhileBusy`), and -/// every request it answers may write, so a quiet watch listens again -/// before the request goes on. -pub async fn keep_listening( - State(watch): State>, - request: axum::extract::Request, - next: axum::middleware::Next, -) -> Response { - let busy = watch.busy().await; - next.run(request).await.map(|body| axum::body::Body::new(Held { body, _busy: busy })) -} - -/// An answer's body, holding its process busy until it ends. -struct Held { - body: axum::body::Body, - _busy: Busy, -} - -impl http_body::Body for Held { - type Data = axum::body::Bytes; - type Error = axum::Error; - - fn poll_frame( - mut self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> std::task::Poll, Self::Error>>> { - std::pin::Pin::new(&mut self.body).poll_frame(cx) - } - - fn is_end_stream(&self) -> bool { - self.body.is_end_stream() - } - - fn size_hint(&self) -> http_body::SizeHint { - self.body.size_hint() - } -} - /// The tick route, for a role placed serverless. pub fn tick_route(t: Tick) -> Router { Router::new().route(TICK_PATH, post(tick)).with_state(t) diff --git a/crates/weft-task-store/Cargo.toml b/crates/weft-task-store/Cargo.toml index 03680ace..e1d5a4a9 100644 --- a/crates/weft-task-store/Cargo.toml +++ b/crates/weft-task-store/Cargo.toml @@ -10,7 +10,10 @@ repository.workspace = true # Postgres via `#[sqlx::test]` + $DATABASE_URL. OFF by default so # `cargo test --workspace` is green on any machine without Postgres; CI runs # `cargo test -p weft-task-store --features db-tests` with a Postgres up. -db-tests = ["dep:chrono"] +db-tests = ["dep:chrono", "serve"] +# The middleware that keeps a process's watch listening while it answers +# a request (`pg_signal::keep_listening`), for the processes that serve. +serve = ["dep:axum", "dep:http-body"] [dependencies] # Only for stamping a generated migration's file name with the date it was @@ -18,6 +21,8 @@ db-tests = ["dep:chrono"] # production builds never link it. chrono = { workspace = true, optional = true } anyhow = { workspace = true } +axum = { workspace = true, optional = true } +http-body = { version = "1", optional = true } async-trait = { workspace = true } futures = { workspace = true } serde = { workspace = true } @@ -29,8 +34,12 @@ tokio-util = { version = "0.7" } tracing = { workspace = true } uuid = { workspace = true } weft-core = { workspace = true } +# What the outbox tells news through (`announce::tell_news_through`). +weft-platform-traits = { workspace = true } [dev-dependencies] +# Driving `pg_signal::keep_listening` in the database tests (`oneshot`). +tower = { workspace = true, features = ["util"] } # `test-util` enables tokio's paused virtual clock (start_paused + # advance) so the timing tests run deterministically without real # sleeps. diff --git a/crates/weft-task-store/migrations/announcement/20261010T121746_announcement_news_outbox.sql b/crates/weft-task-store/migrations/announcement/20261010T121746_announcement_news_outbox.sql new file mode 100644 index 00000000..09947a36 --- /dev/null +++ b/crates/weft-task-store/migrations/announcement/20261010T121746_announcement_news_outbox.sql @@ -0,0 +1,26 @@ +CREATE TABLE weft_news ( + execution_id uuid NOT NULL, + kind text NOT NULL, + project_id uuid NOT NULL, + seq bigint GENERATED ALWAYS AS IDENTITY NOT NULL +); + +ALTER TABLE weft_news ADD CONSTRAINT weft_news_execution_id_not_null NOT NULL execution_id; + +ALTER TABLE weft_news ADD CONSTRAINT weft_news_kind_check CHECK ((kind = ANY (ARRAY['answer'::text, 'cancel'::text]))); + +ALTER TABLE weft_news ADD CONSTRAINT weft_news_kind_not_null NOT NULL kind; + +ALTER TABLE weft_news ADD CONSTRAINT weft_news_pkey PRIMARY KEY (seq); + +ALTER TABLE weft_news ADD CONSTRAINT weft_news_project_id_not_null NOT NULL project_id; + +ALTER TABLE weft_news ADD CONSTRAINT weft_news_seq_not_null NOT NULL seq; + +CREATE OR REPLACE FUNCTION weft_tell(p_project uuid, p_execution uuid, p_kind text) + RETURNS void + LANGUAGE sql +AS $function$ + INSERT INTO weft_news (project_id, execution_id, kind) VALUES (p_project, p_execution, p_kind) + $function$ +; diff --git a/crates/weft-task-store/migrations/parked_fire/20261010T121746_parked_fire_news_outbox.sql b/crates/weft-task-store/migrations/parked_fire/20261010T121746_parked_fire_news_outbox.sql new file mode 100644 index 00000000..aca1c2e9 --- /dev/null +++ b/crates/weft-task-store/migrations/parked_fire/20261010T121746_parked_fire_news_outbox.sql @@ -0,0 +1,18 @@ +CREATE OR REPLACE FUNCTION parked_fire_notify() + RETURNS trigger + LANGUAGE plpgsql +AS $function$ + DECLARE + v_project UUID; + BEGIN + IF NEW.execution_id IS NULL THEN + PERFORM weft_announce('weft_parked_fire', (SELECT s.project_id FROM signal s WHERE s.token = NEW.token)::text); + ELSE + SELECT r.project_id INTO v_project FROM run r WHERE r.execution_id = NEW.execution_id; + PERFORM weft_announce('weft_parked_fire', v_project::text || ' ' || NEW.execution_id::text); + PERFORM weft_tell(v_project, NEW.execution_id, 'answer'); + END IF; + RETURN NULL; + END; + $function$ +; diff --git a/crates/weft-task-store/src/announce.rs b/crates/weft-task-store/src/announce.rs index 162869e6..b15b27da 100644 --- a/crates/weft-task-store/src/announce.rs +++ b/crates/weft-task-store/src/announce.rs @@ -35,23 +35,44 @@ //! The rare writes (a connection changed, an infra copy's status, a domain) //! still notify from their own trigger: they commit seldom, so the lock //! costs them nothing, and they keep needing no poke. +//! +//! The same outbox carries the install's news (`weft_platform_traits::News`): +//! what happened to a run a worker drives (an answer handed to it, a cancel +//! asked of it), which the broker copy holding that worker's line must hear +//! while the database may sleep. The write leaves it in [`NEWS_TABLE`] +//! (`weft_tell(project, execution, kind)`), committing with it, and its +//! writer then says so ([`news_written`]: only a cancel or an answer +//! reaching a run pays a look at the news, never the run path's writes). Beside the flushing, once the process gave it its news +//! ([`tell_news_through`]), the flusher's task reads a batch, announces it +//! in one call, and only then takes exactly those rows out. A batch that +//! cannot go out is kept in memory and announced again, loudly, with a +//! growing pause, until it does; the database is not touched meanwhile, +//! and the announcements never wait on it. use std::collections::HashMap; use std::sync::{Arc, Mutex, OnceLock}; use std::time::Duration; use sqlx::postgres::PgPool; +use weft_platform_traits::{News, NewsKind, RunNews}; /// Where the writes a run makes leave what they announce. pub const TABLE: &str = "weft_announcement"; +/// Where the writes leave the news they make, until it is told. +pub const NEWS_TABLE: &str = "weft_news"; + /// How many announcements one flush sends at most; a fuller outbox is /// flushed again at once. const BATCH: i32 = 10_000; +/// How many pieces of news one batch tells at most: one announcement each +/// (Pub/Sub takes up to a thousand messages in one publish). +const NEWS_BATCH: i64 = 100; + pub static GROUP: crate::SchemaGroup = crate::SchemaGroup { name: "announcement", - tables: &["weft_announcement"], + tables: &["weft_announcement", "weft_news"], ddl: &[ // UNLOGGED: an announcement only says when to look at rows that // are durable on their own, so a database crash losing the ones @@ -60,6 +81,20 @@ pub static GROUP: crate::SchemaGroup = crate::SchemaGroup { channel TEXT NOT NULL, payload TEXT NOT NULL )"#, + // The news a write makes, kept until the install was told + // ([`tell_when_wanted`]). Logged: a run waiting on it would otherwise + // hear it only when its hold gives up. + // SYNC: the kinds <-> kind_named + r#"CREATE TABLE IF NOT EXISTS weft_news ( + seq BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + project_id UUID NOT NULL, + execution_id UUID NOT NULL, + kind TEXT NOT NULL CHECK (kind IN ('answer', 'cancel')) + )"#, + // What a write calls to make news, in the write's transaction. + r#"CREATE OR REPLACE FUNCTION weft_tell(p_project UUID, p_execution UUID, p_kind TEXT) RETURNS VOID AS $$ + INSERT INTO weft_news (project_id, execution_id, kind) VALUES (p_project, p_execution, p_kind) + $$ LANGUAGE sql"#, // What a trigger on a run's path calls instead of `pg_notify`. r#"CREATE OR REPLACE FUNCTION weft_announce(p_channel TEXT, p_payload TEXT) RETURNS VOID AS $$ INSERT INTO weft_announcement (channel, payload) VALUES (p_channel, p_payload) @@ -96,12 +131,38 @@ pub fn committed(pool: &PgPool) { flusher(pool).wanted.notify_one(); } +/// Tell this process's teller for `pool`'s database that a write which +/// left news (`weft_tell`: a cancel asked of a run a worker drives, an +/// answer handed to one) just committed. A cancel and an answer reaching +/// its run call it, an answer written into a paused run's record included +/// (that one leaves no news, and costs one empty look); the run path's +/// writes never do. +pub fn news_written(pool: &PgPool) { + flusher(pool).news_wanted.notify_one(); +} + +/// From now on, this process tells the install's news that writes on +/// `pool`'s database leave, through `news`, starting with whatever news a +/// process that stopped left waiting: beside the announcements, never +/// holding them up. +pub fn tell_news_through(pool: &PgPool, news: Arc) { + let flusher = flusher(pool); + *flusher.news.lock().expect("announcement flusher news") = Some(news); + flusher.news_wanted.notify_one(); +} + /// One database's flusher in this process, and the pool it sends through: /// the last one a write poked with that is still open, so a pool closing -/// (a test's, at its end) never leaves the database without one. +/// (a test's, at its end) never leaves the database without one. Its task +/// sends the announcements and tells the news, side by side. struct Flusher { wanted: tokio::sync::Notify, pool: Mutex, + /// Told when news may be waiting ([`news_written`], [`tell_news_through`]). + news_wanted: tokio::sync::Notify, + /// What tells the install's news, once the process gave it + /// ([`tell_news_through`]); until then news waits in its table. + news: Mutex>>, } /// The flushers of this process, by database ([`database_key`]). An entry @@ -131,7 +192,12 @@ fn flusher(pool: &PgPool) -> Arc { } return flusher.clone(); } - let flusher = Arc::new(Flusher { wanted: tokio::sync::Notify::new(), pool: Mutex::new(pool.clone()) }); + let flusher = Arc::new(Flusher { + wanted: tokio::sync::Notify::new(), + pool: Mutex::new(pool.clone()), + news_wanted: tokio::sync::Notify::new(), + news: Mutex::new(None), + }); tokio::spawn(flush_when_wanted(key.clone(), flusher.clone())); flushers.insert(key, flusher.clone()); flusher @@ -156,14 +222,18 @@ impl Drop for Stopped { /// Flush at once (what a process that died between its commit and its /// flush left waiting goes out with the first write after it), then each -/// time a write asks. +/// time a write asks; and, side by side in the same task (so neither waits +/// on the other, and both live and go with the flusher), tell the news. async fn flush_when_wanted(key: String, flusher: Arc) { let _stopped = Stopped { key, flusher: flusher.clone() }; - loop { - let pool = flusher.pool.lock().expect("announcement flusher pool").clone(); - flush_waiting(&pool).await; - flusher.wanted.notified().await; - } + let announcements = async { + loop { + let pool = flusher.pool.lock().expect("announcement flusher pool").clone(); + flush_waiting(&pool).await; + flusher.wanted.notified().await; + } + }; + tokio::join!(announcements, tell_when_wanted(&flusher)); } /// How long a flush that failed waits before it is tried again, at first @@ -200,3 +270,147 @@ async fn flush(pool: &PgPool) -> Result { let taken: i32 = sqlx::query_scalar("SELECT weft_flush_announcements($1)").bind(BATCH).fetch_one(pool).await?; Ok(taken.max(0) as usize) } + +/// Tell the news waiting in `flusher`'s database, a batch at a time, each +/// time some may be waiting, once the process gave it its news. A failure +/// to read or take out waits a growing pause (or, the pool gone, the next +/// wake, which brings a pool that is open). +async fn tell_when_wanted(flusher: &Flusher) { + let mut retry = RETRY_FIRST; + loop { + let news = flusher.news.lock().expect("announcement flusher news").clone(); + let Some(news) = news else { + flusher.news_wanted.notified().await; + continue; + }; + let pool = flusher.pool.lock().expect("announcement flusher pool").clone(); + let waiting = match waiting_news(&pool).await { + Ok(waiting) => waiting, + Err(_) if pool.is_closed() => { + flusher.news_wanted.notified().await; + continue; + } + Err(e) => { + tracing::warn!(target: "weft_task_store::announce", error = %format!("{e:#}"), "could not read the news waiting; trying again"); + tokio::time::sleep(retry).await; + retry = (retry * 2).min(RETRY_LONGEST); + continue; + } + }; + retry = RETRY_FIRST; + if waiting.is_empty() { + flusher.news_wanted.notified().await; + continue; + } + let batch: Vec = waiting.iter().filter_map(|(_, news)| *news).collect(); + if !batch.is_empty() { + tell_until_told(news.as_ref(), &batch).await; + } + let seqs: Vec = waiting.iter().map(|(seq, _)| *seq).collect(); + while let Err(e) = sqlx::query("DELETE FROM weft_news WHERE seq = ANY($1)").bind(&seqs).execute(&pool).await { + if pool.is_closed() { + break; + } + tracing::warn!(target: "weft_task_store::announce", error = %format!("{e:#}"), "could not take the news told out of its table; trying again (it would only be told twice)"); + tokio::time::sleep(retry).await; + retry = (retry * 2).min(RETRY_LONGEST); + } + retry = RETRY_FIRST; + } +} + +/// The oldest [`NEWS_BATCH`] pieces of news waiting, each with the news it +/// is (`None` for one of a kind nobody tells, which goes out of the table +/// unsaid, loudly). Read with no lock: two processes telling the same +/// piece only ring the same doorbell twice. +async fn waiting_news(pool: &PgPool) -> Result)>, sqlx::Error> { + let rows: Vec<(i64, uuid::Uuid, uuid::Uuid, String)> = + sqlx::query_as("SELECT seq, project_id, execution_id, kind FROM weft_news ORDER BY seq LIMIT $1").bind(NEWS_BATCH).fetch_all(pool).await?; + Ok(rows + .into_iter() + .map(|(seq, project, execution, kind)| { + let news = kind_named(&kind).map(|kind| RunNews { project, execution, kind }); + if news.is_none() { + tracing::error!(target: "weft_task_store::announce", seq, %kind, "news of a kind nobody tells; dropped"); + } + (seq, news) + }) + .collect()) +} + +/// Announce `batch` in one call until it goes out: a refusal is logged +/// loudly and the same batch is announced again, after a growing pause. +/// Nothing here touches the database. +async fn tell_until_told(news: &dyn News, batch: &[RunNews]) { + let mut retry = RETRY_FIRST; + while let Err(e) = news.announce(batch).await { + tracing::error!( + target: "weft_task_store::announce", + error = %format!("{e:#}"), pieces = batch.len(), retry_in_ms = retry.as_millis() as u64, + "could not tell the install's news; the runs it is for hear of it late; trying again" + ); + tokio::time::sleep(retry).await; + retry = (retry * 2).min(RETRY_LONGEST); + } +} + +/// The kind a row of [`NEWS_TABLE`] names. +// SYNC: kind_named <-> the kinds in weft_news (GROUP), crates/weft-task-store/src/parked_fires.rs (parked_fire_notify), crates/weft-dispatcher/src/journal/postgres.rs (cancel_execution) +fn kind_named(kind: &str) -> Option { + match kind { + "answer" => Some(NewsKind::Answer), + "cancel" => Some(NewsKind::Cancel), + _ => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::{AtomicUsize, Ordering}; + + /// Refuses the first `fails` calls, then records each batch. + struct Flaky { + fails: AtomicUsize, + told: Mutex>>, + } + + #[async_trait::async_trait] + impl News for Flaky { + async fn announce(&self, news: &[RunNews]) -> anyhow::Result<()> { + if self.fails.load(Ordering::SeqCst) > 0 { + self.fails.fetch_sub(1, Ordering::SeqCst); + anyhow::bail!("the bus is away"); + } + self.told.lock().unwrap().push(news.to_vec()); + Ok(()) + } + fn listen(&self) -> weft_platform_traits::Listening { + unreachable!() + } + async fn close(&self) -> anyhow::Result<()> { + Ok(()) + } + } + + /// A batch refused is announced again whole, in one call, until it + /// goes out, after a growing pause. + #[tokio::test(start_paused = true)] + async fn a_refused_batch_is_announced_again_whole_until_it_goes_out() { + let bus = Flaky { fails: AtomicUsize::new(3), told: Mutex::default() }; + let batch: Vec = (1..=3) + .map(|n| RunNews { project: uuid::Uuid::from_u128(1), execution: uuid::Uuid::from_u128(n), kind: NewsKind::Answer }) + .collect(); + let started = tokio::time::Instant::now(); + tell_until_told(&bus, &batch).await; + assert_eq!(*bus.told.lock().unwrap(), vec![batch], "the whole batch, in one call"); + assert_eq!(started.elapsed(), RETRY_FIRST * 7, "paused 1, 2 and 4 times the first pause"); + } + + #[test] + fn every_kind_a_write_makes_is_told() { + assert_eq!(super::kind_named("answer"), Some(weft_platform_traits::NewsKind::Answer)); + assert_eq!(super::kind_named("cancel"), Some(weft_platform_traits::NewsKind::Cancel)); + assert_eq!(super::kind_named("other"), None); + } +} diff --git a/crates/weft-task-store/src/parked_fires.rs b/crates/weft-task-store/src/parked_fires.rs index b3f7eb30..7b7d31e5 100644 --- a/crates/weft-task-store/src/parked_fires.rs +++ b/crates/weft-task-store/src/parked_fires.rs @@ -11,9 +11,11 @@ use serde::{Deserialize, Serialize}; use serde_json::Value; /// The channel an event parked for a trigger is announced on, with its -/// project's id: the dispatcher's drain wakes on it, and so does a worker -/// holding for the answers of a run it drives. -// SYNC: PARKED_FIRE_CHANNEL <-> 'weft_parked_fire' in parked_fire_notify (GROUP below) +/// project's id, and an answer handed to a run a worker drives, with that +/// run (`crate::runs::run_payload`): the dispatcher's drain wakes on it, +/// and the broker pushes the second down the line of the run's worker, +/// which then reads the run's answers. +// SYNC: PARKED_FIRE_CHANNEL <-> 'weft_parked_fire' in parked_fire_notify (GROUP below), weft_broker_client::line::LINE_CHANNELS, crates/weft-broker/src/line.rs (audience) pub const PARKED_FIRE_CHANNEL: &str = "weft_parked_fire"; /// The events waiting for their trigger, one row each, first in first out @@ -67,12 +69,25 @@ pub static GROUP: crate::SchemaGroup = crate::SchemaGroup { // and a run's worker when an answer is handed to it. Through the // outbox (`crate::announce`): these are writes on a run's path, and // their writers poke the flusher once they commit. + // An event names its trigger's project; an answer handed to a run + // names the run, the way every announcement does + // (`crate::runs::run_payload`), so whoever holds the run hears it is + // for them, and it is the install's news too (`crate::announce`), for + // the copy holding the line of the run's worker. // SYNC: 'weft_parked_fire' <-> PARKED_FIRE_CHANNEL + // SYNC: the handed answer's payload <-> crate::runs::run_payload + // SYNC: 'answer' <-> crate::announce::kind_named r#"CREATE OR REPLACE FUNCTION parked_fire_notify() RETURNS trigger AS $$ + DECLARE + v_project UUID; BEGIN - PERFORM weft_announce('weft_parked_fire', COALESCE( - (SELECT s.project_id FROM signal s WHERE s.token = NEW.token), - (SELECT r.project_id FROM run r WHERE r.execution_id = NEW.execution_id))::text); + IF NEW.execution_id IS NULL THEN + PERFORM weft_announce('weft_parked_fire', (SELECT s.project_id FROM signal s WHERE s.token = NEW.token)::text); + ELSE + SELECT r.project_id INTO v_project FROM run r WHERE r.execution_id = NEW.execution_id; + PERFORM weft_announce('weft_parked_fire', v_project::text || ' ' || NEW.execution_id::text); + PERFORM weft_tell(v_project, NEW.execution_id, 'answer'); + END IF; RETURN NULL; END; $$ LANGUAGE plpgsql"#, diff --git a/crates/weft-task-store/src/pg_signal.rs b/crates/weft-task-store/src/pg_signal.rs index 04419a63..2b572fdd 100644 --- a/crates/weft-task-store/src/pg_signal.rs +++ b/crates/weft-task-store/src/pg_signal.rs @@ -47,12 +47,6 @@ use anyhow::Result; use sqlx::postgres::{PgConnectOptions, PgListener, PgPool, PgPoolOptions}; use tokio::sync::broadcast; -/// The longest any request is held open waiting on a signal. A client -/// that wants to wait longer asks again; a hold this short never meets -/// an idle timeout between a process and the broker, and a lost signal -/// costs a waiter at most this long. -pub const MAX_HOLD: Duration = Duration::from_secs(25); - /// What a waiter hears: one notification, or "every waiter should look /// at its row again" because notifications may have been lost. #[derive(Debug, Clone, PartialEq, Eq)] @@ -145,6 +139,71 @@ impl Drop for Busy { } } +/// The watch a request holds busy while it is answered ([`keep_listening`]), +/// in the request's extensions. A handler that goes on to wait only on what +/// reaches the process another way (a worker's answer relayed to its +/// caller, news of a run heard from the install's news) lets go of it, so a +/// wait of hours leaves the database free to sleep; it holds +/// [`PgSignalWatch::busy`] itself around whatever it reads or writes. +#[cfg(feature = "serve")] +#[derive(Clone)] +pub struct RequestListening(Arc>>); + +#[cfg(feature = "serve")] +impl RequestListening { + /// The request no longer needs the watch. + pub fn let_go(&self) { + self.0.lock().expect("a request's listening").take(); + } +} + +/// Keep `watch` listening while a request is answered, to the end of its +/// answer's body: a stream of events is listened for until it closes. A +/// process that scales to zero lets its watch go quiet only between +/// requests ([`Listening::WhileBusy`]), and every request it answers may +/// write, so a quiet watch listens again before the request goes on. A +/// handler that let go ([`RequestListening`]) holds nothing. +#[cfg(feature = "serve")] +pub async fn keep_listening( + axum::extract::State(watch): axum::extract::State>, + mut request: axum::extract::Request, + next: axum::middleware::Next, +) -> axum::response::Response { + let listening = RequestListening(Arc::new(Mutex::new(Some(watch.busy().await)))); + request.extensions_mut().insert(listening.clone()); + let answer = next.run(request).await; + let Some(busy) = listening.0.lock().expect("a request's listening").take() else { return answer }; + answer.map(|body| axum::body::Body::new(Held { body, _busy: busy })) +} + +/// An answer's body, holding its process busy until it ends. +#[cfg(feature = "serve")] +struct Held { + body: axum::body::Body, + _busy: Busy, +} + +#[cfg(feature = "serve")] +impl http_body::Body for Held { + type Data = axum::body::Bytes; + type Error = axum::Error; + + fn poll_frame( + mut self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + ) -> std::task::Poll, Self::Error>>> { + std::pin::Pin::new(&mut self.body).poll_frame(cx) + } + + fn is_end_stream(&self) -> bool { + self.body.is_end_stream() + } + + fn size_hint(&self) -> http_body::SizeHint { + self.body.size_hint() + } +} + /// The channel a watch proves it can hear on, every time it listens. const PROBE_CHANNEL: &str = "weft_listen_probe"; @@ -169,6 +228,9 @@ pub struct PgSignalWatch { /// either reads the state here. listening: Arc, demand: Arc, + /// What [`AlsoTold`] sends through: weak, so the pump stays the one + /// sender that keeps waiters hearing. + told: broadcast::WeakSender, /// The listening task. It lives exactly as long as the watch: when /// the process part that owns the watch goes, so does its connection /// (a test's database cannot be dropped while it is held). @@ -196,6 +258,7 @@ impl PgSignalWatch { .await?; let listener = listen(&own, channels).await?; let (tx, template) = broadcast::channel(FANOUT_CAPACITY); + let told = tx.downgrade(); let listening_now = Arc::new(AtomicBool::new(true)); let demand = Arc::new(Demand { listening, @@ -206,7 +269,7 @@ impl PgSignalWatch { listening_again: Notify::new(), }); let pump = tokio::spawn(pump(listener, own, channels, tx, listening_now.clone(), demand.clone())); - Ok(Arc::new(Self { channels, template: Mutex::new(template), listening: listening_now, demand, pump })) + Ok(Arc::new(Self { channels, template: Mutex::new(template), listening: listening_now, demand, told, pump })) } /// Hold this while doing work that writes, or that waits on what this @@ -263,6 +326,12 @@ impl PgSignalWatch { held } + /// A way to tell every waiter what this process heard of the database's + /// writes another way ([`AlsoTold`]). + pub fn also_told(&self) -> AlsoTold { + AlsoTold { channels: self.channels, told: self.told.clone() } + } + /// Subscribe BEFORE reading the row: a change that lands between the /// read and the subscription would otherwise go unheard. pub fn subscribe(&self) -> Subscription { @@ -286,6 +355,46 @@ impl PgSignalWatch { } } +/// What a process heard of the database's writes without its connection +/// (the install's news, which carries a few announcements while the +/// database sleeps and the watch is quiet), told to every waiter as the +/// watch would have told it. A waiter cannot tell the two apart, and hearing +/// one twice only costs it a look. +pub struct AlsoTold { + channels: &'static [&'static str], + told: broadcast::WeakSender, +} + +impl AlsoTold { + /// Tell to waiters through `told` on `channels` (a test drives waiters + /// with it, as [`Subscription::from`] does). + pub fn over(told: &broadcast::Sender, channels: &'static [&'static str]) -> Self { + Self { channels, told: told.downgrade() } + } + + /// `payload` was announced on `channel`. Refused on a channel the + /// watch does not listen on (no waiter would be waiting on it); `false` + /// once the watch stopped. + pub fn signal(&self, channel: &str, payload: &str) -> Result { + let Some(channel) = self.channels.iter().find(|c| **c == channel) else { + anyhow::bail!("this process's signal watch does not listen on '{channel}' (it listens on {:?})", self.channels); + }; + Ok(self.tell(Heard::Signal { channel, payload: payload.into() })) + } + + /// Announcements may have been missed: every waiter looks again. + pub fn recheck(&self) -> bool { + self.tell(Heard::Recheck) + } + + fn tell(&self, heard: Heard) -> bool { + let Some(told) = self.told.upgrade() else { return false }; + // No waiter right now is fine: nobody waits on it. + let _ = told.send(heard); + true + } +} + /// One waiter's view of the watch. pub struct Subscription { rx: broadcast::Receiver, @@ -532,6 +641,24 @@ mod tests { assert!(!wakes(&signal("b", "x"), &concerns)); } + /// What a process heard another way reaches every waiter as the watch + /// would have told it; a channel nobody listens on is refused, and a + /// watch that stopped hears nothing more. + #[tokio::test] + async fn what_is_told_another_way_reaches_every_waiter() { + let (tx, rx) = broadcast::channel(8); + let mut subscription = Subscription::from(rx); + let told = AlsoTold::over(&tx, &["a"]); + assert!(told.signal("a", "x").unwrap()); + assert!(told.recheck()); + assert!(told.signal("b", "x").is_err(), "a channel this watch does not listen on"); + assert_eq!(subscription.next().await.unwrap(), signal("a", "x")); + assert_eq!(subscription.next().await.unwrap(), Heard::Recheck); + drop(tx); + assert!(!told.recheck(), "the watch stopped"); + assert!(subscription.next().await.is_err(), "and its waiters hear so: telling it another way keeps nothing alive"); + } + #[tokio::test] async fn clearing_forgets_what_was_heard_and_keeps_listening() { let (tx, rx) = broadcast::channel(8); diff --git a/crates/weft-task-store/src/runs.rs b/crates/weft-task-store/src/runs.rs index bc98c503..12ef1a35 100644 --- a/crates/weft-task-store/src/runs.rs +++ b/crates/weft-task-store/src/runs.rs @@ -44,33 +44,35 @@ pub async fn in_flight<'e, E: sqlx::PgExecutor<'e>>(executor: E, execution_id: w // SYNC: RUN_QUEUED_CHANNEL <-> weft_journal::record::notify_queued_in pub const RUN_QUEUED_CHANNEL: &str = "weft_run_queued"; -/// The channel a cancel of a running run is announced on, with -/// ` ` ([`cancel_payload`]), in the same -/// transaction that sets the run's `cancel_requested`: the broker pushes it -/// down the lines of the project's workers, and the one driving the run -/// fires its flag. A worker that may have missed one (its line came back, -/// it just claimed a run) reads `cancel_requested` of the runs it drives. +/// The channel a cancel of a running run is announced on, with its run +/// ([`run_payload`]), in the same transaction that sets the run's +/// `cancel_requested`: the broker pushes it down the lines of the project's +/// workers, and the one driving the run fires its flag. A worker that may +/// have missed one (its line came back, it just claimed a run) reads +/// `cancel_requested` of the runs it drives. // SYNC: CANCEL_CHANNEL <-> weft_broker_client::line::LINE_CHANNELS, crates/weft-broker/src/line.rs (audience) pub const CANCEL_CHANNEL: &str = "weft_cancel"; -/// A [`CANCEL_CHANNEL`] payload. -pub fn cancel_payload(project_id: uuid::Uuid, execution_id: weft_core::ExecutionId) -> String { +/// How an announcement names a run: ` `. +// SYNC: run_payload <-> parked_fire_notify in crates/weft-task-store/src/parked_fires.rs (GROUP) +pub fn run_payload(project_id: uuid::Uuid, execution_id: weft_core::ExecutionId) -> String { format!("{project_id} {execution_id}") } -/// The project and execution a [`CANCEL_CHANNEL`] payload names. -pub fn parse_cancel_payload(payload: &str) -> Option<(uuid::Uuid, weft_core::ExecutionId)> { +/// The project and execution a [`run_payload`] names. +pub fn parse_run_payload(payload: &str) -> Option<(uuid::Uuid, weft_core::ExecutionId)> { let (project, execution) = payload.split_once(' ')?; Some((project.parse().ok()?, execution.parse().ok()?)) } #[cfg(test)] mod tests { - /// A cancel's payload reads back as the project and run it named. + /// A run named in an announcement reads back as the project and run. #[test] - fn a_cancel_payload_reads_back() { + fn a_run_payload_reads_back() { let (project, run) = (uuid::Uuid::from_u128(1), uuid::Uuid::from_u128(2)); - assert_eq!(super::parse_cancel_payload(&super::cancel_payload(project, run)), Some((project, run))); - assert_eq!(super::parse_cancel_payload("not a cancel"), None); + assert_eq!(super::parse_run_payload(&super::run_payload(project, run)), Some((project, run))); + assert_eq!(super::parse_run_payload("not a run"), None); + assert_eq!(super::parse_run_payload(&project.to_string()), None, "a project alone names no run"); } } diff --git a/crates/weft-task-store/src/terminal.rs b/crates/weft-task-store/src/terminal.rs index 2ca7f75d..5cb547b1 100644 --- a/crates/weft-task-store/src/terminal.rs +++ b/crates/weft-task-store/src/terminal.rs @@ -63,7 +63,7 @@ where F: Fn() -> Fut, Fut: std::future::Future>>, { - let deadline = tokio::time::Instant::now() + timeout; + let deadline = weft_core::deadline::after(tokio::time::Instant::now(), timeout); // Subscribed before the first read, so a finish between the read and // the wait is still heard. let mut signals = signals.subscribe(); diff --git a/crates/weft-task-store/src/worker_door.rs b/crates/weft-task-store/src/worker_door.rs index e249ab98..24d2c933 100644 --- a/crates/weft-task-store/src/worker_door.rs +++ b/crates/weft-task-store/src/worker_door.rs @@ -4,15 +4,31 @@ //! every worker that has work under way, through the broker //! (`/v1/door/tick`). A worker with nothing under way stops ticking after //! one last tick that says so, and its lease lapses: it owns no run then, -//! and it ticks again before it takes one. A worker's live lease is half -//! of what says a run is being worked on ([`crate::in_flight_sql!`]), and -//! what it drives is what a drain waits for. +//! and it ticks again before it takes one. A worker whose runs only wait on +//! something outside them stops ticking too, after a last tick that vouches +//! for it until the earliest of those waits gives up ([`lease_until`]), so +//! its lease holds that long. A worker's live lease is half of what says a +//! run is being worked on ([`crate::in_flight_sql!`]), and what it drives +//! is what a drain waits for. /// How long a worker's lease (`worker_lease`) holds without a tick, at /// this install's pace: a few missed ticks, so a busy second never reads /// as a dead worker. A run its lapsed lease owned is lost. pub const WORKER_LEASE_SECS: i64 = 15; +/// Until when (unix seconds) a tick at `now` keeps a worker's lease: one +/// lease length (`lease_secs`, [`WORKER_LEASE_SECS`] at this install's +/// pace), or, for the last tick of a worker whose runs only wait on +/// something outside them, until `vouch_until` (the earliest moment one of +/// those waits gives up, and a lease length). A vouch is held to the +/// longest a wait can hold (`weft_core::run_settings::MAX_HOLD_SECS`), and +/// one shorter than a lease length changes nothing. +pub fn lease_until(now: i64, lease_secs: i64, vouch_until: Option) -> i64 { + let renewed = now.saturating_add(lease_secs); + let longest = renewed.saturating_add(i64::from(weft_core::run_settings::MAX_HOLD_SECS)); + vouch_until.map_or(renewed, |vouch| vouch.clamp(renewed, longest)) +} + /// The one rule for a worker being alive: its lease (`worker_lease` under /// the alias `$lease`) is ahead of the database's clock. Every reader /// (whether a run is driven, the copies a door counts with, a run seed the @@ -97,3 +113,19 @@ concat!( r#"CREATE OR REPLACE FUNCTION weft_door_tick( ], seed: &[], }; + +#[cfg(test)] +mod tests { + use super::*; + + /// A tick keeps the lease one length; a vouch keeps it until then, held + /// to the longest a wait can hold, and never shorter than a tick. + #[test] + fn a_vouch_keeps_the_lease_until_then_within_the_longest_hold() { + let longest = i64::from(weft_core::run_settings::MAX_HOLD_SECS); + assert_eq!(lease_until(1_000, 15, None), 1_015); + assert_eq!(lease_until(1_000, 15, Some(5_000)), 5_000); + assert_eq!(lease_until(1_000, 15, Some(1_001)), 1_015, "a vouch shorter than a lease changes nothing"); + assert_eq!(lease_until(1_000, 15, Some(i64::MAX)), 1_015 + longest, "held to the longest hold"); + } +} diff --git a/crates/weft-task-store/tests/news_outbox.rs b/crates/weft-task-store/tests/news_outbox.rs new file mode 100644 index 00000000..4dc2bbb7 --- /dev/null +++ b/crates/weft-task-store/tests/news_outbox.rs @@ -0,0 +1,146 @@ +//! Layer-3 tests for the install's news going out through the outbox, +//! against a REAL Postgres: news a write left is told in one call per +//! batch, taken out only once told, kept in memory and told again (without +//! reading the database) while it cannot be, and never holds up the +//! database's own announcements. +//! +//! Gated behind `db-tests` like the rest of this crate's database tests; +//! `scripts/run-db-tests.sh weft-task-store` runs them. +#![cfg(feature = "db-tests")] + +mod support; + +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use async_trait::async_trait; +use sqlx::PgPool; +use weft_platform_traits::{Listening, News, NewsHeard, NewsKind, RunNews}; + +use support::setup; + +/// Refuses the first `fails` calls, then records each batch announced. +struct Flaky { + fails: AtomicUsize, + told: Mutex>>, + heard: tokio::sync::broadcast::Sender, +} + +impl Flaky { + fn failing(fails: usize) -> Arc { + Arc::new(Self { fails: AtomicUsize::new(fails), told: Mutex::default(), heard: tokio::sync::broadcast::channel(1).0 }) + } + + fn told(&self) -> Vec> { + self.told.lock().unwrap().clone() + } +} + +#[async_trait] +impl News for Flaky { + async fn announce(&self, news: &[RunNews]) -> anyhow::Result<()> { + if self.fails.load(Ordering::SeqCst) > 0 { + self.fails.fetch_sub(1, Ordering::SeqCst); + anyhow::bail!("the bus is away"); + } + self.told.lock().unwrap().push(news.to_vec()); + Ok(()) + } + + fn listen(&self) -> Listening { + self.heard.subscribe().into() + } + + async fn close(&self) -> anyhow::Result<()> { + Ok(()) + } +} + +async fn tell(pool: &PgPool, execution: u128) -> RunNews { + let news = RunNews { project: uuid::Uuid::from_u128(1), execution: uuid::Uuid::from_u128(execution), kind: NewsKind::Cancel }; + sqlx::query("SELECT weft_tell($1, $2, 'cancel')").bind(news.project).bind(news.execution).execute(pool).await.unwrap(); + news +} + +async fn waiting(pool: &PgPool) -> i64 { + sqlx::query_scalar("SELECT COUNT(*) FROM weft_news").fetch_one(pool).await.unwrap() +} + +async fn until(what: &str, holds: impl AsyncFn() -> bool) { + tokio::time::timeout(Duration::from_secs(20), async { + while !holds().await { + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await + .unwrap_or_else(|_| panic!("{what}")); +} + +/// News waiting goes out in one call per batch, once, and is then taken +/// out of the outbox. +#[sqlx::test] +async fn waiting_news_goes_out_in_one_call_and_once(pool: PgPool) { + setup(&pool).await; + let batch = vec![tell(&pool, 2).await, tell(&pool, 3).await, tell(&pool, 4).await]; + let bus = Flaky::failing(0); + weft_task_store::announce::tell_news_through(&pool, bus.clone()); + until("taken out once told", async || waiting(&pool).await == 0).await; + weft_task_store::announce::committed(&pool); + tokio::time::sleep(Duration::from_millis(300)).await; + assert_eq!(bus.told(), vec![batch], "one call, once"); +} + +/// A batch refused is told again as it was, from memory: news written +/// meanwhile is not read into it, and goes out after it. +#[sqlx::test] +async fn a_refused_batch_is_told_again_without_reading_the_database(pool: PgPool) { + setup(&pool).await; + let first = tell(&pool, 2).await; + let bus = Flaky::failing(2); + weft_task_store::announce::tell_news_through(&pool, bus.clone()); + until("the first batch was tried", async || bus.fails.load(Ordering::SeqCst) < 2).await; + let second = tell(&pool, 3).await; + weft_task_store::announce::news_written(&pool); + until("both told", async || bus.told().len() == 2).await; + assert_eq!(bus.told(), vec![vec![first], vec![second]], "the refused batch went out as it was read"); + until("taken out once told", async || waiting(&pool).await == 0).await; +} + +/// A commit that left no news costs no look at the news: only a write that +/// left some says so, and that wakes the teller. +#[sqlx::test] +async fn only_a_write_that_left_news_wakes_the_teller(pool: PgPool) { + setup(&pool).await; + let bus = Flaky::failing(0); + weft_task_store::announce::tell_news_through(&pool, bus.clone()); + // The teller's first look, for news a stopped process left, finds none. + tokio::time::sleep(Duration::from_millis(300)).await; + let news = tell(&pool, 2).await; + weft_task_store::announce::committed(&pool); + tokio::time::sleep(Duration::from_millis(500)).await; + assert!(bus.told().is_empty() && waiting(&pool).await == 1, "a commit alone looked at no news"); + weft_task_store::announce::news_written(&pool); + until("told once its write said so", async || bus.told() == vec![vec![news]]).await; +} + +/// A bus that refuses everything holds up none of the database's own +/// announcements: they go out at once, while the news waits. +#[sqlx::test] +async fn a_bus_that_refuses_holds_up_no_announcement(pool: PgPool) { + setup(&pool).await; + tell(&pool, 2).await; + let bus = Flaky::failing(usize::MAX); + weft_task_store::announce::tell_news_through(&pool, bus.clone()); + until("the bus refused", async || bus.fails.load(Ordering::SeqCst) < usize::MAX).await; + let watch = support::signals(&pool).await; + let mut heard = watch.subscribe(); + sqlx::query("SELECT weft_announce($1, 'x')").bind(weft_task_store::tasks::TASK_READY_CHANNEL).execute(&pool).await.unwrap(); + weft_task_store::announce::committed(&pool); + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + assert!( + heard.woken_before(deadline, |channel, payload| channel == weft_task_store::tasks::TASK_READY_CHANNEL && payload == "x").await.unwrap(), + "the announcement went out while the news waits" + ); + assert_eq!(waiting(&pool).await, 1, "the news still waits"); +} diff --git a/crates/weft-task-store/tests/signals.rs b/crates/weft-task-store/tests/signals.rs index 4d7f6483..79cd88ca 100644 --- a/crates/weft-task-store/tests/signals.rs +++ b/crates/weft-task-store/tests/signals.rs @@ -193,3 +193,74 @@ async fn an_announcement_goes_out_once_its_write_commits(pool: PgPool) { let left: i64 = sqlx::query_scalar("SELECT count(*) FROM weft_announcement").fetch_one(&pool).await.unwrap(); assert_eq!(left, 0, "the outbox is empty once sent"); } + +/// What a process hears another way (the install's news) reaches its +/// waiters while its watch is quiet and the database asleep, without the +/// watch listening again. +#[sqlx::test] +async fn what_is_told_another_way_reaches_waiters_while_the_watch_is_quiet(pool: PgPool) { + setup(&pool).await; + let watch = weft_task_store::pg_signal::PgSignalWatch::start( + &pool.connect_options(), + support::CHANNELS, + weft_task_store::pg_signal::Listening::WhileBusy, + ) + .await + .unwrap(); + let mut heard = watch.subscribe(); + tokio::time::sleep(Duration::from_secs(6)).await; + cut_the_listener(&pool).await; + let next = tokio::time::timeout(Duration::from_secs(20), heard.next()).await.expect("heard in time"); + assert_eq!(next.unwrap(), Heard::Lost); + let told = watch.also_told(); + assert!(told.signal(TASK_READY_CHANNEL, "x").unwrap()); + assert!(told.recheck()); + let next = tokio::time::timeout(Duration::from_secs(5), heard.next()).await.expect("heard in time"); + assert_eq!(next.unwrap(), Heard::Signal { channel: TASK_READY_CHANNEL, payload: "x".into() }); + let next = tokio::time::timeout(Duration::from_secs(5), heard.next()).await.expect("heard in time"); + assert_eq!(next.unwrap(), Heard::Recheck); + assert!(!heard.listening(), "the watch stayed quiet"); +} + +/// A request holds its process's watch busy while it is answered, unless +/// its handler let go of it: a call that then waits for hours holds +/// nothing that would keep the database awake. +#[sqlx::test] +async fn a_request_that_let_go_of_its_listening_holds_the_watch_no_longer(pool: PgPool) { + use std::sync::Arc; + use axum::extract::Extension; + use axum::routing::get; + use tower::ServiceExt as _; + use weft_task_store::pg_signal::{keep_listening, PgSignalWatch, RequestListening}; + setup(&pool).await; + let watch = PgSignalWatch::start(&pool.connect_options(), support::CHANNELS, weft_task_store::pg_signal::Listening::WhileBusy).await.unwrap(); + let (waiting, release) = (Arc::new(tokio::sync::Notify::new()), Arc::new(tokio::sync::Notify::new())); + let handler = |lets_go: bool| { + let (waiting, release) = (waiting.clone(), release.clone()); + move |Extension(listening): Extension| { + let (waiting, release) = (waiting.clone(), release.clone()); + async move { + if lets_go { + listening.let_go(); + } + waiting.notify_one(); + release.notified().await; + "done" + } + } + }; + let app = axum::Router::new() + .route("/holds", get(handler(false))) + .route("/lets-go", get(handler(true))) + .layer(axum::middleware::from_fn_with_state(watch.clone(), keep_listening)); + for (path, holds) in [("/holds", true), ("/lets-go", false)] { + let request = axum::http::Request::get(path).body(axum::body::Body::empty()).unwrap(); + let call = tokio::spawn(app.clone().oneshot(request)); + waiting.notified().await; + let now = tokio::time::Instant::now(); + assert_eq!(watch.worked_since(now), holds, "{path}"); + release.notify_one(); + let answer = call.await.unwrap().unwrap(); + axum::body::to_bytes(answer.into_body(), usize::MAX).await.unwrap(); + } +} diff --git a/deploy/terraform/gcp/news.tf b/deploy/terraform/gcp/news.tf new file mode 100644 index 00000000..f8873fe5 --- /dev/null +++ b/deploy/terraform/gcp/news.tf @@ -0,0 +1,55 @@ +# What happens to a run a worker drives (an answer handed to it, a cancel), +# told across weft's copies without the database, so the database can sleep +# while a run only waits (crates/weft-platform-gcp/src/news.rs). The copies +# that write (the dispatcher's and the broker's) announce on the topic what +# their writes left in the outbox; every broker copy makes a subscription of +# its own to it while it runs, pulls from it, and deletes it as it stops. +# Both roles run as the core account. + +resource "google_pubsub_topic" "news" { + name = "${var.name}-news" + depends_on = [google_project_service.apis] +} + +# The copies that write announce. +resource "google_pubsub_topic_iam_member" "core_announces_news" { + topic = google_pubsub_topic.news.id + role = "roles/pubsub.publisher" + member = local.core +} + +# A broker copy attaches its subscription to this topic, and to no other. +resource "google_project_iam_custom_role" "news_attach" { + role_id = "${replace(var.name, "-", "_")}_news_attach" + title = "weft news: attach a subscription" + permissions = ["pubsub.topics.attachSubscription"] +} + +resource "google_pubsub_topic_iam_member" "core_attaches_to_news" { + topic = google_pubsub_topic.news.id + role = google_project_iam_custom_role.news_attach.id + member = local.core +} + +# Making, pulling from and deleting the copy's own subscription. Its name +# is made while it runs, so these rights cannot sit on the subscription +# itself. An IAM Condition cannot match Pub/Sub subscriptions by name +# either: of Pub/Sub, only Pub/Sub Lite is among the services whose +# resources IAM Conditions' resource attributes cover. So these three are +# granted on the project, and the core account may also consume and delete +# the install's other subscriptions there (the supervisor's push one). +resource "google_project_iam_custom_role" "news_subscriptions" { + role_id = "${replace(var.name, "-", "_")}_news_subscriptions" + title = "weft news subscriptions" + permissions = [ + "pubsub.subscriptions.create", + "pubsub.subscriptions.consume", + "pubsub.subscriptions.delete", + ] +} + +resource "google_project_iam_member" "core_news_subscriptions" { + project = var.project_id + role = google_project_iam_custom_role.news_subscriptions.id + member = local.core +} diff --git a/deploy/terraform/gcp/serverless.tf b/deploy/terraform/gcp/serverless.tf index 89fb2f86..b9b11e5d 100644 --- a/deploy/terraform/gcp/serverless.tf +++ b/deploy/terraform/gcp/serverless.tf @@ -33,6 +33,7 @@ locals { buildBucket = google_storage_bucket.builds.name builderServiceAccount = google_service_account.builder.email tasksQueue = google_cloud_tasks_queue.wakes.name + newsTopic = google_pubsub_topic.news.name coreServiceAccount = google_service_account.core.email holderPool = "projects/${var.project_id}/locations/${var.region}/workerPools/${local.holder_pool}" dispatcherService = "${var.name}-role-dispatcher" @@ -147,7 +148,10 @@ resource "google_cloud_run_v2_service" "role" { template { service_account = google_service_account.core.email - # A wake or an internal call may run as long as its work does. + # A wake or an internal call may run as long as its work does. The + # broker tells each line this limit, and a worker whose runs only wait + # stops before it. + # SYNC: timeout <-> crates/weft-platform-gcp/src/runner.rs (REQUEST_CAP) timeout = "3600s" scaling { diff --git a/deploy/terraform/gcp/versions.tf b/deploy/terraform/gcp/versions.tf index 8e721733..a988cb07 100644 --- a/deploy/terraform/gcp/versions.tf +++ b/deploy/terraform/gcp/versions.tf @@ -45,6 +45,7 @@ resource "google_project_service" "apis" { "iamcredentials.googleapis.com", # carrying infra machine events to the supervisor (machines.tf) "logging.googleapis.com", + # and those, and what happens to a run, across weft's copies (news.tf) "pubsub.googleapis.com", "run.googleapis.com", "secretmanager.googleapis.com", diff --git a/docs/src/running/architecture.md b/docs/src/running/architecture.md index 5ea76177..5e430b49 100644 --- a/docs/src/running/architecture.md +++ b/docs/src/running/architecture.md @@ -326,13 +326,23 @@ they finish. A call the broker holds until something happens (new journal rows) costs nothing while it waits, so a worker takes one seat at the broker however many calls its runs hold open, and a call whose caller stops waiting is dropped at the broker too. The broker also pushes down the same connection -the changes a worker keeps a copy of (see [the worker](#the-worker)), and the -cancel of a run the worker drives. A connection with nothing to do for a -minute closes, and opens again on the next call, so an idle process holds -nothing open at the broker, except a worker. A worker driving runs keeps its -connection open, to hear their cancels, and an idle one reports to the broker -every second (that it is alive, and how many runs it drives), so its -connection opens again within a second of closing. When the connection breaks, it comes back by +the changes a worker keeps a copy of (see [the worker](#the-worker)), and, +for a run the worker drives, its cancel and every answer handed to it. A +connection with nothing to do for a minute closes, and opens again on the +next call, so an idle process holds nothing open at the broker, except a +worker. A worker driving runs keeps its connection open, to hear their +cancels and answers, and an idle one reports to the broker every second +(that it is alive, and how many runs it drives), so its connection opens +again within a second of closing. On Google Cloud a connection is cut after +an hour, the most Cloud Run lets one request last, and the broker says so +when it opens. A busy worker lets it be cut and opens another, like any +broken connection. A worker whose runs all only wait never does, since +opening one makes it read again, which would wake the database: once its +connection is within a minute of the cut and its runs only wait, it stops +the way it does when Cloud Run stops it. The runs that can pause are +handed back for a fresh worker to carry on, the others end with a message +naming the limit, and the worker exits, so the next request starts a new +one. When the connection breaks, it comes back by itself: a call not yet sent goes out on the new one if it is back within the call's wait. Past that, or when a sent call got no answer, two kinds of call are made again because doing so is safe: a batch of records (a batch that @@ -378,10 +388,32 @@ A lost fast run is not marked cancelled the moment its worker dies: the dispatcher notices and ends it. Every worker reports that it is alive through the broker once a second, and counts as gone 15 seconds after its last report. The dispatcher ends a run once its worker has been gone for another 15 -seconds, and it checks every 15 seconds, so a fast run ends 30 to 45 seconds -after its worker last reported. Its journal holds what reached the database +seconds, and it looks for that at most once every 15 seconds, so a fast run +ends 30 to 45 seconds after its worker last reported. Its journal holds what reached the database before the worker died. +A worker whose runs are all only waiting on something outside them (a +person's answer, while the run cannot pause) stops reporting, so the +database is not woken every second for a wait that can last hours. Its last +report says it is alive until the first of those waits would give up (its +`holdSecs` running out), plus 15 seconds, and it reports again as soon as +one of its runs moves (that first wait giving up included) or a new run +comes, saying again how long it is alive for. The answer or the cancel that ends +the wait is written by the dispatcher, which also tells the broker copy +holding the worker's connection, without going through the database (on +Google Cloud, over Pub/Sub). That broker copy stays up for the length of the +wait, and pushes the answer or the cancel down the worker's connection, and +only then does the worker read it. Nothing else reads while a run waits: the +worker read its answers once as the run started waiting, and the broker +reads only for the calls a worker makes. So the database sleeps for as long +as the run waits, and wakes for the answer, the cancel, or the run's end, +or when the connection breaks, after which the worker reads once to catch +up. On Google Cloud such a wait lasts at most until the worker stops before +its connection is cut, within the hour (see [the +broker](#the-broker-and-who-may-talk-to-what)). If the worker dies during the wait, its +runs are ended (or carried on, if durable) 30 to 45 seconds after that +first wait would have given up. + In that same check, a durable run whose worker died is queued again, and the dispatcher hands it to another worker at once, which carries it on from its record. A step that had started but not finished is failed on the new worker diff --git a/docs/src/running/cloud.md b/docs/src/running/cloud.md index 3c00774a..57a03839 100644 --- a/docs/src/running/cloud.md +++ b/docs/src/running/cloud.md @@ -191,13 +191,28 @@ moment something is due (a schedule, a retry, a hibernation ending), and a program's infrastructure that runs fine keeps nothing awake: each machine says so itself when its state changes. +A run that waits on a person while it cannot pause (a route whose caller is +still on the line, an unrecorded run) keeps its worker and one copy of the +broker up for as long as it waits, which is cheap, and leaves your database +asleep for all of it. The answer or the cancel that ends the wait reaches +the broker over Pub/Sub, not through the database, and is pushed to the +worker on its connection. The database wakes when the answer or the cancel +arrives, or the run ends. If you want a wait longer than an hour, make the +run one that can pause: Cloud Run cuts a request after an hour, so when a +worker's runs are all only waiting and its connection is a minute from that +cut, the worker ends them with a message naming the limit, and stops. A +caller that reaches that worker in the few seconds it takes to stop is told +to try again. Cloud Run may send the retry to the same stopping worker for +those few seconds, which turns it away again; once the worker is gone, the +retry reaches a fresh one. + | Piece | What it is | |---|---| | Cloud Run | weft's dispatcher, broker, listener and supervisor, a service each, and your programs' workers, one service per program. Each scales on its own load, and to zero between calls. A service at zero starts again when it is called: by the part of weft that just wrote work for it, by a Cloud Tasks wake, or by a call from outside (your CLI, a route, a provider's webhook) | | The holders | a Cloud Run worker pool for the triggers that keep a connection open between events (a stream, a socket, an event subscription that dials out). weft runs one holder per 200 such triggers and none when there are none | | Your database | everything weft keeps, at the address you gave it | | Cloud Build | builds your programs' images when you deploy | -| Pub/Sub | a topic carrying Compute Engine's record of an infrastructure machine stopping or failing to the supervisor | +| Pub/Sub | a topic carrying Compute Engine's record of an infrastructure machine stopping or failing to the supervisor, and one weft announces on when a run a worker drives gets an answer or is cancelled, so word of it travels between weft's copies without your database | | Cloud Tasks | every timer, schedule and poll your programs set, and the wakes weft schedules for itself to check on its own pending work | | Compute Engine | one machine per infrastructure unit your programs start (a database, a GPU model) | | A storage bucket | weft's files, reached as weft's own service account: no key is made for it, so an organization that forbids service-account keys runs weft as is |