diff --git a/.claude/skills/releasing/SKILL.md b/.claude/skills/releasing/SKILL.md index 956ddc1a..f9cf7d93 100644 --- a/.claude/skills/releasing/SKILL.md +++ b/.claude/skills/releasing/SKILL.md @@ -43,12 +43,36 @@ versions bumped` (PRs only), `build`, `public proxy routes`, `cargo test + clippy`, `cargo test --features db-tests`, and `graph + editor`. -## Run the checks CI runs, not the ones you touched - -`cargo clippy --workspace --all-targets --locked -- -D warnings` is -the CI line. Running clippy on the crates you edited is not the same -thing and has already sent a PR red on a lint in a crate the change -only touched indirectly. +## Before every push to a PR, run what the PR runs + +Before you push a commit to a branch with a PR (the first push that +opens it, and every fix pushed after), run locally everything the PR's +checks run, not only the tests your change reaches. This is the one +place the [test scope] does not apply: a red PR costs a full CI round +trip, and twice a PR has gone red on a test in a crate the change only +touched indirectly. Background each one, and fix everything before the +push, not after. + +The PR runs `.github/workflows/ci.yml` and `.github/workflows/docs.yml`; +read them for the current list. At the time of writing: + +- `scripts/check-extension-bump.sh origin/main` +- `cargo nextest run --workspace --locked` +- `cargo test --workspace --locked --doc` +- `cargo clippy --workspace --all-targets --locked -- -D warnings` +- `cargo check -p weft-compiler --no-default-features --locked` +- `scripts/validate-examples.sh` +- `scripts/run-db-tests.sh` (every crate, no filter) +- `pnpm -C packages/weft-graph test`, `pnpm -C packages/weft-connect test`, + `pnpm -C packages/weft-syntax test`, `pnpm -C extension-vscode test`, + `pnpm -C extension-vscode run compile` +- `node --test catalog/bailey/bridge/images/bridge/src/*.test.js` +- `pnpm -C extension-browser exec tsc --noEmit`, `pnpm -C extension-browser check` +- the docs build: `cd docs && mdbook-mermaid install . && mdbook build` + +Clippy on the crates you edited is not the same as the workspace line +above, and has sent a PR red on a lint in a crate the change only +touched indirectly. `scripts/validate-examples.sh` compiles every example. It fails on ERRORS only, so a warning is advice an example may carry. If an diff --git a/catalog/instances/deactivate_instance_triggers/metadata.json b/catalog/instances/deactivate_instance_triggers/metadata.json index 3544ff6d..db3d18f2 100644 --- a/catalog/instances/deactivate_instance_triggers/metadata.json +++ b/catalog/instances/deactivate_instance_triggers/metadata.json @@ -34,7 +34,7 @@ }, "default": "park", "label": "Triggers", - "description": "park queues what arrives, hibernate keeps accepting for `graceMinutes`, wipe drops it." + "description": "Park keeps everything that arrives (calls, answers, the trigger's own fires) and runs it once the trigger is back, a caller on a route held on the line; hibernate does the same for `graceMinutes` with its questions hidden, then takes nothing more; wipe drops it all." }, { "name": "running", diff --git a/catalog/instances/lifecycle.rs b/catalog/instances/lifecycle.rs index 9d5326f4..b59e5fe3 100644 --- a/catalog/instances/lifecycle.rs +++ b/catalog/instances/lifecycle.rs @@ -4,6 +4,8 @@ //! choices with the same defaults. use weft::instance::InstanceId; +use weft::node::NodeOutput; +use weft::program::InfraDownAnswer; use weft::{node_bail, DeactivateSpec, DeactivationMode, ExecutionContext, RunningPolicy, StopSelf, WeftResult}; /// The instance id a node was handed, checked the way every instance id @@ -46,3 +48,14 @@ pub fn take_down_spec(ctx: &ExecutionContext) -> WeftResult { pub fn stop_self(ctx: &ExecutionContext) -> WeftResult { Ok(if ctx.inputs.get::("includeSelf")? { StopSelf::Include } else { StopSelf::Keep }) } + +/// What a node that takes one instance's copy down sends on: `done` when +/// the copy is down (taken down now, or already down), `noCopy` when that +/// instance has no copy of the node (a mistyped id, or one terminated +/// before), so a program answering a person can tell them which. +pub fn taken_down(answer: InfraDownAnswer) -> NodeOutput { + match answer { + InfraDownAnswer::TakenDown | InfraDownAnswer::AlreadyDown => NodeOutput::new().set("done", true), + InfraDownAnswer::NoCopy => NodeOutput::new().set("noCopy", true), + } +} diff --git a/catalog/instances/stop_instance_infra/metadata.json b/catalog/instances/stop_instance_infra/metadata.json index c8467e0f..c4994e46 100644 --- a/catalog/instances/stop_instance_infra/metadata.json +++ b/catalog/instances/stop_instance_infra/metadata.json @@ -35,7 +35,7 @@ }, "default": "park", "label": "Triggers", - "description": "The instance's triggers reading it: park queues what arrives, hibernate keeps accepting for `graceMinutes`, wipe drops it." + "description": "The instance's triggers reading it: park keeps everything that arrives (calls, answers, the trigger's own fires) and runs it once the trigger is back, a caller on a route held on the line; hibernate does the same for `graceMinutes` with its questions hidden, then takes nothing more; wipe drops it all. Starting the copy again brings back the ones it took down." }, { "name": "running", @@ -75,7 +75,12 @@ { "name": "done", "type": "Boolean", - "description": "True once the triggers reading the copy are down and the stop is handed off; the copy scales down afterwards." + "description": "True once the triggers reading the copy are down and the stop is handed off (or the copy was already down); the copy scales down afterwards." + }, + { + "name": "noCopy", + "type": "Boolean", + "description": "True instead of `done` when this instance has no copy of the node: the id names no instance that has one, or its copy was terminated before. Wire it to tell whoever asked." } ], "validate": [ diff --git a/catalog/instances/stop_instance_infra/mod.rs b/catalog/instances/stop_instance_infra/mod.rs index 065f7a5c..b3a9c6f4 100644 --- a/catalog/instances/stop_instance_infra/mod.rs +++ b/catalog/instances/stop_instance_infra/mod.rs @@ -4,10 +4,9 @@ use async_trait::async_trait; -use weft::node::NodeOutput; use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; -use super::lifecycle::{instance, stop_self, take_down_spec}; +use super::lifecycle::{instance, stop_self, take_down_spec, taken_down}; #[derive(NodeManifest)] pub struct StopInstanceInfraNode; @@ -25,7 +24,7 @@ impl Node for StopInstanceInfraNode { async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { let node: String = ctx.inputs.get("node")?; let spec = take_down_spec(&ctx)?; - ctx.infra(node).instance(instance(&ctx)?).stop(spec, stop_self(&ctx)?).await?; - ctx.pulse_downstream(NodeOutput::new().set("done", true)).await + let answer = ctx.infra(node).instance(instance(&ctx)?).stop(spec, stop_self(&ctx)?).await?; + ctx.pulse_downstream(taken_down(answer)).await } } diff --git a/catalog/instances/stop_instance_infra/tests.rs b/catalog/instances/stop_instance_infra/tests.rs index 315e404f..ec47a7b4 100644 --- a/catalog/instances/stop_instance_infra/tests.rs +++ b/catalog/instances/stop_instance_infra/tests.rs @@ -14,9 +14,27 @@ pub fn tests() -> Vec { NodeTest::fake("parks_and_waits_by_default", defaults), NodeTest::fake("the_choices_reach_the_call", choices), NodeTest::fake("waiting_before_a_wipe_is_refused", wipe_wait), + NodeTest::fake("no_copy_is_told_apart_from_done", no_copy), + NodeTest::fake("a_copy_already_down_is_done", already_down), ] } +async fn no_copy(rig: FakeRig) -> WeftResult<()> { + rig.answer_program_call("weft.infra.stop", json!({ "copy": "no_copy" })); + let out = rig.run(&StopInstanceInfraNode, json!({ "node": "bridge", "instance": "adaa" })).await.ok()?; + assert_eq!(out.output("noCopy")?, &json!(true)); + assert!(out.output("done").is_err(), "nothing was taken down"); + Ok(()) +} + +async fn already_down(rig: FakeRig) -> WeftResult<()> { + rig.answer_program_call("weft.infra.stop", json!({ "copy": "already_down" })); + let out = rig.run(&StopInstanceInfraNode, json!({ "node": "bridge", "instance": "ada" })).await.ok()?; + assert_eq!(out.output("done")?, &json!(true)); + assert!(out.output("noCopy").is_err()); + Ok(()) +} + async fn defaults(rig: FakeRig) -> WeftResult<()> { rig.run(&StopInstanceInfraNode, json!({ "node": "bridge", "instance": "ada" })).await.ok()?; match &rig.program_calls()[0] { diff --git a/catalog/instances/terminate_instance_infra/metadata.json b/catalog/instances/terminate_instance_infra/metadata.json index 8c4acf6e..032fc6aa 100644 --- a/catalog/instances/terminate_instance_infra/metadata.json +++ b/catalog/instances/terminate_instance_infra/metadata.json @@ -36,7 +36,7 @@ }, "default": "park", "label": "Triggers", - "description": "The instance's triggers reading it: park queues what arrives, hibernate keeps accepting for `graceMinutes`, wipe drops it." + "description": "The instance's triggers reading it: park keeps everything that arrives (calls, answers, the trigger's own fires) and runs it once the trigger is back, a caller on a route held on the line; hibernate does the same for `graceMinutes` with its questions hidden, then takes nothing more; wipe drops it all." }, { "name": "running", @@ -76,7 +76,12 @@ { "name": "done", "type": "Boolean", - "description": "True once the triggers reading the copy are down and the delete is handed off; the copy is deleted afterwards." + "description": "True once the triggers reading the copy are down and the delete is handed off (or the copy was already going); the copy is deleted afterwards." + }, + { + "name": "noCopy", + "type": "Boolean", + "description": "True instead of `done` when this instance has no copy of the node: the id names no instance that has one, or its copy was terminated before. Wire it to tell whoever asked." } ], "validate": [ diff --git a/catalog/instances/terminate_instance_infra/mod.rs b/catalog/instances/terminate_instance_infra/mod.rs index cf1d11df..2c956855 100644 --- a/catalog/instances/terminate_instance_infra/mod.rs +++ b/catalog/instances/terminate_instance_infra/mod.rs @@ -4,10 +4,9 @@ use async_trait::async_trait; -use weft::node::NodeOutput; use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; -use super::lifecycle::{instance, stop_self, take_down_spec}; +use super::lifecycle::{instance, stop_self, take_down_spec, taken_down}; #[derive(NodeManifest)] pub struct TerminateInstanceInfraNode; @@ -25,7 +24,7 @@ impl Node for TerminateInstanceInfraNode { async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { let node: String = ctx.inputs.get("node")?; let spec = take_down_spec(&ctx)?; - ctx.infra(node).instance(instance(&ctx)?).terminate(spec, stop_self(&ctx)?).await?; - ctx.pulse_downstream(NodeOutput::new().set("done", true)).await + let answer = ctx.infra(node).instance(instance(&ctx)?).terminate(spec, stop_self(&ctx)?).await?; + ctx.pulse_downstream(taken_down(answer)).await } } diff --git a/crates/weft-broker-client/src/activation.rs b/crates/weft-broker-client/src/activation.rs index 15e4b2fb..9d6584fe 100644 --- a/crates/weft-broker-client/src/activation.rs +++ b/crates/weft-broker-client/src/activation.rs @@ -29,10 +29,13 @@ pub struct ActivationLifecycle { /// second past which the drain gives up, cancels what is still running, /// and lands Inactive (enforced by the stuck-transition reaper). pub drain_deadline_unix: Option, - /// True iff the CURRENT deactivation was the health loop parking this - /// activation because the infra it reads broke. Its auto-recover - /// reactivates only such an activation, never one a person took down. - pub deactivated_by_health: bool, + /// Why the CURRENT deactivation happened, when it was the infra the + /// trigger reads going down rather than a person: the health loop + /// parking it because a copy broke, or an infra verb (stop, terminate, + /// upgrade) taking it down with its copies. The copy coming back + /// brings it back (the health loop's auto-recover, an infra start); + /// one a person took down stays down until they switch it on. + pub went_down: Option, /// The trigger-setup run of the activation in flight, reserved before /// setup is queued; every write the activation makes is guarded by it. /// `None` outside Activating. @@ -53,7 +56,7 @@ impl ActivationLifecycle { fires_visible_to_consumers: true, fires_deadline_unix: None, drain_deadline_unix: None, - deactivated_by_health: false, + went_down: None, activating_execution_id: None, } } @@ -80,8 +83,8 @@ impl ActivationLifecycle { } } - /// Hibernate: parking until the deadline, then refusing; hidden the - /// whole time. + /// Hibernate: parking until the deadline, then refusing (the dispatcher + /// clears `accepting_fires` once it passes); hidden the whole time. pub fn hibernating(deadline_unix: i64) -> Self { Self { status: ProjectStatus::Inactive, @@ -107,6 +110,20 @@ impl ActivationLifecycle { } } + /// Whether this is a wipe: inactive, refusing work, with no grace + /// window (a hibernation whose window ended refuses work too, and keeps + /// its deadline). + // SYNC: is_wiped <-> WIPE_FORGETS_SQL (crates/weft-dispatcher/src/activation_store.rs) + pub fn is_wiped(&self) -> bool { + self.status == ProjectStatus::Inactive && !self.accepting_fires && self.fires_deadline_unix.is_none() + } + + /// Whether the health loop took this down and it is still down: what + /// its auto-recover brings back. + pub fn parked_by_health(&self) -> bool { + self.went_down.is_some_and(|w| w.with == DownWith::Health) && self.status == ProjectStatus::Inactive + } + /// Where the activation stands, as a person reads it: `registered`, /// `activating`, `active`, `deactivating`, or for an inactive /// activation how it went down: `wipe`, `hibernate`, `park`. @@ -118,7 +135,7 @@ impl ActivationLifecycle { ProjectStatus::Activating => ActivationMode::Activating, ProjectStatus::Active => ActivationMode::Active, ProjectStatus::Deactivating => ActivationMode::Deactivating, - ProjectStatus::Inactive => ActivationMode::Down(if !self.accepting_fires { + ProjectStatus::Inactive => ActivationMode::Down(if self.is_wiped() { DeactivationMode::Wipe } else if !self.fires_visible_to_consumers { DeactivationMode::Hibernate @@ -151,7 +168,7 @@ pub fn aggregate<'a>(lifecycles: impl IntoIterator(lifecycles: impl IntoIterator Option { + let with = lifecycles.first()?.went_down?.with; + let mut at_unix = i64::MIN; + for lifecycle in lifecycles { + let went = lifecycle.went_down.filter(|w| w.with == with)?; + at_unix = at_unix.max(went.at_unix); + } + Some(WentDown { with, at_unix }) +} + +/// What took a trigger down when it was not a person, and when (see +/// [`ActivationLifecycle::went_down`]). The moment is what tells a copy +/// that came back up since from one still going down: only the first +/// brings the trigger back. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct WentDown { + pub with: DownWith, + pub at_unix: i64, +} + +/// What took a trigger down when it was not a person ([`WentDown`]). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DownWith { + /// The health loop parked it: a copy it reads broke. + Health, + /// An infra verb took it down with the copies it reads. + Infra, +} + +impl DownWith { + // SYNC: the spellings <-> trigger_activation.went_down_with (crates/weft-dispatcher/src/activation_store.rs) + pub fn as_str(self) -> &'static str { + match self { + DownWith::Health => "health", + DownWith::Infra => "infra", + } + } + + pub fn parse(s: &str) -> Option { + [DownWith::Health, DownWith::Infra].into_iter().find(|cause| cause.as_str() == s) + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/weft-broker-client/src/client.rs b/crates/weft-broker-client/src/client.rs index 317a1251..1fcee8f8 100644 --- a/crates/weft-broker-client/src/client.rs +++ b/crates/weft-broker-client/src/client.rs @@ -189,6 +189,17 @@ fn never_sent(e: &anyhow::Error) -> bool { e.chain().any(|cause| matches!(cause.downcast_ref::(), Some(LineError::NotSent { .. }))) } +/// Whether the broker answered that the call did nothing at all +/// ([`NOT_DONE`]), so sending it again repeats nothing. +// SYNC: NOT_DONE <-> crates/weft-broker/src/handlers.rs unavailable_or_internal +fn did_nothing(e: &anyhow::Error) -> bool { + e.chain().any(|cause| { + cause + .downcast_ref::() + .is_some_and(|refused| refused.status == reqwest::StatusCode::SERVICE_UNAVAILABLE && refused.body == NOT_DONE) + }) +} + /// The first wait before asking again after the broker could not answer /// a read; each further failure doubles it, up to /// [`READ_RETRY_LONGEST`]. @@ -240,6 +251,38 @@ where } } +/// How long a call the broker answered [`NOT_DONE`] is sent again for, at +/// first and at most between tries, and in all: the broker's database +/// connections were all busy, which passes in moments, and one busy this +/// long is a broker in trouble the caller hears about. +const NOT_DONE_FIRST: Duration = Duration::from_millis(50); +const NOT_DONE_LONGEST: Duration = Duration::from_secs(2); +const NOT_DONE_FOR: Duration = Duration::from_secs(20); + +/// Make a call, and send it again for as long as the broker answers that it +/// did nothing ([`did_nothing`]), within [`NOT_DONE_FOR`]: a claim or a +/// task's end the broker could not even start is safe to send again, and +/// failing it instead would fail a caller's run, or leave a finished task +/// claimed until its lease runs out and it runs again. +async fn until_done(path: &str, mut call: F) -> Result +where + F: FnMut() -> Fut, + Fut: std::future::Future>, +{ + let started = tokio::time::Instant::now(); + let mut wait = NOT_DONE_FIRST; + loop { + match call().await { + Err(e) if did_nothing(&e) && started.elapsed() < NOT_DONE_FOR => { + tracing::warn!(target: "weft_broker_client", path, retry_in_ms = wait.as_millis() as u64, "the broker did nothing with a call (its database connections were busy); sending it again"); + tokio::time::sleep(wait).await; + wait = (wait * 2).min(NOT_DONE_LONGEST); + } + answer => return answer, + } + } +} + // ---------- Journal ---------- pub struct BrokerJournalClient { @@ -260,7 +303,7 @@ impl JournalClient for BrokerJournalClient { } fn never_reached(&self, error: &anyhow::Error) -> bool { - never_sent(error) + never_sent(error) || did_nothing(error) } async fn record_event( @@ -386,7 +429,8 @@ impl TaskStoreClient for BrokerTaskStoreClient { async fn claim_execution(&self, replica: &str, project_id: Uuid, execution_id: &str) -> Result> { let req = TaskClaimExecutionRequest { replica: replica.to_string(), project_id, execution_id: execution_id.to_string() }; - let resp: TaskClaimExecutionResponse = self.http.post("/v1/task/claim_execution", &req).await?; + let path = "/v1/task/claim_execution"; + let resp: TaskClaimExecutionResponse = until_done(path, || self.http.post(path, &req)).await?; Ok(resp.claimed) } @@ -414,7 +458,8 @@ impl TaskStoreClient for BrokerTaskStoreClient { replica: replica.to_string(), result, }; - let _: TaskCompleteResponse = self.http.post("/v1/task/complete", &req).await?; + let path = "/v1/task/complete"; + let _: TaskCompleteResponse = until_done(path, || self.http.post(path, &req)).await?; Ok(()) } @@ -424,23 +469,15 @@ impl TaskStoreClient for BrokerTaskStoreClient { replica: replica.to_string(), error, }; - let _: TaskFailResponse = self.http.post("/v1/task/fail", &req).await?; + let path = "/v1/task/fail"; + let _: TaskFailResponse = until_done(path, || self.http.post(path, &req)).await?; Ok(()) } - async fn wait_cancels(&self, project_id: Uuid, execution_ids: Vec, wait: Duration) -> Result> { - held( - wait, - |hold| { - let req = TaskWaitCancelsRequest { project_id, execution_ids: execution_ids.clone(), wait_ms: hold.as_millis() as u64 }; - async move { - let resp: TaskWaitCancelsResponse = self.http.post_held("/v1/task/wait_cancels", &req, hold).await?; - Ok(resp.cancels) - } - }, - |cancels: &Vec| !cancels.is_empty(), - ) - .await + async fn cancels_asked(&self, project_id: Uuid, execution_ids: Vec) -> Result> { + let req = TaskCancelsAskedRequest { project_id, execution_ids }; + let resp: TaskCancelsAskedResponse = self.http.post("/v1/task/cancels_asked", &req).await?; + Ok(resp.cancels) } } @@ -550,6 +587,13 @@ impl BrokerInfraClient { pub fn new(link: BrokerLink) -> Arc { Arc::new(Self { http: HttpCore::new(link) }) } + + /// Ask for `project`'s infra health to be looked at now: what a machine + /// running its infra says when how its units stand changed. + pub async fn ask_for_a_look(&self, project: Uuid) -> Result<()> { + let _: InfraLookResponse = self.http.post("/v1/infra/look", &InfraLookRequest { project_id: project }).await?; + Ok(()) + } } #[async_trait] @@ -1173,6 +1217,41 @@ mod read_retry_tests { assert_eq!(calls.load(Ordering::SeqCst), 3); } + /// Only the broker's own "nothing was done" makes a write safe to send + /// again: any other 503 may have half happened. + #[test] + fn only_nothing_done_is_safe_to_send_again() { + let not_done: anyhow::Error = + BrokerRefused { path: "/v1/task/complete".into(), status: reqwest::StatusCode::SERVICE_UNAVAILABLE, body: NOT_DONE.into() }.into(); + assert!(did_nothing(¬_done)); + assert!(!did_nothing(&refused(reqwest::StatusCode::SERVICE_UNAVAILABLE))); + assert!(!did_nothing(&refused(reqwest::StatusCode::INTERNAL_SERVER_ERROR))); + } + + /// A call the broker did nothing with is sent again until it is done; + /// one that failed otherwise is not. + #[tokio::test(start_paused = true)] + async fn a_call_that_did_nothing_is_sent_again() { + let calls = AtomicU32::new(0); + let answer = until_done("/v1/task/complete", || async { + match calls.fetch_add(1, Ordering::SeqCst) { + 0 | 1 => Err(BrokerRefused { path: "/v1/task/complete".into(), status: reqwest::StatusCode::SERVICE_UNAVAILABLE, body: NOT_DONE.into() }.into()), + _ => Ok(7), + } + }) + .await + .unwrap(); + assert_eq!((answer, calls.load(Ordering::SeqCst)), (7, 3)); + let calls = AtomicU32::new(0); + let failed: Result = until_done("/v1/task/complete", || async { + calls.fetch_add(1, Ordering::SeqCst); + Err(refused(reqwest::StatusCode::SERVICE_UNAVAILABLE)) + }) + .await; + assert!(failed.is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 1, "a 503 that may have half happened is not sent again"); + } + /// A refusal would say the same thing again: it fails at once. #[tokio::test(start_paused = true)] async fn a_refusal_fails_at_once() { diff --git a/crates/weft-broker-client/src/lifecycle_command.rs b/crates/weft-broker-client/src/lifecycle_command.rs index e49bb683..cb15d224 100644 --- a/crates/weft-broker-client/src/lifecycle_command.rs +++ b/crates/weft-broker-client/src/lifecycle_command.rs @@ -156,24 +156,43 @@ pub fn live_lease_exists(replica_param: Option<&str>, project_col: &str) -> Stri /// mid-command. pub fn ownable_project(project_alias: &str, held_param: &str) -> String { format!( - "({p}.has_infra OR {p}.id = ANY({held_param}) OR {work})", + "({p}.has_infra OR {p}.id = ANY({held_param}) OR {nodes} OR {work})", p = project_alias, + nodes = has_infra_nodes(project_alias), work = supervisor_work(project_alias) ) } /// SQL condition that is true iff the `project` row aliased -/// `project_alias` gives its owner something to look at now: infra -/// nodes whose health it watches, or a supervisor command waiting. A -/// project that only declares infra, or whose host holds only kept disks -/// of it, is ownable ([`ownable_project`]) but gives its owner nothing to -/// do until a command is issued, which wakes it, so a supervisor that -/// scales to zero sleeps over it. +/// `project_alias` has infra copies (`infra_node` rows), whatever their +/// status: copies a supervisor may still have to take down. +pub fn has_infra_nodes(project_alias: &str) -> String { + format!("EXISTS (SELECT 1 FROM infra_node own_n WHERE own_n.project_id = {project_alias}.id)") +} + +/// SQL condition that is true iff the `project` row aliased +/// `project_alias` has infra copies expected to run: what a supervisor +/// watches the health of, and is woken for when one of them changes. +// SYNC: the statuses <-> weft_core::infra::InfraNodeStatus::expects_running_units +pub fn runs_infra(project_alias: &str) -> String { + format!( + "EXISTS (SELECT 1 FROM infra_node run_n WHERE run_n.project_id = {project_alias}.id \ + AND run_n.status IN ('running', 'flaky'))" + ) +} + +/// SQL condition that is true iff the `project` row aliased +/// `project_alias` gives its owner something to do now that nothing +/// else will announce: a supervisor command waiting. A project whose +/// infra runs is ownable ([`ownable_project`]) but gives a supervisor +/// that scales to zero nothing to look at on a clock: a command being +/// issued, a machine saying how its unit stands changed +/// ([`LOOK_WAKE`]) and the cloud saying a machine went away each wake it, +/// and its health keeps it looking only while it is not settled. pub fn supervisor_work(project_alias: &str) -> String { format!( - "(EXISTS (SELECT 1 FROM infra_node work_n WHERE work_n.project_id = {p}.id) \ - OR EXISTS (SELECT 1 FROM infra_lifecycle_command work_c \ - WHERE work_c.project_id = {p}.id AND {pending}))", + "EXISTS (SELECT 1 FROM infra_lifecycle_command work_c \ + WHERE work_c.project_id = {p}.id AND {pending})", p = project_alias, pending = pending_supervisor_command("work_c"), ) @@ -255,7 +274,10 @@ pub fn commands_overlap(a: &str, b: &str) -> String { /// `infra_command_notify` trigger in the dispatcher's /// `infra_lifecycle_command::GROUP`: once when the command is issued, /// and once when it completes. The claimers wake on the first, and -/// whoever waits on a command's outcome wakes on the second. +/// whoever waits on a command's outcome wakes on the second. The broker +/// also announces here when a machine running a project's infra says +/// how it stands changed (`look:`), which wakes a supervisor +/// that scales to zero to look at it. pub const INFRA_COMMAND_CHANNEL: &str = "weft_infra_command"; /// What an [`INFRA_COMMAND_CHANNEL`] notification says. @@ -265,6 +287,9 @@ pub enum InfraCommandSignal<'a> { Issued { project_id: &'a str }, /// This command completed (`done:`). Done { id: i64 }, + /// Something running this project's infra changed how it stands, so + /// its health is looked at now (`look:`). + Look { project_id: &'a str }, } impl<'a> InfraCommandSignal<'a> { @@ -276,6 +301,9 @@ impl<'a> InfraCommandSignal<'a> { if let Some(project_id) = payload.strip_prefix("issued:") { return Some(Self::Issued { project_id }); } + if let Some(project_id) = payload.strip_prefix("look:") { + return Some(Self::Look { project_id }); + } payload.strip_prefix("done:")?.parse().ok().map(|id| Self::Done { id }) } } @@ -290,6 +318,20 @@ pub const ISSUED_WAKE: weft_task_store::drain::WakeOn = weft_task_store::drain:: concerns: |payload| matches!(InfraCommandSignal::parse(payload), Some(InfraCommandSignal::Issued { .. })), }; +/// What wakes a supervisor that scales to zero to look at a project's +/// health: something running its infra said how it stands changed. Its +/// health is otherwise looked at only while something is not settled, so +/// infra that runs fine costs no look at all. +pub const LOOK_WAKE: weft_task_store::drain::WakeOn = weft_task_store::drain::WakeOn { + channel: INFRA_COMMAND_CHANNEL, + concerns: |payload| matches!(InfraCommandSignal::parse(payload), Some(InfraCommandSignal::Look { .. })), +}; + +/// The [`INFRA_COMMAND_CHANNEL`] payload asking for a look at `project`. +pub fn look_payload(project: uuid::Uuid) -> String { + format!("look:{project}") +} + #[cfg(test)] mod verb_tests { use crate::protocol::InfraLifecycleVerb; diff --git a/crates/weft-broker-client/src/line.rs b/crates/weft-broker-client/src/line.rs index 5e971355..2e4028b5 100644 --- a/crates/weft-broker-client/src/line.rs +++ b/crates/weft-broker-client/src/line.rs @@ -35,11 +35,24 @@ //! run's history is asked again on either (`client::read_until_answered`), //! and any other call fails the way a request whose connection reset does. //! +//! A caller that stops waiting for a call it already wrote tells the +//! broker (`Notice::Forget`), which stops working on it, the way a request +//! whose caller hung up stops its handler: a wait the broker holds for +//! nobody would otherwise go on reading the database for its whole hold. +//! +//! A line with nothing to do closes ([`LINE_IDLE`]) and opens again on the +//! next call: an open line is an open request on the broker, and one held +//! by a process with nothing to ask would keep a broker that scales to +//! zero up for as long as that process lives. While it is closed nothing +//! is heard, so what is kept on it is not trusted until it is back. A +//! process that must hear while it asks nothing (a worker driving runs, +//! which hears their cancels) holds it open ([`BrokerLink::stay_open`]). +//! //! Byte streams (`ctx.storage` uploads and downloads) stay ordinary //! requests: a large body on the line would hold up every call behind it. use std::collections::HashMap; -use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex, OnceLock, Weak}; use std::time::Duration; @@ -106,6 +119,12 @@ const PONG_WAIT: Duration = Duration::from_secs(15); /// told to look again instead (`broadcast`'s lag). const NOTICE_CAPACITY: usize = 1024; +/// How long a line with no call out and nothing holding it open stays +/// open before it closes (see the module doc), and how often that is +/// looked at. +pub const LINE_IDLE: Duration = Duration::from_secs(60); +const IDLE_LOOK: Duration = Duration::from_secs(5); + /// Why a call on the line got no answer. #[derive(Debug, Clone, thiserror::Error)] pub enum LineError { @@ -146,13 +165,18 @@ pub struct AnswerHead { pub status: u16, } -/// What the broker says on the line besides answers, as text frames. +/// What either end says on the line besides calls and answers, as text +/// frames: the caller's requests (`Follow`, `Forget`), and the broker's +/// news. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(tag = "t", rename_all = "snake_case")] pub enum Notice { /// Sent by the caller once the line is open: push the notifications /// this caller may see. Follow, + /// Sent by the caller: it stopped waiting for the call `id`, so the + /// broker stops working on it. + Forget { id: u64 }, /// The broker follows for this caller; `listening` is whether its own /// database listener hears right now. Following { listening: bool }, @@ -186,8 +210,8 @@ pub fn unframe Deserialize<'de>>(bytes: &[u8]) -> anyhow::Result<(H, /// The notification channels a line may carry, by name: what turns a /// name the broker sent back into the `&'static str` a [`Heard`] holds. -// SYNC: LINE_CHANNELS <-> crates/weft-broker/src/line.rs (audience), crates/weft-dispatcher/src/infra_node.rs (infra_node_status_notify), crates/weft-dispatcher/src/project_store.rs (project_declared_infra_notify), crates/weft-access-store/src/lib.rs (access_notify) -pub const LINE_CHANNELS: &[&str] = &[INFRA_STATUS_CHANNEL, ACCESS_CHANNEL]; +// SYNC: LINE_CHANNELS <-> crates/weft-broker/src/line.rs (audience), crates/weft-task-store/src/tasks.rs (task_ready_notify), crates/weft-dispatcher/src/infra_node.rs (infra_node_status_notify), crates/weft-dispatcher/src/project_store.rs (project_declared_infra_notify), crates/weft-access-store/src/lib.rs (access_notify) +pub const LINE_CHANNELS: &[&str] = &[INFRA_STATUS_CHANNEL, ACCESS_CHANNEL, CANCEL_CHANNEL]; /// Announced with a project's id when one of its infra copies comes, /// goes, changes status or answers at another address, and when the @@ -200,6 +224,11 @@ pub const INFRA_STATUS_CHANNEL: &str = "weft_infra_status"; /// `tenant:` for a connection shared across the tenant's projects. pub const ACCESS_CHANNEL: &str = "weft_access"; +/// Announced with ` ` when a running execution +/// is asked to stop: the worker driving it asks for the cancel +/// (`weft_task_store::tasks::cancels_asked`). +pub use weft_task_store::tasks::CANCEL_CHANNEL; + /// A process's way to the broker. Every broker client of the process /// holds the same one, so the process has one line, opened on the first /// call and kept. @@ -214,6 +243,8 @@ struct LinkInner { /// Byte streams, which stay requests (see the module doc). http: reqwest::Client, line: OnceLock>, + /// How long the line stays open with nothing to do ([`LINE_IDLE`]). + idle_after: Duration, /// Dropped with the last [`BrokerLink`], which tells the line's task /// to close the line: nothing will call on it again. _closing: tokio::sync::watch::Sender<()>, @@ -233,12 +264,20 @@ impl BrokerLink { token, http, line: OnceLock::new(), + idle_after: weft_core::time_scale::scaled(LINE_IDLE), _closing: closing, closed, }), } } + /// Close the line after `after` with nothing to do instead of + /// [`LINE_IDLE`]. Only on a link no clone of which was made yet. + pub fn closing_when_idle_after(mut self, after: Duration) -> Self { + Arc::get_mut(&mut self.inner).expect("set before the link is shared").idle_after = after; + self + } + /// The broker's base URL. pub fn base_url(&self) -> &str { &self.inner.base_url @@ -270,8 +309,32 @@ impl BrokerLink { Subscription::with_listening(line.notices.subscribe(), line.listening.clone()) } + /// Hold the line open while the guard lives, calls or not, so what the + /// broker pushes is heard (see the module doc). A line closed for want + /// of anything to do opens again at once. + pub fn stay_open(&self) -> StayOpen { + let line = self.line().clone(); + if line.keeps.fetch_add(1, Ordering::AcqRel) == 0 { + let _ = line.outbox.send(Out::Wake); + } + StayOpen { line } + } + fn line(&self) -> &Arc { - self.inner.line.get_or_init(|| Line::open(Arc::downgrade(&self.inner), self.inner.closed.clone())) + self.inner + .line + .get_or_init(|| Line::open(Arc::downgrade(&self.inner), self.inner.closed.clone(), self.inner.idle_after)) + } +} + +/// Holds the line open (see [`BrokerLink::stay_open`]). +pub struct StayOpen { + line: Arc, +} + +impl Drop for StayOpen { + fn drop(&mut self) { + self.line.keeps.fetch_sub(1, Ordering::AcqRel); } } @@ -283,22 +346,38 @@ struct Waiting { } /// Takes a call off the line when its caller stops waiting for it, however -/// it stops (an answer already took it off; that is fine). +/// it stops (an answer already took it off; that is fine). One already +/// written is forgotten by the broker too. struct Forget<'a> { - calls: &'a Mutex>, + line: &'a Line, id: u64, } impl Drop for Forget<'_> { fn drop(&mut self) { - self.calls.lock().expect("line calls").remove(&self.id); + let removed = self.line.calls.lock().expect("line calls").remove(&self.id); + if removed.is_some_and(|waiting| waiting.frame.is_none()) { + let _ = self.line.outbox.send(Out::Forget(self.id)); + } } } +/// What the line's task is asked to write, in order. +enum Out { + /// The call with this id, unless its caller gave up first. + Call(u64), + /// That the caller stopped waiting for this written call. + Forget(u64), + /// Nothing: the line is wanted open (a [`StayOpen`] was taken). + Wake, +} + struct Line { calls: Mutex>, - /// The ids of calls to write, in order. - outbox: mpsc::UnboundedSender, + /// What to write, in order. + outbox: mpsc::UnboundedSender, + /// How many [`StayOpen`]s hold the line open. + keeps: AtomicUsize, next_id: AtomicU64, notices: broadcast::Sender, /// Whether what the line pushes can be trusted right now (see @@ -309,18 +388,19 @@ struct Line { } impl Line { - fn open(link: Weak, closed: tokio::sync::watch::Receiver<()>) -> Arc { + fn open(link: Weak, closed: tokio::sync::watch::Receiver<()>, idle_after: Duration) -> Arc { let (outbox, ids) = mpsc::unbounded_channel(); let (notices, _) = broadcast::channel(NOTICE_CAPACITY); let line = Arc::new(Self { calls: Mutex::new(HashMap::new()), outbox, + keeps: AtomicUsize::new(0), next_id: AtomicU64::new(1), notices, listening: Arc::new(AtomicBool::new(false)), down: Mutex::new(Some("not opened yet".into())), }); - tokio::spawn(keep_open(link, Arc::downgrade(&line), ids, closed)); + tokio::spawn(keep_open(link, Arc::downgrade(&line), ids, closed, idle_after)); line } @@ -331,11 +411,12 @@ impl Line { self.calls.lock().expect("line calls").insert(id, Waiting { frame: Some(frame), answer }); // A caller that stops waiting (its own timeout, a cancelled task) // takes its call off the line, so an abandoned write is never sent - // later behind its back. - let _forget = Forget { calls: &self.calls, id }; + // later behind its back, and one already sent is dropped by the + // broker too. + let _forget = Forget { line: self, id }; // The receiving end lives as long as the line's task, which lives // as long as this line: a send can only fail once neither does. - let _ = self.outbox.send(id); + let _ = self.outbox.send(Out::Call(id)); let start = tokio::time::Instant::now(); let to_send = wait.to_send.min(wait.to_answer); tokio::pin!(answered); @@ -379,6 +460,19 @@ impl Line { } } + /// The line closed for want of anything to do: nothing is heard until + /// it opens again. + fn idled(&self) { + self.listening.store(false, Ordering::Release); + let _ = self.notices.send(Heard::Lost); + *self.down.lock().expect("line down") = Some("closed while idle; it opens again on the next call".into()); + } + + /// Whether the line may close: no call is out and nothing holds it open. + fn idle(&self) -> bool { + self.keeps.load(Ordering::Acquire) == 0 && self.calls.lock().expect("line calls").is_empty() + } + fn answered(&self, id: u64, answer: Answer) { if let Some(waiting) = self.calls.lock().expect("line calls").remove(&id) { let _ = waiting.answer.send(Ok(answer)); @@ -408,7 +502,9 @@ impl Line { } None => tracing::warn!(target: "weft_broker_client::line", %channel, "the broker pushed a channel this build does not know"), }, - Notice::Follow => tracing::warn!(target: "weft_broker_client::line", "the broker sent a follow request"), + Notice::Follow | Notice::Forget { .. } => { + tracing::warn!(target: "weft_broker_client::line", ?notice, "the broker sent what only a caller sends") + } } } } @@ -419,10 +515,13 @@ impl Line { async fn keep_open( link: Weak, line: Weak, - mut ids: mpsc::UnboundedReceiver, + mut outbox: mpsc::UnboundedReceiver, mut closed: tokio::sync::watch::Receiver<()>, + idle_after: Duration, ) { let mut retry = RECONNECT_FIRST; + // What woke a closed line, written first once it is open again. + let mut first: Option = None; loop { let (Some(inner), Some(held)) = (link.upgrade(), line.upgrade()) else { return }; // What opening needs, taken out so nothing holds the link while it @@ -439,12 +538,24 @@ async fn keep_open( Ok(socket) => { retry = RECONNECT_FIRST; *held.down.lock().expect("line down") = None; - match serve(socket, &held, &mut ids, &mut closed).await { + match serve(socket, &held, first.take(), &mut outbox, &mut closed, idle_after).await { Served::LetGo => return, Served::Broke(why) => { tracing::warn!(target: "weft_broker_client::line", reason = %why, "the line to the broker broke; opening it again"); held.broke(&why); } + Served::Idle => { + held.idled(); + drop(held); + // Closed until something is asked of it. + tokio::select! { + out = outbox.recv() => match out { + Some(out) => first = Some(out), + None => return, + }, + _ = closed.changed() => return, + } + } } } Err(e) => { @@ -466,6 +577,9 @@ async fn keep_open( enum Served { /// It broke, for this reason; it is opened again. Broke(String), + /// It had nothing to do for [`LINE_IDLE`] and closed; it opens again + /// when something is asked of it. + Idle, /// The process let go of its last link: nothing will call on it again. LetGo, } @@ -516,17 +630,28 @@ fn line_url(base_url: &str) -> anyhow::Result { async fn serve( socket: Socket, line: &Line, - ids: &mut mpsc::UnboundedReceiver, + first: Option, + outbox: &mut mpsc::UnboundedReceiver, closed: &mut tokio::sync::watch::Receiver<()>, + idle_after: Duration, ) -> Served { let (mut sink, mut stream) = socket.split(); let follow = serde_json::to_string(&Notice::Follow).expect("a notice serializes"); if let Err(why) = write(&mut sink, Message::Text(follow), "asking to follow").await { return Served::Broke(why); } + if let Some(out) = first { + if let Err(why) = write_out(&mut sink, line, out).await { + return Served::Broke(why); + } + } let mut ping = tokio::time::interval(PING_EVERY); ping.tick().await; let mut pong_due: Option = None; + let mut idle_look = tokio::time::interval(IDLE_LOOK.min(idle_after)); + idle_look.tick().await; + // Since when the line has had nothing to do, while it has not. + let mut idle_since: Option = None; loop { let pong_deadline = pong_due.unwrap_or_else(|| tokio::time::Instant::now() + Duration::from_secs(3600)); let written = tokio::select! { @@ -536,11 +661,25 @@ async fn serve( let _ = write(&mut sink, Message::Close(None), "closing").await; return Served::LetGo; } - id = ids.recv() => { - let Some(id) = id else { return Served::LetGo }; - // A call whose caller gave up has no frame left to write. - let Some(frame) = line.take_frame(id) else { continue }; - write(&mut sink, Message::Binary(frame), "writing a call").await + out = outbox.recv() => { + let Some(out) = out else { return Served::LetGo }; + write_out(&mut sink, line, out).await + } + _ = idle_look.tick() => { + // Looked at from this task alone, which is the one that + // writes calls: none can be written between the look and + // the close, and one asked for meanwhile waits in the + // outbox and opens the line again. + match (line.idle(), idle_since) { + (false, _) => idle_since = None, + (true, None) => idle_since = Some(tokio::time::Instant::now()), + (true, Some(since)) if since.elapsed() >= idle_after => { + let _ = write(&mut sink, Message::Close(None), "closing while idle").await; + return Served::Idle; + } + (true, Some(_)) => {} + } + Ok(()) } message = stream.next() => { pong_due = None; @@ -585,6 +724,22 @@ async fn serve( type Sink = futures::stream::SplitSink; +/// Write what the outbox asked for. +async fn write_out(sink: &mut Sink, line: &Line, out: Out) -> Result<(), String> { + match out { + // A call whose caller gave up has no frame left to write. + Out::Call(id) => match line.take_frame(id) { + Some(frame) => write(sink, Message::Binary(frame), "writing a call").await, + None => Ok(()), + }, + Out::Forget(id) => { + let forget = serde_json::to_string(&Notice::Forget { id }).expect("a notice serializes"); + write(sink, Message::Text(forget), "forgetting a call").await + } + Out::Wake => Ok(()), + } +} + /// Write one message, within [`PONG_WAIT`]; `doing` names it in the reason /// the line broke. async fn write(sink: &mut Sink, message: Message, doing: &str) -> Result<(), String> { @@ -620,6 +775,7 @@ mod tests { fn a_notice_reads_back() { for notice in [ Notice::Follow, + Notice::Forget { id: 7 }, Notice::Following { listening: true }, Notice::Signal { channel: INFRA_STATUS_CHANNEL.into(), payload: "p".into() }, Notice::Lost, @@ -710,6 +866,10 @@ pub mod server { // for journal rows, a claim) would otherwise go on waiting for a // caller that is gone. let mut running = tokio::task::JoinSet::new(); + // Each running call by its id, so a caller that stops waiting + // stops it (`Notice::Forget`), the way a request whose caller hung + // up stops its handler. A call takes itself out when it ends. + let by_id: Arc>> = Arc::default(); let bytes = Arc::new(tokio::sync::Semaphore::new(BYTES_IN_FLIGHT)); let mut follower = follower; // The tasks pushing notices down this line, stopped when it goes. @@ -731,13 +891,19 @@ pub mod server { let size = u32::try_from(message.len()).expect("a line message is at most MAX_MESSAGE"); let Ok(call) = calls.clone().acquire_owned().await else { break }; let Ok(held) = bytes.clone().acquire_many_owned(size).await else { break }; - let (api, out) = (api.clone(), out.clone()); + let (api, out, ended) = (api.clone(), out.clone(), by_id.clone()); while running.try_join_next().is_some() {} - running.spawn(async move { + let id = head.id; + // Spawned under the lock its end takes, so a call that + // ends at once still finds itself there to take out. + let mut ids = by_id.lock().expect("line calls by id"); + let handle = running.spawn(async move { let (status, answer) = run(api, &head, message.slice(body_at..)).await; + ended.lock().expect("line calls by id").remove(&head.id); let _ = out.send(Message::Binary(frame(&AnswerHead { id: head.id, status }, &answer).into())).await; drop((call, held)); }); + ids.insert(id, handle); } Ok(Message::Text(text)) => match serde_json::from_str::(&text) { Ok(Notice::Follow) if following.is_empty() => { @@ -758,6 +924,11 @@ pub mod server { } } } + Ok(Notice::Forget { id }) => { + if let Some(call) = by_id.lock().expect("line calls by id").remove(&id) { + call.abort(); + } + } Ok(_) => {} Err(e) => { tracing::warn!(target: "weft_broker_client::line", error = %e, "a line sent a notice that does not read; closing it"); diff --git a/crates/weft-broker-client/src/protocol.rs b/crates/weft-broker-client/src/protocol.rs index 1fdd23e4..86dd685b 100644 --- a/crates/weft-broker-client/src/protocol.rs +++ b/crates/weft-broker-client/src/protocol.rs @@ -110,6 +110,14 @@ wire_enum! { } } +/// The body of a 503 the broker answers when a call did nothing at all: it +/// could not even get a connection to its database (every connection +/// busy, or the database refusing new ones), so no statement ran and +/// sending the call again cannot repeat anything. Any other 503 may have +/// half happened; a write is sent again only on this one. +// SYNC: NOT_DONE <-> crates/weft-broker/src/handlers.rs (unavailable_or_internal), crates/weft-broker-client/src/client.rs (did_nothing) +pub const NOT_DONE: &str = "the broker could not get a connection to its database, so nothing was done; send it again"; + // ---------- Journal ---------- /// The most one `/v1/journal/record` body may weigh. The heaviest event @@ -335,23 +343,31 @@ pub struct TaskFailRequest { #[derive(Debug, Clone, Serialize, Deserialize)] pub struct TaskFailResponse {} -/// The cancels asked for any of `execution_ids` of `project_id` (the executions -/// the asking worker drives), holding up to `wait_ms` (the broker caps it -/// at `pg_signal::MAX_HOLD`) for one when there are none. +/// The cancels asked for any of `execution_ids` of `project_id`, answered +/// for those the asking worker drives; asking only reads. #[derive(Debug, Clone, Serialize, Deserialize)] -pub struct TaskWaitCancelsRequest { +pub struct TaskCancelsAskedRequest { pub project_id: Uuid, pub execution_ids: Vec, - pub wait_ms: u64, } #[derive(Debug, Clone, Serialize, Deserialize)] -pub struct TaskWaitCancelsResponse { +pub struct TaskCancelsAskedResponse { pub cancels: Vec, } // ---------- Infra ---------- +/// A machine running `project_id`'s infra asking for its health to be +/// looked at now (`BrokerInfraClient::ask_for_a_look`). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct InfraLookRequest { + pub project_id: Uuid, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct InfraLookResponse {} + #[derive(Debug, Clone, Serialize, Deserialize)] pub struct InfraEndpointUrlRequest { /// The asking run. The broker resolves its project from it, so a @@ -658,9 +674,9 @@ pub struct SupervisorSyncOwnershipResponse { /// on one of them while nobody held it woke none of this replica's /// claims, so it asks again at once when this is not empty. pub claimed: Vec, - /// Whether a project this replica owns gives it something to look at - /// now (`lifecycle_command::supervisor_work`): its health is looked at - /// again soon while this is true. + /// Whether a project this replica owns gives it something to do now + /// that nothing will announce (`lifecycle_command::supervisor_work`: a + /// command waiting): it looks again soon while this is true. pub owns_work: bool, /// In how many seconds (on the database's clock, 0 when already) the /// soonest lease this replica does not hold lapses, over the projects @@ -1213,7 +1229,7 @@ pub enum LifecycleSpec { /// Take down what a health protocol reaches, under `spec`. Deactivate(TakeDownReaders), /// Bring back what the health loop took down: each activation it - /// took carries the mark (`deactivated_by_health`) the take-down + /// took carries the mark (`went_down_with`: health) the take-down /// set, and of those this restores every one that reads none of the /// `still_broken` copies. Reactivate(RestoreReaders), @@ -1635,18 +1651,21 @@ impl InFlightCommand { // ---------- Signals ---------- -/// The activation statuses whose signal rows belong in a listener process's -/// in-RAM registry: the activation governing the signal (its trigger's -/// for an entry, the trigger's that fired its run for a wait) is being -/// activated (the rehydrate at the end of activate runs before the flip -/// to active) or live. A signal no activation governs reads as live. A -/// hibernated or parked activation keeps its rows in the table so -/// reactivate can restore them, but the process was told to forget them at -/// deactivate, and a listener restarting must not bring them back. The -/// broker's `signal/list_held` (what a starting listener rehydrates) -/// reads this list. -pub const LISTENER_HELD_STATUSES: [&str; 2] = - [ProjectStatus::Activating.as_str(), ProjectStatus::Active.as_str()]; +/// Whether the activation `a` governing a signal (its trigger's for an +/// entry, the trigger's that fired its run for a wait; read through +/// [`SIGNAL_ACTIVATION_JOIN`]) has its listeners listen: being activated +/// (the rehydrate at the end of activate runs before the flip to active), +/// live, or off in a way that takes work while it is off (parked, or +/// hibernating within its grace window), since what a parked trigger hears +/// waits for it to be back on (`weft_dispatcher::arrival`). A signal no +/// activation governs listens. A wiped activation's rows are gone, and a +/// hibernation past its grace window stops listening. What a starting +/// listener rehydrates, what holders claim, and how many holders run all +/// read this. +// SYNC: ACTIVATION_LISTENS <-> crates/weft-dispatcher/src/arrival.rs (Standing::arrival), crates/weft-dispatcher/src/activation_store.rs (IN_GRACE_WINDOW_SQL) +pub const ACTIVATION_LISTENS: &str = "(a.status IS NULL OR a.status IN ('activating', 'active') \ + OR (a.status IN ('deactivating', 'inactive') AND a.accepting_fires \ + AND (a.fires_deadline_unix IS NULL OR a.fires_deadline_unix >= EXTRACT(EPOCH FROM NOW())::BIGINT)))"; /// The join every "is this signal held" read goes through: the /// activation governing signal `s` as `a` (absent when none governs it). @@ -1663,7 +1682,7 @@ pub const SIGNAL_ACTIVATION_JOIN: &str = "LEFT JOIN trigger_activation a \ pub const STILL_HELD_BY: &str = "SELECT EXISTS (SELECT 1 FROM signal WHERE token = $1 AND holds AND held_by = $2)"; /// Every signal the listener must hold: the rows whose governing -/// activation's status is one of [`LISTENER_HELD_STATUSES`], of one +/// activation listens ([`ACTIVATION_LISTENS`]), of one /// project when `project` names it (an activation's rehydrate), of every /// project when it is `None` (a listener's boot). #[derive(Debug, Clone, Serialize, Deserialize)] diff --git a/crates/weft-broker/src/auth.rs b/crates/weft-broker/src/auth.rs index 57d2a5b3..338b87ec 100644 --- a/crates/weft-broker/src/auth.rs +++ b/crates/weft-broker/src/auth.rs @@ -51,8 +51,11 @@ pub enum Role { InfraSupervisor, } -// NOTE: there is deliberately no `Infra` role. What an infra unit runs -// never talks to the broker: its endpoints are resolved by the WORKER via +// NOTE: there is deliberately no `Infra` role. An infra machine's own +// agent asks the broker for one thing, a look at its project's health +// (`/v1/infra/look`), and does so as its project's worker, which is the +// account the machine runs as: anything on that machine can ask the same, +// for that project only. A unit's endpoints are resolved by the WORKER via // `ctx.endpoint()`, and its lifecycle is the supervisor's. /// The tenant authority of a caller. @@ -248,6 +251,7 @@ pub async fn resolve_storage_caller( let (execution_id, instance) = match execution_id { None => (None, None), Some(execution_id) => { + // SYNC: execution.owner_replica <-> weft_bind_execution_id_owner, bind_execution_id_owner (crates/weft-task-store/src/tasks.rs) let row: Option<(String, uuid::Uuid, Option, Option)> = sqlx::query_as( "SELECT tenant_id, project_id, owner_replica, instance_id FROM execution WHERE execution_id = $1", ) diff --git a/crates/weft-broker/src/handlers.rs b/crates/weft-broker/src/handlers.rs index 42fbce8f..ac556823 100644 --- a/crates/weft-broker/src/handlers.rs +++ b/crates/weft-broker/src/handlers.rs @@ -55,7 +55,7 @@ pub async fn journal_record( .await .map_err(|e| match e { weft_journal::RecordError::MixedExecutions { .. } => (StatusCode::BAD_REQUEST, e.to_string()), - other => internal(other), + other => unavailable_or_internal(other.into()), })?; if written == 0 { require_owner(&state, &caller, execution_id, &req.replica).await?; @@ -171,13 +171,14 @@ async fn require_owner( execution_id: weft_core::ExecutionId, claimed_replica: &str, ) -> Result<(), (StatusCode, String)> { + // SYNC: execution.owner_replica <-> weft_bind_execution_id_owner, bind_execution_id_owner (crates/weft-task-store/src/tasks.rs) let owner: Option<(Option,)> = sqlx::query_as( "SELECT owner_replica FROM execution WHERE execution_id = $1", ) .bind(execution_id.to_string()) .fetch_optional(&state.pool) .await - .map_err(internal)?; + .map_err(|e| unavailable_or_internal(anyhow::Error::from(e).context("execution owner lookup")))?; let owner_replica = owner.and_then(|(p,)| p).ok_or(( StatusCode::FORBIDDEN, "execution has no owning replica yet; worker may not act on it".into(), @@ -231,6 +232,7 @@ pub async fn execution_tag( .await .map_err(internal)?; tx.commit().await.map_err(internal)?; + weft_task_store::announce::committed(&state.pool); Ok(Json(ExecutionTagResponse {})) } @@ -652,7 +654,7 @@ pub async fn task_claim_execution( state.tasks.claim_execution(&req.replica, req.project_id, &req.execution_id), scope::warm_execution_id_scope(&state.scope_cache, &state.pool, &req.execution_id), ); - let claimed = claimed.map_err(internal)?; + let claimed = claimed.map_err(unavailable_or_internal)?; // Latest-claim-wins execution ownership is bound IN the claim's own // transaction by the `task_claim_binds_execution_id_owner` DB trigger: // claiming an execution-bearing task atomically stamps @@ -707,7 +709,7 @@ pub async fn task_complete( .tasks .complete(req.task_id, &req.replica, req.result) .await - .map_err(internal)?; + .map_err(unavailable_or_internal)?; Ok(Json(TaskCompleteResponse {})) } @@ -723,21 +725,22 @@ pub async fn task_fail( .tasks .fail(req.task_id, &req.replica, req.error) .await - .map_err(internal)?; + .map_err(unavailable_or_internal)?; Ok(Json(TaskFailResponse {})) } -pub async fn task_wait_cancels( +pub async fn task_cancels_asked( State(state): State>, AuthedCaller(caller): AuthedCaller, - Json(req): Json, -) -> Resp { + Json(req): Json, +) -> Resp { require_worker(&caller)?; scope::require_project_owned_by(&state.scope_cache, &state.pool, &caller, req.project_id).await?; - // Only the executions the calling replica drives: a cancel is taken as it - // is answered, so a worker waiting on an execution it does not own would - // swallow a cancel meant for the one that does. + // Only the executions the calling replica drives. One it asks about and + // no longer drives (its claim moved to another worker, or its drive is + // ending) is answered nothing, without failing the ask for the others. let replica = caller.replica.as_deref().ok_or((StatusCode::FORBIDDEN, "a worker names its replica".into()))?; + // SYNC: execution.owner_replica <-> weft_bind_execution_id_owner, bind_execution_id_owner (crates/weft-task-store/src/tasks.rs) let owned: Vec = sqlx::query_scalar( "SELECT execution_id FROM execution WHERE execution_id = ANY($1) AND project_id = $2 AND owner_replica = $3", ) @@ -747,11 +750,8 @@ pub async fn task_wait_cancels( .fetch_all(&state.pool) .await .map_err(|e| unavailable_or_internal(anyhow::Error::from(e).context("owned executions")))?; - if owned.len() != req.execution_ids.len() { - return Err((StatusCode::FORBIDDEN, "a worker waits only on the executions it drives".into())); - } - let cancels = state.tasks.wait_cancels(req.project_id, owned, held(req.wait_ms)).await.map_err(internal)?; - Ok(Json(TaskWaitCancelsResponse { cancels })) + let cancels = state.tasks.cancels_asked(req.project_id, owned).await.map_err(internal)?; + Ok(Json(TaskCancelsAskedResponse { cancels })) } // ---------- Infra ---------- @@ -846,6 +846,26 @@ fn require_declared_infra( )) } +/// A machine running `project_id`'s infra says how its units stand +/// changed: announce it, which wakes a supervisor that scales to zero to +/// look at the project's health (`lifecycle_command::LOOK_WAKE`). The +/// machine runs as its project's own account, so a project asks only +/// about itself; weft's own roles may ask about any. +pub async fn infra_look( + State(state): State>, + AuthedCaller(caller): AuthedCaller, + Json(req): Json, +) -> Resp { + scope::require_project_owned_by(&state.scope_cache, &state.pool, &caller, req.project_id).await?; + sqlx::query("SELECT pg_notify($1, $2)") + .bind(weft_broker_client::lifecycle_command::INFRA_COMMAND_CHANNEL) + .bind(weft_broker_client::lifecycle_command::look_payload(req.project_id)) + .execute(&state.pool) + .await + .map_err(|e| unavailable_or_internal(anyhow::Error::from(e).context("announce the look")))?; + Ok(Json(InfraLookResponse {})) +} + pub async fn infra_endpoint_url( State(state): State>, AuthedCaller(caller): AuthedCaller, @@ -2439,7 +2459,7 @@ async fn require_task_owned_by( .bind(task_id) .fetch_optional(&state.pool) .await - .map_err(|e| internal(anyhow::anyhow!("{e}")))?; + .map_err(|e| unavailable_or_internal(anyhow::Error::from(e).context("task owner lookup")))?; let owner = row.and_then(|(t,)| t).ok_or(( StatusCode::NOT_FOUND, format!("unknown task {task_id}"), @@ -2473,18 +2493,31 @@ pub(crate) fn internal(e: E) -> (StatusCode, String) { /// A failed database call, as a caller that can wait must tell it /// apart: 503 when the database could not be reached right now (asking -/// again will work once it can), 500 for anything else (a row that does -/// not decode fails the same way every time). The worker's journal -/// client waits out a 503 and fails the run on a 500. -// SYNC: 503 = ask again <-> crates/weft-broker-client/src/client.rs broker_unavailable +/// again will work once it can), with [`NOT_DONE`] when no connection +/// could even be had, so nothing ran and a write may be sent again; 500 +/// for anything else (a row that does not decode fails the same way every +/// time). The worker's journal client sends a write again on a 503 that +/// did nothing, waits out any 503 on a read, and fails the run otherwise. +// SYNC: 503 = ask again <-> crates/weft-broker-client/src/client.rs broker_unavailable, did_nothing pub(crate) fn unavailable_or_internal(e: anyhow::Error) -> (StatusCode, String) { - if e.chain().filter_map(|cause| cause.downcast_ref::()).any(database_unreachable) { + let causes = || e.chain().filter_map(|cause| cause.downcast_ref::()); + if causes().any(no_connection) { + tracing::warn!(target: "weft_broker", "no connection to the database could be had: {e:#}"); + return (StatusCode::SERVICE_UNAVAILABLE, NOT_DONE.into()); + } + if causes().any(database_unreachable) { tracing::warn!(target: "weft_broker", "could not reach the database: {e:#}"); return (StatusCode::SERVICE_UNAVAILABLE, "the database is unavailable; ask again".into()); } internal(e) } +/// Whether a database error means no connection could be had for the +/// call, so not one of its statements ran. +fn no_connection(e: &sqlx::Error) -> bool { + matches!(e, sqlx::Error::PoolTimedOut | sqlx::Error::PoolClosed) +} + /// Whether a database error means the server could not be reached or /// is restarting, rather than that the statement itself failed: no /// connection to be had, a connection that broke, or the server diff --git a/crates/weft-broker/src/held_signals.rs b/crates/weft-broker/src/held_signals.rs index 847bb2ce..c09c723c 100644 --- a/crates/weft-broker/src/held_signals.rs +++ b/crates/weft-broker/src/held_signals.rs @@ -10,24 +10,22 @@ use sqlx::{PgPool, Row}; use weft_broker_client::protocol::{ - HeldNow, HeldServing, SignalAuthKind, SignalHoldResponse, SignalRowWire, SignalSurfaceKind, LISTENER_HELD_STATUSES, + HeldNow, HeldServing, SignalAuthKind, SignalHoldResponse, SignalRowWire, SignalSurfaceKind, ACTIVATION_LISTENS, SIGNAL_ACTIVATION_JOIN, STILL_HELD_BY, }; /// The signal rows a booting or rehydrating listener must hold: every row -/// whose governing activation is in [`LISTENER_HELD_STATUSES`] (or that -/// none governs). A hibernated or parked activation keeps its rows -/// (reactivate restores them from here) while deactivate told the -/// listener to forget them; handing them back to a restarting listener -/// would revive a parked trigger's timers behind the user's back, so those -/// rows never come out of this query. Mixed tenants: each row carries its -/// own. `project` narrows it to one project's rows. +/// whose governing activation listens ([`ACTIVATION_LISTENS`], or that +/// none governs). A wiped activation's rows are gone; a hibernation past +/// its grace window keeps its rows (a reactivate restores them from here) +/// but no longer listens, so they never come out of this query. Mixed +/// tenants: each row carries its own. `project` narrows it to one +/// project's rows. pub async fn signals_held(pool: &PgPool, project: Option) -> anyhow::Result> { let rows = sqlx::query(&format!( - "{} WHERE COALESCE(a.status, 'active') = ANY($1) AND ($2::uuid IS NULL OR s.project_id = $2)", + "{} WHERE {ACTIVATION_LISTENS} AND ($1::uuid IS NULL OR s.project_id = $1)", held_select() )) - .bind(&LISTENER_HELD_STATUSES[..]) .bind(project) .fetch_all(pool) .await?; @@ -35,14 +33,10 @@ pub async fn signals_held(pool: &PgPool, project: Option) -> anyhow: } /// One held signal by its token, or `None` when no row with that token is -/// held (gone, or its activation parked): what the listener loads a -/// signal it has not seen yet from. +/// held (gone, or its activation no longer listens): what the listener +/// loads a signal it has not seen yet from. pub async fn signal_held(pool: &PgPool, token: &str) -> anyhow::Result> { - let row = sqlx::query(&format!( - "{} WHERE s.token = $2 AND COALESCE(a.status, 'active') = ANY($1)", - held_select() - )) - .bind(&LISTENER_HELD_STATUSES[..]) + let row = sqlx::query(&format!("{} WHERE s.token = $1 AND {ACTIVATION_LISTENS}", held_select())) .bind(token) .fetch_optional(pool) .await?; @@ -55,9 +49,8 @@ pub async fn signal_held(pool: &PgPool, token: &str) -> anyhow::Result anyhow::Result> { let row: Option<(uuid::Uuid,)> = sqlx::query_as(&format!( "SELECT s.project_id FROM signal s {SIGNAL_ACTIVATION_JOIN} \ - WHERE s.token = $2 AND COALESCE(a.status, 'active') = ANY($1)" + WHERE s.token = $1 AND {ACTIVATION_LISTENS}" )) - .bind(&LISTENER_HELD_STATUSES[..]) .bind(token) .fetch_optional(pool) .await?; @@ -110,21 +103,19 @@ pub async fn hold( FROM unnest($1::text[], $2::jsonb[]) AS x(token, serving) \ WHERE r.token = x.token AND r.holds AND r.held_by = $4 AND {held} \ RETURNING r.token", - held = activation_held("r", "$5"), + held = activation_held("r"), )) .bind(&tokens) .bind(&servings) .bind(lease_secs) .bind(replica) - .bind(&LISTENER_HELD_STATUSES[..]) .fetch_all(&mut *tx) .await?; let ended: Vec = sqlx::query_scalar(&format!( "SELECT x.token FROM unnest($1::text[]) AS x(token) WHERE NOT {held}", - held = activation_held("x", "$2"), + held = activation_held("x"), )) .bind(&tokens) - .bind(&LISTENER_HELD_STATUSES[..]) .fetch_all(&mut *tx) .await?; let room = room.map(|r| i64::from(r).saturating_sub(kept.len() as i64).max(0)); @@ -140,14 +131,13 @@ pub async fn hold( UPDATE signal s SET held_by = $4, held_until = EXTRACT(EPOCH FROM NOW())::BIGINT + $5, serving = NULL \ FROM free WHERE s.token = free.token \ RETURNING s.token", - held = activation_held("r", "$6"), + held = activation_held("r"), )) .bind(&tokens) .bind(want) .bind(room) .bind(replica) .bind(lease_secs) - .bind(&LISTENER_HELD_STATUSES[..]) .fetch_all(&mut *tx) .await?; let rows = sqlx::query(&format!("{} WHERE s.token = ANY($1)", held_select())) @@ -222,14 +212,14 @@ pub async fn let_go(pool: &PgPool, replica: &str) -> anyhow::Result<()> { } /// Whether the token `row.token` names a signal row that a listener holds -/// (`statuses` names the bound list of [`LISTENER_HELD_STATUSES`]): false -/// when the row is gone or its activation is parked. A condition an -/// UPDATE can carry, read through the one [`SIGNAL_ACTIVATION_JOIN`]. -/// That join names its signal `s`, so `row` is any other alias. -fn activation_held(row: &str, statuses: &str) -> String { +/// ([`ACTIVATION_LISTENS`]): false when the row is gone or its activation +/// no longer listens. A condition an UPDATE can carry, read through the +/// one [`SIGNAL_ACTIVATION_JOIN`]. That join names its signal `s`, so +/// `row` is any other alias. +fn activation_held(row: &str) -> String { format!( "EXISTS (SELECT 1 FROM signal s {SIGNAL_ACTIVATION_JOIN} \ - WHERE s.token = {row}.token AND COALESCE(a.status, 'active') = ANY({statuses}))" + WHERE s.token = {row}.token AND {ACTIVATION_LISTENS})" ) } diff --git a/crates/weft-broker/src/lib.rs b/crates/weft-broker/src/lib.rs index e9e00967..3fbf4129 100644 --- a/crates/weft-broker/src/lib.rs +++ b/crates/weft-broker/src/lib.rs @@ -126,12 +126,14 @@ fn api(state: Arc) -> Router { .route("/v1/task/complete", post(handlers::task_complete)) .route("/v1/task/fail", post(handlers::task_fail)) // The cancels for the executions a worker drives. - .route("/v1/task/wait_cancels", post(handlers::task_wait_cancels)) + .route("/v1/task/cancels_asked", post(handlers::task_cancels_asked)) // Infra reads .route( "/v1/infra/endpoint_url", post(handlers::infra_endpoint_url), ) + // A machine running a project's infra asking for a look at it. + .route("/v1/infra/look", post(handlers::infra_look)) // Project (worker fetches its own ProjectDefinition) .route( "/v1/project/fetch_definition", diff --git a/crates/weft-broker/src/lifecycle_writes.rs b/crates/weft-broker/src/lifecycle_writes.rs index 80895197..a6927b4b 100644 --- a/crates/weft-broker/src/lifecycle_writes.rs +++ b/crates/weft-broker/src/lifecycle_writes.rs @@ -112,19 +112,23 @@ pub async fn sync_ownership( // 3. When this supervisor next has something to look at: now, while a // project it owns gives it work; or when the soonest lease a - // sibling holds over such a project, or over one the host holds - // copies of (which a sibling's gone-copy sweep may have left - // half done), lapses: the sibling may be gone, and only a lapsed - // lease is taken over. One of those that nobody holds (past this - // tick's batch) counts as lapsed. + // sibling holds lapses, over a project with work, with infra copies + // expected to run (a machine's word that one changed may have woken + // this replica rather than the sibling holding the project, which + // sleeps; a stopped copy says nothing), or + // whose copies the host holds (which a sibling's gone-copy sweep may + // have left half done): the sibling may be gone, and only a lapsed + // lease is taken over, then looked at. One of those that nobody + // holds (past this tick's batch) counts as lapsed. let (owns_work, others_lapse_in_secs): (bool, Option) = sqlx::query_as(&format!( "SELECT \ EXISTS (SELECT 1 FROM project p JOIN infra_owner io ON io.project_id = p.id \ WHERE io.supervisor_replica = $1 AND {work}), \ (SELECT MIN(GREATEST(COALESCE(io.leased_until_unix, 0) - EXTRACT(EPOCH FROM NOW())::BIGINT, 0)) \ FROM project p LEFT JOIN infra_owner io ON io.project_id = p.id \ - WHERE io.supervisor_replica IS DISTINCT FROM $1 AND ({work} OR p.id = ANY($2)))", + WHERE io.supervisor_replica IS DISTINCT FROM $1 AND ({work} OR {nodes} OR p.id = ANY($2)))", work = supervisor_work("p"), + nodes = weft_broker_client::lifecycle_command::runs_infra("p"), )) .bind(replica) .bind(held_projects) @@ -157,7 +161,7 @@ where let sql = format!( "SELECT p.id AS project_id, p.tenant_id, \ a.status, a.accepting_fires, a.fires_visible_to_consumers, a.fires_deadline_unix, \ - a.drain_deadline_unix, a.deactivated_by_health, a.activating_execution_id \ + a.drain_deadline_unix, a.went_down_with, a.went_down_at_unix, a.activating_execution_id \ FROM infra_owner io \ JOIN project p ON p.id = io.project_id \ LEFT JOIN trigger_activation a ON a.project_id = p.id \ @@ -188,7 +192,18 @@ where fires_visible_to_consumers: r.try_get("fires_visible_to_consumers").context("decode visibility")?, fires_deadline_unix: r.try_get("fires_deadline_unix").context("decode deadline")?, drain_deadline_unix: r.try_get("drain_deadline_unix").context("decode drain deadline")?, - deactivated_by_health: r.try_get("deactivated_by_health").context("decode deactivated_by_health")?, + went_down: match ( + r.try_get::, _>("went_down_with").context("decode went_down_with")?, + r.try_get::, _>("went_down_at_unix").context("decode went_down_at_unix")?, + ) { + (None, None) => None, + (Some(cause), Some(at_unix)) => Some(weft_broker_client::activation::WentDown { + with: weft_broker_client::activation::DownWith::parse(&cause) + .with_context(|| format!("trigger_activation.went_down_with='{cause}' is not a known cause"))?, + at_unix, + }), + (cause, at) => anyhow::bail!("trigger_activation.went_down_with={cause:?} and went_down_at_unix={at:?} must be set together"), + }, activating_execution_id: r.try_get("activating_execution_id").context("decode activating_execution_id")?, }; projects.last_mut().expect("pushed above").2.push(lifecycle); @@ -203,7 +218,7 @@ where status: aggregate.status, health_parked: lifecycles .iter() - .any(|l| l.deactivated_by_health && l.status == ProjectStatus::Inactive), + .any(weft_broker_client::activation::ActivationLifecycle::parked_by_health), } }) .collect()) diff --git a/crates/weft-broker/src/line.rs b/crates/weft-broker/src/line.rs index 865d94e1..e5017845 100644 --- a/crates/weft-broker/src/line.rs +++ b/crates/weft-broker/src/line.rs @@ -27,7 +27,7 @@ use axum::Router; use tokio::sync::mpsc; use weft_broker_client::line::server::{self, Follower}; -use weft_broker_client::line::{Notice, ACCESS_CHANNEL, INFRA_STATUS_CHANNEL, LINE_PATH}; +use weft_broker_client::line::{Notice, ACCESS_CHANNEL, CANCEL_CHANNEL, INFRA_STATUS_CHANNEL, LINE_PATH}; use weft_platform_traits::identity::Principal; use weft_task_store::pg_signal::{Heard, PgSignalWatch}; @@ -103,11 +103,13 @@ fn follower(fanout: Arc, principal: Principal) -> Follower { /// Who a notification on `channel` is for, by its `payload`: an infra /// copy's change is its project's; a connection's is its project's, or its /// tenant's when it is shared across the tenant's projects -/// (`tenant:`). Every other channel is for nobody on a line. -// SYNC: the channels and payloads <-> weft_broker_client::line::LINE_CHANNELS, crates/weft-dispatcher/src/infra_node.rs (infra_node_status_notify), crates/weft-dispatcher/src/project_store.rs (project_declared_infra_notify), crates/weft-access-store/src/lib.rs (access_notify), crates/weft-broker/src/caller_auth.rs (names), crates/weft-dispatcher/src/held.rs (access_changed) +/// (`tenant:`); a cancel is its run's project's +/// (` `). Every other channel is for nobody on a line. +// SYNC: the channels and payloads <-> weft_broker_client::line::LINE_CHANNELS, crates/weft-dispatcher/src/infra_node.rs (infra_node_status_notify), crates/weft-dispatcher/src/project_store.rs (project_declared_infra_notify), crates/weft-access-store/src/lib.rs (access_notify), crates/weft-task-store/src/tasks.rs (task_ready_notify, cancel_payload), crates/weft-broker/src/caller_auth.rs (names), crates/weft-dispatcher/src/held.rs (access_changed) fn audience<'a>(channel: &str, payload: &'a str) -> Option> { match channel { INFRA_STATUS_CHANNEL => Some(Audience::Project(payload)), + CANCEL_CHANNEL => payload.split_once(' ').map(|(project, _)| Audience::Project(project)), ACCESS_CHANNEL => Some(match payload.strip_prefix("tenant:") { Some(tenant) => Audience::Tenant(tenant), None => Audience::Project(payload), @@ -345,6 +347,80 @@ mod tests { assert_eq!(counted.load(Ordering::SeqCst), 1, "only the call still waited on reached the broker"); } + /// A call whose caller stopped waiting stops at the broker too: a held + /// wait goes on for nobody no longer. + #[tokio::test] + async fn a_call_its_caller_gave_up_on_stops_at_the_broker() { + struct Stopped(StdArc); + impl Drop for Stopped { + fn drop(&mut self) { + self.0.notify_one(); + } + } + let stopped = StdArc::new(tokio::sync::Notify::new()); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let api = api(StdArc::default()).route( + "/v1/forever", + post({ + let stopped = stopped.clone(); + move || { + let guard = Stopped(stopped.clone()); + async move { + let _guard = guard; + std::future::pending::<&'static str>().await + } + } + }), + ); + let _server = serve(listener, server::with_line(api)).await; + let link = link(address); + let wait = CallWait { to_send: Duration::from_secs(10), to_answer: Duration::from_millis(300) }; + let gave_up = link.call("/v1/forever", Vec::new(), wait).await.unwrap_err(); + assert!(format!("{gave_up:#}").contains("did not answer"), "{gave_up:#}"); + tokio::time::timeout(Duration::from_secs(10), stopped.notified()).await.expect("the broker stopped the call"); + let answer = link.call("/v1/echo", b"still".to_vec(), CallWait::within(Duration::from_secs(10))).await.unwrap(); + assert!(String::from_utf8(answer.body).unwrap().starts_with("still|"), "the line stays usable"); + } + + /// A line with nothing to do closes, so the broker holds no request for + /// it, and opens again for the next call; one held open stays open. + #[tokio::test] + async fn an_idle_line_closes_and_opens_again_for_the_next_call() { + use axum::extract::ws::Message; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let (opened_tx, mut opened) = mpsc::unbounded_channel::<()>(); + let (closed_tx, mut closed) = mpsc::unbounded_channel::<()>(); + let app = Router::new().route( + LINE_PATH, + get(move |ws: WebSocketUpgrade| { + let (opened, closed) = (opened_tx.clone(), closed_tx.clone()); + async move { + ws.on_upgrade(move |mut socket| async move { + let _ = opened.send(()); + while let Some(Ok(message)) = socket.recv().await { + if matches!(message, Message::Close(_)) { + break; + } + } + let _ = closed.send(()); + }) + } + }), + ); + let _server = serve(listener, app).await; + let link = link(address).closing_when_idle_after(Duration::from_millis(300)); + let held = link.stay_open(); + tokio::time::timeout(Duration::from_secs(10), opened.recv()).await.expect("held open, the line opened").unwrap(); + tokio::time::sleep(Duration::from_millis(900)).await; + assert!(closed.try_recv().is_err(), "a line held open stays open"); + drop(held); + tokio::time::timeout(Duration::from_secs(10), closed.recv()).await.expect("with nothing to do, it closed").unwrap(); + let _woken = link.stay_open(); + tokio::time::timeout(Duration::from_secs(10), opened.recv()).await.expect("asked for again, it opened").unwrap(); + } + /// A process that lets go of its last link closes its line: nothing /// stays open at the broker for a link nobody holds. #[tokio::test] @@ -426,6 +502,8 @@ mod tests { assert_eq!(audience(INFRA_STATUS_CHANNEL, &project), Some(Audience::Project(&project))); assert_eq!(audience(ACCESS_CHANNEL, &project), Some(Audience::Project(&project))); assert_eq!(audience(ACCESS_CHANNEL, "tenant:t"), Some(Audience::Tenant("t"))); + let cancel = format!("{project} e-1"); + assert_eq!(audience(CANCEL_CHANNEL, &cancel), Some(Audience::Project(&project))); assert_eq!(audience("weft_task_ready", "worker:x"), None); } diff --git a/crates/weft-broker/src/scope.rs b/crates/weft-broker/src/scope.rs index 78ad6d90..47f978fa 100644 --- a/crates/weft-broker/src/scope.rs +++ b/crates/weft-broker/src/scope.rs @@ -27,7 +27,6 @@ use sqlx::postgres::PgPool; use tokio::sync::Mutex; use crate::auth::CallerIdentity; -use crate::handlers::internal; /// Cache size per resource kind. 100k is well above any realistic /// active-execution count and avoids the perf cliff DashMap's "drop @@ -224,7 +223,7 @@ async fn lookup_project_tenant( .bind(project_id) .fetch_optional(pool) .await - .map_err(|e| internal(anyhow::anyhow!("project lookup: {e}")))?; + .map_err(|e| crate::handlers::unavailable_or_internal(anyhow::Error::from(e).context("project lookup")))?; let tenant = row .ok_or((StatusCode::NOT_FOUND, "unknown project".into()))? .0; diff --git a/crates/weft-broker/src/state.rs b/crates/weft-broker/src/state.rs index cec73aff..eaec4ba4 100644 --- a/crates/weft-broker/src/state.rs +++ b/crates/weft-broker/src/state.rs @@ -30,6 +30,7 @@ pub const BROKER_CHANNELS: &[&str] = &[ // What a line pushes to the workers following it (`crate::line`). weft_broker_client::line::INFRA_STATUS_CHANNEL, weft_broker_client::line::ACCESS_CHANNEL, + weft_broker_client::line::CANCEL_CHANNEL, ]; pub struct BrokerState { diff --git a/crates/weft-broker/tests/held_signals.rs b/crates/weft-broker/tests/held_signals.rs index dd1199c1..639dcbc0 100644 --- a/crates/weft-broker/tests/held_signals.rs +++ b/crates/weft-broker/tests/held_signals.rs @@ -1,6 +1,7 @@ //! Layer-3 contract tests for `held_signals`, the statements the listener -//! depends on, against a REAL Postgres: the rule "a parked trigger's rows -//! never come back into a listener" lives in the SQL's status filter, and +//! depends on, against a REAL Postgres: the rule "a listener holds the +//! rows of a trigger that takes work, and never a wiped one's or a +//! hibernation's past its window" lives in the SQL's filter, and //! "of two listeners woken for the same moment exactly one acts" lives in //! the write's WHERE, so only the statements themselves can prove them. //! @@ -58,31 +59,59 @@ const ACTIVE: uuid::Uuid = uuid::Uuid::from_u128(0xa); const ACTIVATING: uuid::Uuid = uuid::Uuid::from_u128(0xb); const PARKED: uuid::Uuid = uuid::Uuid::from_u128(0xc); const DEACTIVATING: uuid::Uuid = uuid::Uuid::from_u128(0xd); +const WIPED: uuid::Uuid = uuid::Uuid::from_u128(0xe); +const HIBERNATING: uuid::Uuid = uuid::Uuid::from_u128(0xf); +/// Take the activation of `token` off work as a wipe leaves it: inactive +/// and refusing, with no grace window. +async fn wipe(pool: &PgPool, token: &str) { + sqlx::query("UPDATE trigger_activation SET status = 'inactive', accepting_fires = FALSE, fires_deadline_unix = NULL WHERE trigger = $1") + .bind(token) + .execute(pool) + .await + .expect("wipe"); +} + +/// The listener holds the rows of every trigger that takes work (on, being +/// set up, draining, parked, or hibernating within its grace window: what +/// those hear waits for the trigger to be back), and never a wiped one's or +/// a hibernation's past its window. #[sqlx::test] -async fn the_listener_holds_live_projects_and_never_a_parked_one(pool: PgPool) { +async fn the_listener_holds_what_takes_work_and_never_what_refuses_it(pool: PgPool) { schema(&pool).await; - for id in [ACTIVE, ACTIVATING, PARKED, DEACTIVATING] { + for id in [ACTIVE, ACTIVATING, PARKED, DEACTIVATING, WIPED, HIBERNATING] { project(&pool, id).await; } signal(&pool, "active", ACTIVE, ProjectStatus::Active).await; signal(&pool, "activating", ACTIVATING, ProjectStatus::Activating).await; signal(&pool, "parked", PARKED, ProjectStatus::Inactive).await; signal(&pool, "deactivating", DEACTIVATING, ProjectStatus::Deactivating).await; + signal(&pool, "wiped", WIPED, ProjectStatus::Inactive).await; + wipe(&pool, "wiped").await; + for (token, deadline) in [("in_grace", i64::MAX / 2), ("past_grace", 1)] { + signal(&pool, token, HIBERNATING, ProjectStatus::Inactive).await; + sqlx::query("UPDATE trigger_activation SET fires_visible_to_consumers = FALSE, fires_deadline_unix = $2 WHERE trigger = $1") + .bind(token) + .bind(deadline) + .execute(&pool) + .await + .unwrap(); + } let mut tokens: Vec = signals_held(&pool, None).await.expect("query").into_iter().map(|r| r.token).collect(); tokens.sort(); - assert_eq!(tokens, vec!["activating".to_string(), "active".to_string()]); + assert_eq!(tokens, vec!["activating", "active", "deactivating", "in_grace", "parked"]); // An activation's rehydrate reads its own project's rows only. let one: Vec = signals_held(&pool, Some(ACTIVATING)).await.expect("query").into_iter().map(|r| r.token).collect(); assert_eq!(one, vec!["activating".to_string()]); - assert!(signals_held(&pool, Some(PARKED)).await.expect("query").is_empty(), "scoped, still only held rows"); + assert!(signals_held(&pool, Some(WIPED)).await.expect("query").is_empty(), "scoped, still only held rows"); // One by token follows the same rule. - assert_eq!(signal_held(&pool, "active").await.unwrap().map(|r| r.token).as_deref(), Some("active")); - assert!(signal_held(&pool, "parked").await.unwrap().is_none(), "a parked row never comes back"); + assert_eq!(signal_held(&pool, "parked").await.unwrap().map(|r| r.token).as_deref(), Some("parked")); + assert!(signal_held(&pool, "wiped").await.unwrap().is_none(), "a wiped row never comes back"); + assert!(signal_held(&pool, "past_grace").await.unwrap().is_none(), "nor one past its grace window"); assert!(signal_held(&pool, "nothing").await.unwrap().is_none()); } @@ -123,18 +152,19 @@ fn tokens(rows: &[weft_broker_client::protocol::SignalRowWire]) -> Vec { } /// Two holders share the held signals: each takes only what no live -/// holder claims, up to its room, and never a parked project's or one that +/// holder claims, up to its room, and never a wiped project's or one that /// holds nothing. #[sqlx::test] async fn holders_take_disjoint_shares_within_their_room(pool: PgPool) { schema(&pool).await; - for id in [ACTIVE, PARKED] { + for id in [ACTIVE, WIPED] { project(&pool, id).await; } for t in ["a", "b", "c"] { held_signal(&pool, t, ACTIVE, ProjectStatus::Active).await; } - held_signal(&pool, "parked", PARKED, ProjectStatus::Inactive).await; + held_signal(&pool, "wiped", WIPED, ProjectStatus::Inactive).await; + wipe(&pool, "wiped").await; signal(&pool, "form", ACTIVE, ProjectStatus::Active).await; let one = hold(&pool, "h1", &[], Some(2), &[], 30).await.unwrap(); @@ -144,7 +174,7 @@ async fn holders_take_disjoint_shares_within_their_room(pool: PgPool) { let mut all = tokens(&one.taken); all.extend(tokens(&two.taken)); all.sort(); - assert_eq!(all, vec!["a", "b", "c"], "every live held signal once, never the parked one or the form"); + assert_eq!(all, vec!["a", "b", "c"], "every live held signal once, never the wiped one or the form"); assert!(hold(&pool, "h3", &[], None, &[], 30).await.unwrap().taken.is_empty(), "nothing left"); } @@ -201,27 +231,27 @@ async fn a_row_that_stops_holding_lets_its_holder_go(pool: PgPool) { assert!(after.kept.is_empty() && after.ended.is_empty(), "the row lives on, served another way: {after:?}"); } -/// Of what a holder holds, a row that is gone or whose activation parked -/// has ended; a row still held by it is kept, and neither is both. +/// Of what a holder holds, a row that is gone or whose activation takes no +/// work any more has ended; a row still held by it is kept (a parked one +/// included), and none is two of these. #[sqlx::test] -async fn a_gone_or_parked_row_has_ended(pool: PgPool) { +async fn a_gone_or_wiped_row_has_ended(pool: PgPool) { schema(&pool).await; project(&pool, ACTIVE).await; - for t in ["kept", "gone", "parked"] { + for t in ["kept", "parked", "gone", "wiped"] { held_signal(&pool, t, ACTIVE, ProjectStatus::Active).await; } hold(&pool, "h1", &[], None, &[], 30).await.unwrap(); sqlx::query("DELETE FROM signal WHERE token = 'gone'").execute(&pool).await.unwrap(); - sqlx::query("UPDATE trigger_activation SET status = $1 WHERE trigger = 'parked'") - .bind(ProjectStatus::Inactive.as_str()) - .execute(&pool) - .await - .unwrap(); - let look = hold(&pool, "h1", &holding(&["kept", "gone", "parked"]), None, &[], 30).await.unwrap(); - assert_eq!(look.kept, vec!["kept"]); + sqlx::query("UPDATE trigger_activation SET status = 'inactive' WHERE trigger = 'parked'").execute(&pool).await.unwrap(); + wipe(&pool, "wiped").await; + let look = hold(&pool, "h1", &holding(&["kept", "parked", "gone", "wiped"]), None, &[], 30).await.unwrap(); + let mut kept = look.kept.clone(); + kept.sort(); + assert_eq!(kept, vec!["kept", "parked"]); let mut ended = look.ended.clone(); ended.sort(); - assert_eq!(ended, vec!["gone", "parked"]); + assert_eq!(ended, vec!["gone", "wiped"]); assert!(look.taken.is_empty()); } diff --git a/crates/weft-broker/tests/lifecycle_writes.rs b/crates/weft-broker/tests/lifecycle_writes.rs index 7c5d040b..c2f04a1b 100644 --- a/crates/weft-broker/tests/lifecycle_writes.rs +++ b/crates/weft-broker/tests/lifecycle_writes.rs @@ -505,9 +505,12 @@ async fn a_tick_reports_the_projects_it_took_on(pool: PgPool) { assert_eq!(claimed().await, vec![PROJECT], "a lapsed lease taken back is news"); } -/// A tick says when its supervisor next has something to look at: never -/// over a project that only declares infra, at the lapse of a sibling's -/// lease over one with infra nodes, and now once that project is its own. +/// A tick says when its supervisor next has something to do: never over a +/// project of its own whose infra runs (a machine saying how its unit +/// stands changed wakes it instead), at the lapse of a sibling's lease over +/// one with infra copies expected to run (the word that one changed may +/// have woken this replica, not the sibling; a stopped copy says nothing), +/// and now once a command waits on one of its own. #[sqlx::test] async fn a_tick_says_when_there_is_next_something_to_look_at(pool: PgPool) { schema(&pool).await; @@ -517,12 +520,19 @@ async fn a_tick_says_when_there_is_next_something_to_look_at(pool: PgPool) { assert!(!declared.owns_work && declared.others_lapse_in_secs.is_none(), "but gives nothing to look at: {declared:?}"); node_row(&pool, "running", serde_json::json!({})).await; + let running = sync_ownership(&pool, OWNER, &[]).await.unwrap(); + assert!(!running.owns_work && running.others_lapse_in_secs.is_none(), "infra that runs needs no look on a clock: {running:?}"); + lease(&pool, OTHER).await; let sibling = sync_ownership(&pool, OWNER, &[]).await.unwrap(); assert!(sibling.owned.is_empty() && !sibling.owns_work, "a sibling holds the lease"); - let lapse = sibling.others_lapse_in_secs.expect("its lease may lapse"); + let lapse = sibling.others_lapse_in_secs.expect("its lease over a project with infra copies may lapse"); assert!((3590..=3600).contains(&lapse), "the sibling's lease lapses in an hour: {lapse}"); + sqlx::query("UPDATE infra_node SET status = 'stopped'").execute(&pool).await.unwrap(); + let stopped = sync_ownership(&pool, OWNER, &[]).await.unwrap(); + assert!(stopped.others_lapse_in_secs.is_none(), "a stopped copy says nothing to wake for: {stopped:?}"); + command_of(&pool, PROJECT).await; lease(&pool, OWNER).await; let own = sync_ownership(&pool, OWNER, &[]).await.unwrap(); assert!(own.owns_work && own.others_lapse_in_secs.is_none(), "{own:?}"); diff --git a/crates/weft-cli/src/commands/daemon.rs b/crates/weft-cli/src/commands/daemon.rs index e08b9a01..7a29081b 100644 --- a/crates/weft-cli/src/commands/daemon.rs +++ b/crates/weft-cli/src/commands/daemon.rs @@ -785,7 +785,7 @@ fn edge_config(kept: Option) -> Result { #[derive(Debug, Clone, Default, PartialEq)] struct Levers { worker_idle_stop_seconds: Option, - workers: Option, + workers: Option, edge: Option, } @@ -835,10 +835,12 @@ fn install_config( internet_url: Option, previous: Levers, ) -> Result { + // Only what a person set is written: a lever left out follows weft's + // default (`LocalPlatform::worker_idle_stop`), so a changed default + // reaches the install at its next start. let idle_stop = match (previous.worker_idle_stop_seconds, std::env::var("WEFT_WORKER_IDLE_STOP_SECONDS").ok().filter(|v| !v.trim().is_empty())) { - (_, Some(v)) => v.trim().parse().map_err(|_| anyhow::anyhow!("WEFT_WORKER_IDLE_STOP_SECONDS='{v}' is not a whole number"))?, - (Some(kept), None) => kept, - (None, None) => 300, + (_, Some(v)) => Some(v.trim().parse().map_err(|_| anyhow::anyhow!("WEFT_WORKER_IDLE_STOP_SECONDS='{v}' is not a whole number"))?), + (kept, None) => kept, }; let workers = previous.workers.unwrap_or_default(); let edge = edge_config(previous.edge)?; @@ -1478,7 +1480,10 @@ mod tests { let file = dir.path().join("config.json"); std::fs::write(&file, serde_json::to_vec(&edited).unwrap()).unwrap(); let again = install_config(&install(None), ports, "weft-runtime:y".into(), "b".into(), None, Levers::read(&file).unwrap()).unwrap(); - assert_eq!(again.workers.min_instances, 1); + assert_eq!(again.workers.min_instances, Some(1)); + assert_eq!(again.worker_settings().concurrency, weft_platform_traits::WorkerSettings::default().concurrency, "what the file leaves out follows weft's default"); + let written = serde_json::to_value(&closed).unwrap(); + assert!(written.get("workers").is_none() && written["platform"].get("workerIdleStopSeconds").is_none(), "a lever nobody set is not written"); assert_eq!(again.edge.trusted_proxy_hops, ProxyHops { public: 2, outside: 1, domains: 0 }); assert_eq!(again.platform, PlatformConfig::Local(LocalPlatform { runtime_image: "weft-runtime:y".into(), ..match closed.platform { PlatformConfig::Local(l) => l, _ => unreachable!() } })); } diff --git a/crates/weft-cli/src/commands/deactivate.rs b/crates/weft-cli/src/commands/deactivate.rs index 85f8113e..569a70e9 100644 --- a/crates/weft-cli/src/commands/deactivate.rs +++ b/crates/weft-cli/src/commands/deactivate.rs @@ -43,7 +43,7 @@ pub fn prompt_trigger_deactivation( drain_timeout_secs: Option, ) -> anyhow::Result { // Mode resolution priority: explicit --mode flag > interactive - // prompt (human terminal only) > error (a script, or `--json`). + // prompt (human terminal only) > `wipe` (a script, or `--json`). let scripted = json || !crate::prompt::is_interactive(); let mode = match mode { Some(m) => DeactivationMode::parse(m).ok_or_else(|| { @@ -294,8 +294,8 @@ fn prompt_grace() -> anyhow::Result { fn prompt_mode() -> anyhow::Result { println!("Choose preservation mode for in-flight signals:"); println!(" 1) wipe drop all signals, cancel suspended runs (fully fresh on reactivate)"); - println!(" 2) hibernate keep signals; hide pending tasks from extension; park late submissions"); - println!(" 3) park keep signals visible; queue new submissions for reactivate"); + println!(" 2) hibernate like park for a grace window, questions hidden; after it, nothing new is taken"); + println!(" 3) park nothing is dropped: calls, answers and the triggers' own fires wait and run once they are back"); let line = crate::prompt::prompt_line("Enter 1, 2, or 3: ", "--mode wipe | hibernate | park")?; Ok(match line.as_str() { "1" | "wipe" => DeactivationMode::Wipe, diff --git a/crates/weft-cli/src/commands/executions.rs b/crates/weft-cli/src/commands/executions.rs index 798f0b12..b8b04354 100644 --- a/crates/weft-cli/src/commands/executions.rs +++ b/crates/weft-cli/src/commands/executions.rs @@ -56,6 +56,12 @@ async fn executions_page( if let Some(tag) = &filter.tag { path.push_str(&format!("&tag={}", query_escaped(tag))); } + if let Some(through) = &filter.through { + path.push_str(&format!("&node={}", query_escaped(through))); + } + if let Some(search) = &filter.search { + path.push_str(&format!("&search={}", query_escaped(search))); + } serde_json::from_value(client.get_json(&path).await?).context("read the executions listing") } @@ -78,6 +84,10 @@ pub struct ListFilter { pub instance: Option, /// A tag the run carries. pub tag: Option, + /// A node that fired in the run. + pub through: Option, + /// Words the finished run carried among what it recorded. + pub search: Option, } pub async fn list(ctx: Ctx, filter: ListFilter) -> anyhow::Result<()> { diff --git a/crates/weft-cli/src/commands/infra.rs b/crates/weft-cli/src/commands/infra.rs index 07b0f227..eb703696 100644 --- a/crates/weft-cli/src/commands/infra.rs +++ b/crates/weft-cli/src/commands/infra.rs @@ -57,11 +57,10 @@ pub enum InfraAction { /// `--mode` or `--grace` is always sent; without one, the dispatcher /// says when it needs one (a trigger reading this infra is on), and /// then a TTY is asked and `--json` or a script fails naming the flags -/// (see `post_with_trigger_choice`). There is NO auto-reactivate: a user-triggered -/// upgrade leaves the project deactivated, and the user clicks Activate -/// when ready. Automatic reactivation belongs only to the autonomous -/// health-recovery path (deactivate -> fix infra -> reactivate with no -/// human present), not to a verb the user invoked themselves. +/// (see `post_with_trigger_choice`). The triggers a stop, terminate or +/// upgrade takes down come back on by themselves once their infra is up +/// again (a start, the upgrade's own start leg); a trigger the person +/// switched off themselves stays off. #[derive(Default, Clone)] pub struct InfraOpts { pub trigger: super::deactivate::TriggerChoiceFlags, diff --git a/crates/weft-cli/src/main.rs b/crates/weft-cli/src/main.rs index c592f32d..71a243cd 100644 --- a/crates/weft-cli/src/main.rs +++ b/crates/weft-cli/src/main.rs @@ -407,7 +407,9 @@ enum Cmd { }, /// Deactivate a registered project. Choose --mode wipe, hibernate, /// or park explicitly in scripts; a terminal prompts when it is omitted. - /// Wipe cancels suspended work; hibernate and park preserve it. + /// Park keeps everything that arrives and runs it once the triggers are + /// back; hibernate does that for a grace window; wipe drops it all and + /// cancels the runs waiting on the triggers. /// /// `--running-policy` controls how in-flight executions are /// handled: `cancel` (the default) kills running executions and @@ -706,6 +708,17 @@ enum Cmd { /// Only runs carrying this tag (`ctx.tag_execution`). #[arg(long)] tag: Option, + /// Only runs in which this node fired, wherever it sits in the + /// program (`--node` is the node that started the run). Spelled + /// the way `weft events` names nodes. + #[arg(long, value_name = "node")] + through: Option, + /// Only finished runs that carried every one of these words + /// somewhere in what they recorded: the trigger's input, what + /// every node sent on, an error, a log line. An email, an order + /// id, a phrase in quotes. A run still going is not searched yet. + #[arg(long, value_name = "words")] + search: Option, }, /// Print a past execution's events in order, one line each: /// time, kind, node, and a short summary of the value or error. @@ -1020,10 +1033,12 @@ impl From for weft_core::activation::ActivationScope { /// for. #[derive(Debug, clap::Args, Default, Clone)] struct TriggerDeactivationOpts { - /// Preservation mode for active triggers: wipe | hibernate | park. + /// What happens to work arriving while the triggers are down: park + /// keeps it all and runs it once they are back, hibernate does that for + /// a grace window, wipe drops it. #[arg(long, value_name = "wipe|hibernate|park")] mode: Option, - /// Hibernate grace window in minutes (only meaningful with + /// How long hibernate keeps taking work, in minutes (only with /// --mode hibernate). Default 15. #[arg(long, value_name = "minutes")] grace: Option, @@ -1830,11 +1845,11 @@ async fn run(cli: Cli) -> anyhow::Result<()> { Cmd::Login { target, key_stdin } => commands::target::login(ctx, target, key_stdin).await, Cmd::Logout { target } => commands::target::logout(ctx, target).await, Cmd::Token { action } => commands::token::run(ctx, action.into()).await, - Cmd::Executions { limit, project, phase, node, since, offset, status, instance, tag } => { + Cmd::Executions { limit, project, phase, node, since, offset, status, instance, tag, through, search } => { commands::executions::list( ctx, commands::executions::ListFilter { - limit, offset, project, phase, node, since, status, instance, tag, + limit, offset, project, phase, node, since, status, instance, tag, through, search, }, ) .await diff --git a/crates/weft-core/src/context/program_calls.rs b/crates/weft-core/src/context/program_calls.rs index 5950f457..51406bde 100644 --- a/crates/weft-core/src/context/program_calls.rs +++ b/crates/weft-core/src/context/program_calls.rs @@ -25,7 +25,7 @@ use crate::instance::InstanceId; use crate::infra::{InfraNodeStatus, TerminateDisks}; use crate::instance_door::ValuesChanged; use crate::program::{ - CleanOutcome, ConnectionsForgotten, CostFilter, CostRecord, ExecutionPage, InfraCopy, InfraStartAnswer, InstanceHoldings, + CleanOutcome, ConnectionsForgotten, CostFilter, CostRecord, ExecutionPage, InfraCopy, InfraDownAnswer, InfraStartAnswer, InstanceHoldings, MintedInstanceToken, PaidBy, ProgramCall, RunFilter, RunsCounted, TokensRevoked, MAX_RUNS_PAGE, }; use crate::running_policy::{DeactivateSpec, RunningPolicy}; @@ -207,29 +207,31 @@ impl InfraCalls<'_> { /// Scale the copy down, keeping its disk. `spec` says what happens to /// the triggers reading it and the runs using it; `stop_self` whether - /// this run goes too when it uses the copy. - pub async fn stop(self, spec: DeactivateSpec, stop_self: StopSelf) -> WeftResult<()> { + /// this run goes too when it uses the copy. Answers whether there was + /// a copy to take down ([`InfraDownAnswer`]), so a program can tell an + /// instance with no copy (a mistyped id, one terminated before) apart. + pub async fn stop(self, spec: DeactivateSpec, stop_self: StopSelf) -> WeftResult { let call = ProgramCall::InfraStop { node: self.node, instance: instance_id(&self.instance)?, spec }; - self.ctx.program_call::(call, stop_self).await.map(drop) + self.ctx.program_call(call, stop_self).await } /// Delete the copy and its disks, keeping the ones its node lists in /// `keepOnTerminate` (a later start of the same copy finds them /// again). Same `spec` and `stop_self` as [`Self::stop`]. - pub async fn terminate(self, spec: DeactivateSpec, stop_self: StopSelf) -> WeftResult<()> { + pub async fn terminate(self, spec: DeactivateSpec, stop_self: StopSelf) -> WeftResult { self.take_away(spec, stop_self, TerminateDisks::KeepListed).await } /// Delete the copy and every one of its disks, the ones listed in /// `keepOnTerminate` too: the copy's owner is going for good (a /// instance being wiped). Same `spec` and `stop_self` as [`Self::stop`]. - pub async fn wipe(self, spec: DeactivateSpec, stop_self: StopSelf) -> WeftResult<()> { + pub async fn wipe(self, spec: DeactivateSpec, stop_self: StopSelf) -> WeftResult { self.take_away(spec, stop_self, TerminateDisks::DeleteAll).await } - async fn take_away(self, spec: DeactivateSpec, stop_self: StopSelf, disks: TerminateDisks) -> WeftResult<()> { + async fn take_away(self, spec: DeactivateSpec, stop_self: StopSelf, disks: TerminateDisks) -> WeftResult { let call = ProgramCall::InfraTerminate { node: self.node, instance: instance_id(&self.instance)?, spec, disks }; - self.ctx.program_call::(call, stop_self).await.map(drop) + self.ctx.program_call(call, stop_self).await } /// The copy's state now, or `None` when it was never started (or was diff --git a/crates/weft-core/src/infra/resolve.rs b/crates/weft-core/src/infra/resolve.rs index b080b79e..2ec8c054 100644 --- a/crates/weft-core/src/infra/resolve.rs +++ b/crates/weft-core/src/infra/resolve.rs @@ -60,6 +60,7 @@ impl NodeRef { /// resource name on every platform: lowercase letters, digits and `-`, /// starting with a letter, at most 40 characters, so a platform may /// add a unit or volume name after it and stay under 63. + // SYNC: the `wi-` prefix <-> deploy/terraform/gcp/machines.tf (the machine events' filter) pub fn resource_base(&self) -> String { let digest = Sha256::digest(format!("{}|{}|{}", self.project, self.node, self.copy_id).as_bytes()); let hex: String = digest.iter().take(6).map(|b| format!("{b:02x}")).collect(); diff --git a/crates/weft-core/src/infra/status.rs b/crates/weft-core/src/infra/status.rs index ecd926d9..7fe1e0ca 100644 --- a/crates/weft-core/src/infra/status.rs +++ b/crates/weft-core/src/infra/status.rs @@ -71,6 +71,7 @@ impl InfraNodeStatus { /// the health loop should observe. Used by the supervisor's /// health tick: a node mid-apply or mid-stop has no SLO; only /// `Running` / `Flaky` does. + // SYNC: expects_running_units <-> weft_broker_client::lifecycle_command::runs_infra pub fn expects_running_units(self) -> bool { matches!(self, Self::Running | Self::Flaky) } diff --git a/crates/weft-core/src/node_test.rs b/crates/weft-core/src/node_test.rs index 09d544e2..1a552c42 100644 --- a/crates/weft-core/src/node_test.rs +++ b/crates/weft-core/src/node_test.rs @@ -2471,10 +2471,13 @@ impl ContextHandle for TestHandle { async fn program_call(&self, call: crate::program::ProgramCall, stop_self: crate::tag::StopSelf, _call_index: u32) -> WeftResult { let name = call.journal_name(); // A call that only acts, left unanswered by the test, answers the - // way the runtime answers one it carried out: a start is queued, - // a change of values set nothing up again. + // way the runtime answers one it carried out: a start is queued, a + // copy taken down, a change of values set nothing up again. let unanswered = match call { crate::program::ProgramCall::InfraStart { .. } => serde_json::to_value(crate::program::InfraStartAnswer::Started), + crate::program::ProgramCall::InfraStop { .. } | crate::program::ProgramCall::InfraTerminate { .. } => { + serde_json::to_value(crate::program::InfraDownAnswer::TakenDown) + } crate::program::ProgramCall::ValuesChange { .. } | crate::program::ProgramCall::ValuesForget { .. } => { serde_json::to_value(crate::instance_door::ValuesChanged::default()) } diff --git a/crates/weft-core/src/program.rs b/crates/weft-core/src/program.rs index 378993a5..42b907f7 100644 --- a/crates/weft-core/src/program.rs +++ b/crates/weft-core/src/program.rs @@ -206,6 +206,18 @@ impl InfraStartAnswer { } } +/// What `ProgramCall::InfraStop` and `ProgramCall::InfraTerminate` +/// answer: the copy was taken down by this call, it was already down (or +/// on its way down), or there is no copy of that node for that instance, +/// which is also what a copy terminated earlier reads as. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "copy", rename_all = "snake_case")] +pub enum InfraDownAnswer { + TakenDown, + AlreadyDown, + NoCopy, +} + /// What `ProgramCall::ConnectionsForget` answers: how many connections /// went. #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] @@ -290,7 +302,7 @@ impl InstanceHoldings { crate::wire_enum! { /// Where a run stands, as a filter asks for it: each word is the /// [`ExecutionSummary::status`] a run reads in a listing. - // SYNC: RunStatus <-> crates/weft-dispatcher/src/journal/postgres.rs list_executions (status clause) + // SYNC: RunStatus <-> crates/weft-dispatcher/src/journal/postgres.rs list_executions (status clause), extension-vscode/src/sidebar/runSearch.ts STATUSES pub enum RunStatus { /// Not ended: no terminal event yet. A run parked on a wait has /// not ended either, so it is here too. diff --git a/crates/weft-dispatcher/src/activation_store.rs b/crates/weft-dispatcher/src/activation_store.rs index 65ce552b..b679b757 100644 --- a/crates/weft-dispatcher/src/activation_store.rs +++ b/crates/weft-dispatcher/src/activation_store.rs @@ -68,10 +68,12 @@ pub enum LifecycleWrite { /// signals still registered. #[derive(Debug, Clone, Copy)] pub enum SignalsGoing { - /// None (a park or a hibernate keeps them for the reactivation). The - /// ones the written activations govern are read in the same - /// transaction, so a reactivation committing after it cannot slip - /// its fresh signals into the listener cleanup. + /// None, and the listener keeps holding them: a parked or + /// hibernating trigger goes on listening, and what it hears waits for + /// it to be back on (`crate::arrival`). Once a hibernation's grace + /// window ends, the listener lets go of them (`crate::reaper`'s + /// `hibernations`), and the holders of its held ones already have + /// (their claims read the window). Kept, /// Every signal of the project, except the ones `except`'s run /// holds (the run that asked for the take-down). @@ -214,26 +216,76 @@ pub trait ActivationStoreOps: Send + Sync { async fn list_deactivating(&self) -> anyhow::Result>; } +/// Whether a `trigger_activation` row (`a`) is a hibernation still in its +/// grace window: going or gone down, taking work until a deadline. +// SYNC: IN_GRACE_WINDOW_SQL <-> crate::arrival::Standing::arrival, weft_broker_client::protocol::ACTIVATION_LISTENS +const IN_GRACE_WINDOW_SQL: &str = + "a.accepting_fires AND a.fires_deadline_unix IS NOT NULL AND a.status IN ('inactive', 'deactivating')"; + +/// The hibernations whose grace window is over and still take work, with +/// their project. Over by the database's clock, the one the listening +/// condition (`ACTIVATION_LISTENS`) reads. +pub async fn grace_windows_over(pool: &PgPool) -> anyhow::Result> { + let rows = sqlx::query_as(&format!( + "SELECT a.project_id, a.trigger, a.instance_id FROM trigger_activation a \ + WHERE {IN_GRACE_WINDOW_SQL} AND a.fires_deadline_unix < EXTRACT(EPOCH FROM NOW())::BIGINT" + )) + .fetch_all(pool) + .await?; + project_keys(rows) +} + +/// End the hibernations of `keys` of `project_id` whose grace window is +/// over: they stop taking work. The arrival rule and the listening +/// condition already read the deadline against the clock; the flip is what +/// is announced (holder sizing, the routes). One reactivated meanwhile no +/// longer matches and is left alone. Answers how many it ended. +pub async fn end_grace_windows(pool: &PgPool, project_id: uuid::Uuid, keys: &[ActivationKey]) -> anyhow::Result { + let (triggers, instances) = key_arrays(keys); + Ok(sqlx::query(&format!( + "UPDATE trigger_activation a SET accepting_fires = FALSE, updated_at = EXTRACT(EPOCH FROM NOW())::BIGINT \ + WHERE a.project_id = $1 AND {KEYS_JOIN} \ + AND {IN_GRACE_WINDOW_SQL} AND a.fires_deadline_unix < EXTRACT(EPOCH FROM NOW())::BIGINT" + )) + .bind(project_id) + .bind(&triggers) + .bind(&instances) + .execute(pool) + .await? + .rows_affected()) +} + +/// When the earliest grace window still open ends, if one is. +pub async fn next_grace_end(pool: &PgPool) -> anyhow::Result> { + Ok(sqlx::query_scalar(&format!("SELECT MIN(a.fires_deadline_unix) FROM trigger_activation a WHERE {IN_GRACE_WINDOW_SQL}")) + .fetch_one(pool) + .await?) +} + pub type ActivationStore = Arc; /// Whether an activation of `key` landing at `lifecycle` leaves its row -/// behind. An instance's copy of a trigger wiped (inactive and refusing -/// fires) is the same as one never activated, so its row goes: nothing -/// lists the instance any more, and activating it again starts fresh. The -/// shared copy keeps its row, which is what the project's status reads. +/// behind. An instance's copy of a trigger wiped is the same as one never +/// activated, so its row goes: nothing lists the instance any more, and +/// activating it again starts fresh. The shared copy keeps its row, which +/// is what the project's status reads. fn wipe_forgets(key: &ActivationKey, lifecycle: &ActivationLifecycle) -> bool { - key.instance().is_some() && lands_wiped(lifecycle) + key.instance().is_some() && forgets_instance_rows(lifecycle) } -/// Whether `lifecycle` is a wipe (inactive, refusing fires): the half of -/// [`wipe_forgets`] about where a row lands. -fn lands_wiped(lifecycle: &ActivationLifecycle) -> bool { - lifecycle.status == ProjectStatus::Inactive && !lifecycle.accepting_fires +/// Whether an instance's row landing at `lifecycle` goes ([`wipe_forgets`] +/// with the owner left out): a wipe ([`ActivationLifecycle::is_wiped`]), +/// unless it went down with its infra, whose row is what brings it back +/// once that infra is up again. +fn forgets_instance_rows(lifecycle: &ActivationLifecycle) -> bool { + lifecycle.is_wiped() && lifecycle.went_down.map(|w| w.with) != Some(weft_broker_client::activation::DownWith::Infra) } /// The same test as [`wipe_forgets`], over a `trigger_activation` row /// (`a`) landing at status `$1`. -const WIPE_FORGETS_SQL: &str = "a.instance_id IS NOT NULL AND $1 = 'inactive' AND NOT a.accepting_fires"; +// SYNC: WIPE_FORGETS_SQL <-> wipe_forgets, weft_broker_client::activation::ActivationLifecycle::is_wiped, 'infra' = DownWith::Infra +const WIPE_FORGETS_SQL: &str = "a.instance_id IS NOT NULL AND $1 = 'inactive' AND NOT a.accepting_fires \ + AND a.fires_deadline_unix IS NULL AND a.went_down_with IS DISTINCT FROM 'infra'"; /// The `trigger_activation` table: one row per trigger per owner that /// was ever activated. The canonical CREATE lives here, edited in place; @@ -258,6 +310,22 @@ pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { fires_visible_to_consumers BOOLEAN NOT NULL, fires_deadline_unix BIGINT, drain_deadline_unix BIGINT, + -- Why it is down when the infra it reads took it down rather + -- than a person: 'health' (the health loop parked it) or + -- 'infra' (an infra verb took it down with its copies). NULL + -- otherwise. The copy coming back brings it back. + -- SYNC: the spellings <-> weft_broker_client::activation::DownWith + went_down_with TEXT, + -- When it went down so: a copy it reads that was applied after + -- this has come back up since, which is what brings it back. + went_down_at_unix BIGINT, + -- What `went_down_with` replaced, read by nothing here: it stays + -- for the release that stopped reading it, kept equal to + -- `went_down_with = 'health'` on every write, so a replica still + -- on the release before reads the health loop's parks right + -- while the new one rolls out (the seed below carries the parks + -- it made before the other way). Dropped in the release after, + -- with the seed. deactivated_by_health BOOLEAN NOT NULL DEFAULT FALSE, -- The trigger-setup run of the activation in flight; one run -- sets up every activation a verb names, so rows share it. @@ -274,21 +342,28 @@ pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { ON trigger_activation (activating_execution_id) WHERE activating_execution_id IS NOT NULL"#, r#"CREATE INDEX IF NOT EXISTS trigger_activation_transitional ON trigger_activation (status) WHERE status IN ('activating', 'deactivating')"#, - // An activation changing status, coming or going can start or stop - // the holding of its held signals without any signal row changing - // (a parked or hibernated trigger woken, a wiped instance's row + // An activation changing status, whether it takes work or until + // when, coming or going can start or stop the holding of its held + // signals without any signal row changing (a trigger woken, a + // hibernation's grace window set or ending, a wiped instance's row // forgotten, which a signal with no activation reads as active), so - // each wakes the holder sizing too. The function is the journal - // group's, which applies first. + // each wakes the holder sizing too, and the hibernation sweep + // (`crate::reaper`), which then knows when the next window ends. The + // function is the journal group's, which applies first. r#"DROP TRIGGER IF EXISTS trigger_activation_held_on_status ON trigger_activation"#, r#"CREATE TRIGGER trigger_activation_held_on_status - AFTER UPDATE OF status ON trigger_activation + AFTER UPDATE OF status, accepting_fires, fires_deadline_unix ON trigger_activation FOR EACH ROW - WHEN (NEW.status IS DISTINCT FROM OLD.status) + WHEN (NEW.status IS DISTINCT FROM OLD.status + OR NEW.accepting_fires IS DISTINCT FROM OLD.accepting_fires + OR NEW.fires_deadline_unix IS DISTINCT FROM OLD.fires_deadline_unix) EXECUTE FUNCTION signal_held_notify()"#, - // An activation's status is part of how its routes read (only an - // active one takes a caller), so a change tells every dispatcher - // to read its tenant's routes again (`crate::held::Held::routes`). + // An activation's status, whether it takes work while off and until + // when, are part of how its routes read (an active one serves a + // caller, a parked one holds them, until its grace window ends: + // `crate::arrival`), so a change tells every dispatcher to read its + // tenant's routes again (`crate::held::Held::routes`), and wakes the + // callers held there. // SYNC: 'weft_routes' <-> crate::held::ROUTES_CHANNEL r#"CREATE OR REPLACE FUNCTION trigger_activation_routes_notify() RETURNS trigger AS $$ DECLARE @@ -305,22 +380,59 @@ pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { $$ LANGUAGE plpgsql"#, r#"DROP TRIGGER IF EXISTS trigger_activation_routes_on_status ON trigger_activation"#, r#"CREATE TRIGGER trigger_activation_routes_on_status - AFTER UPDATE OF status ON trigger_activation + AFTER UPDATE OF status, accepting_fires, fires_deadline_unix ON trigger_activation FOR EACH ROW - WHEN (NEW.status IS DISTINCT FROM OLD.status) + WHEN (NEW.status IS DISTINCT FROM OLD.status + OR NEW.accepting_fires IS DISTINCT FROM OLD.accepting_fires + OR NEW.fires_deadline_unix IS DISTINCT FROM OLD.fires_deadline_unix) EXECUTE FUNCTION trigger_activation_routes_notify()"#, r#"DROP TRIGGER IF EXISTS trigger_activation_routes_on_row ON trigger_activation"#, r#"CREATE TRIGGER trigger_activation_routes_on_row AFTER INSERT OR DELETE ON trigger_activation FOR EACH ROW EXECUTE FUNCTION trigger_activation_routes_notify()"#, + // While a release rolls out, a replica on the release before writes + // `deactivated_by_health` and never the cause columns: its activate + // leaves an old cause behind, its health park sets none, and a park + // after this release's health park clears only the old column. This + // keeps the cause in step with what such a replica writes, the old + // column being right about the health loop's parks either way. Goes + // with the old column. + r#"CREATE OR REPLACE FUNCTION trigger_activation_old_health_flag() RETURNS trigger AS $$ + BEGIN + IF NEW.status = 'activating' THEN + NEW.went_down_with := NULL; + NEW.went_down_at_unix := NULL; + ELSIF NEW.deactivated_by_health AND NEW.went_down_with IS NULL THEN + NEW.went_down_with := 'health'; + NEW.went_down_at_unix := EXTRACT(EPOCH FROM NOW())::BIGINT; + ELSIF NOT NEW.deactivated_by_health AND NEW.went_down_with = 'health' THEN + NEW.went_down_with := NULL; + NEW.went_down_at_unix := NULL; + END IF; + RETURN NEW; + END; + $$ LANGUAGE plpgsql"#, + r#"DROP TRIGGER IF EXISTS trigger_activation_old_health_flag ON trigger_activation"#, + r#"CREATE TRIGGER trigger_activation_old_health_flag + BEFORE INSERT OR UPDATE ON trigger_activation + FOR EACH ROW + EXECUTE FUNCTION trigger_activation_old_health_flag()"#, r#"DROP TRIGGER IF EXISTS trigger_activation_held_on_row ON trigger_activation"#, r#"CREATE TRIGGER trigger_activation_held_on_row AFTER INSERT OR DELETE ON trigger_activation FOR EACH ROW EXECUTE FUNCTION signal_held_notify()"#, ], - seed: &[], + seed: &[ + // A health-loop park the release before wrote sets only the old + // column: carried to the cause this release reads (its moment + // unknown, so the row's last write) by the boot that applies this + // release, so the health loop's auto-recover still brings it back. + // Goes with the old column. + r#"UPDATE trigger_activation SET went_down_with = 'health', went_down_at_unix = updated_at + WHERE deactivated_by_health AND went_down_with IS NULL"#, + ], }; #[derive(Clone)] @@ -342,14 +454,15 @@ type LifecycleRow = ( bool, Option, Option, - bool, + Option, + Option, Option, Option, Option, ); fn row_to_activation(row: LifecycleRow) -> anyhow::Result { - let (trigger, instance, status, accepting, visible, deadline, drain, by_health, execution_id, program, version) = row; + let (trigger, instance, status, accepting, visible, deadline, drain, went_down_with, went_down_at, execution_id, program, version) = row; let instance = instance.map(InstanceId::new).transpose().map_err(|e| anyhow::anyhow!("trigger_activation.instance_id: {e}"))?; Ok(Activation { key: ActivationKey::new(trigger, Owner::from_instance(instance)), @@ -360,7 +473,15 @@ fn row_to_activation(row: LifecycleRow) -> anyhow::Result { fires_visible_to_consumers: visible, fires_deadline_unix: deadline, drain_deadline_unix: drain, - deactivated_by_health: by_health, + went_down: match (went_down_with, went_down_at) { + (None, None) => None, + (Some(cause), Some(at_unix)) => Some(weft_broker_client::activation::WentDown { + with: weft_broker_client::activation::DownWith::parse(&cause) + .ok_or_else(|| anyhow::anyhow!("unknown trigger_activation.went_down_with '{cause}'"))?, + at_unix, + }), + (cause, at) => anyhow::bail!("trigger_activation.went_down_with={cause:?} and went_down_at_unix={at:?} must be set together"), + }, activating_execution_id: execution_id, }, program: program.map(serde_json::from_value).transpose()?, @@ -412,7 +533,7 @@ impl ActivationStoreOps for PostgresActivationStore { async fn list(&self, project_id: uuid::Uuid) -> anyhow::Result> { let rows: Vec = sqlx::query_as( "SELECT trigger, instance_id, status, accepting_fires, fires_visible_to_consumers, \ - fires_deadline_unix, drain_deadline_unix, deactivated_by_health, activating_execution_id, \ + fires_deadline_unix, drain_deadline_unix, went_down_with, went_down_at_unix, activating_execution_id, \ activation_program, activation_version \ FROM trigger_activation WHERE project_id = $1 ORDER BY trigger, instance_id NULLS FIRST", ) @@ -446,7 +567,7 @@ impl ActivationStoreOps for PostgresActivationStore { } let before: Vec = sqlx::query_as(&format!( "SELECT a.trigger, a.instance_id, a.status, a.accepting_fires, a.fires_visible_to_consumers, \ - a.fires_deadline_unix, a.drain_deadline_unix, a.deactivated_by_health, a.activating_execution_id, \ + a.fires_deadline_unix, a.drain_deadline_unix, a.went_down_with, a.went_down_at_unix, a.activating_execution_id, \ a.activation_program, a.activation_version \ FROM trigger_activation a WHERE a.project_id = $1 AND {KEYS_JOIN} FOR UPDATE" )) @@ -463,9 +584,9 @@ impl ActivationStoreOps for PostgresActivationStore { sqlx::query( "INSERT INTO trigger_activation \ (project_id, trigger, instance_id, status, accepting_fires, fires_visible_to_consumers, \ - fires_deadline_unix, drain_deadline_unix, deactivated_by_health, activating_execution_id, \ + fires_deadline_unix, drain_deadline_unix, went_down_with, went_down_at_unix, activating_execution_id, \ heartbeat_unix, activation_program, activation_version, updated_at) \ - SELECT $1, t, NULLIF(m, ''), $4, $5, $6, NULL, NULL, FALSE, $7, $8, NULL, NULL, $8 \ + SELECT $1, t, NULLIF(m, ''), $4, $5, $6, NULL, NULL, NULL, NULL, $7, $8, NULL, NULL, $8 \ FROM unnest($2::text[], $3::text[]) AS k(t, m) \ ON CONFLICT (project_id, trigger, instance_id) DO UPDATE SET \ status = EXCLUDED.status, \ @@ -473,6 +594,8 @@ impl ActivationStoreOps for PostgresActivationStore { fires_visible_to_consumers = EXCLUDED.fires_visible_to_consumers, \ fires_deadline_unix = NULL, \ drain_deadline_unix = NULL, \ + went_down_with = NULL, \ + went_down_at_unix = NULL, \ deactivated_by_health = FALSE, \ activating_execution_id = EXCLUDED.activating_execution_id, \ heartbeat_unix = EXCLUDED.heartbeat_unix, \ @@ -506,7 +629,7 @@ impl ActivationStoreOps for PostgresActivationStore { let owned = "project_id = $1 AND activating_execution_id = $2 AND status = 'activating'"; // An instance's rows landing wiped go ([`wipe_forgets`]); the rest // move to `to`. - let mut ended: Vec<(String, Option)> = if lands_wiped(to) { + let mut ended: Vec<(String, Option)> = if forgets_instance_rows(to) { sqlx::query_as(&format!( "DELETE FROM trigger_activation WHERE {owned} AND instance_id IS NOT NULL RETURNING trigger, instance_id" )) @@ -520,7 +643,8 @@ impl ActivationStoreOps for PostgresActivationStore { let moved: Vec<(String, Option)> = sqlx::query_as( "UPDATE trigger_activation \ SET status = $1, accepting_fires = $2, fires_visible_to_consumers = $3, \ - fires_deadline_unix = $4, deactivated_by_health = $5, drain_deadline_unix = $6, \ + fires_deadline_unix = $4, went_down_with = $5, drain_deadline_unix = $6, \ + went_down_at_unix = $10, deactivated_by_health = ($5::text IS NOT DISTINCT FROM 'health'), \ activating_execution_id = NULL, \ activation_program = CASE WHEN $1 = 'active' THEN activation_program ELSE NULL END, \ activation_version = CASE WHEN $1 = 'active' THEN activation_version ELSE NULL END, \ @@ -532,11 +656,12 @@ impl ActivationStoreOps for PostgresActivationStore { .bind(to.accepting_fires) .bind(to.fires_visible_to_consumers) .bind(to.fires_deadline_unix) - .bind(to.deactivated_by_health) + .bind(to.went_down.map(|w| w.with.as_str())) .bind(to.drain_deadline_unix) .bind(crate::lease::now_unix()) .bind(project_id) .bind(execution_id) + .bind(to.went_down.map(|w| w.at_unix)) .fetch_all(&mut *tx) .await?; ended.extend(moved); @@ -576,7 +701,8 @@ impl ActivationStoreOps for PostgresActivationStore { restored += sqlx::query( "UPDATE trigger_activation \ SET status = $1, accepting_fires = $2, fires_visible_to_consumers = $3, \ - fires_deadline_unix = $4, deactivated_by_health = $5, drain_deadline_unix = $6, \ + fires_deadline_unix = $4, went_down_with = $5, drain_deadline_unix = $6, went_down_at_unix = $14, \ + deactivated_by_health = ($5::text IS NOT DISTINCT FROM 'health'), \ activating_execution_id = NULL, activation_program = $7, activation_version = $8, updated_at = $9 \ WHERE project_id = $10 AND activating_execution_id = $11 AND status = 'activating' \ AND trigger = $12 AND instance_id IS NOT DISTINCT FROM $13", @@ -585,7 +711,7 @@ impl ActivationStoreOps for PostgresActivationStore { .bind(before.accepting_fires) .bind(before.fires_visible_to_consumers) .bind(before.fires_deadline_unix) - .bind(before.deactivated_by_health) + .bind(before.went_down.map(|w| w.with.as_str())) .bind(before.drain_deadline_unix) .bind(activation.program.as_ref().map(sqlx::types::Json)) .bind(activation.source_version.as_deref()) @@ -594,6 +720,7 @@ impl ActivationStoreOps for PostgresActivationStore { .bind(execution_id) .bind(&activation.key.trigger) .bind(activation.key.instance().map(InstanceId::as_str)) + .bind(before.went_down.map(|w| w.at_unix)) .execute(&mut *tx) .await? .rows_affected(); @@ -679,7 +806,8 @@ impl ActivationStoreOps for PostgresActivationStore { sqlx::query(&format!( "UPDATE trigger_activation a \ SET status = $4, accepting_fires = $5, fires_visible_to_consumers = $6, \ - fires_deadline_unix = $7, deactivated_by_health = $8, drain_deadline_unix = $9, \ + fires_deadline_unix = $7, went_down_with = $8, drain_deadline_unix = $9, went_down_at_unix = $11, \ + deactivated_by_health = ($8::text IS NOT DISTINCT FROM 'health'), \ activating_execution_id = NULL, \ activation_program = CASE WHEN $4 IN ('inactive', 'deactivating') THEN NULL ELSE activation_program END, \ activation_version = CASE WHEN $4 IN ('inactive', 'deactivating') THEN NULL ELSE activation_version END, \ @@ -693,13 +821,14 @@ impl ActivationStoreOps for PostgresActivationStore { .bind(lifecycle.accepting_fires) .bind(lifecycle.fires_visible_to_consumers) .bind(lifecycle.fires_deadline_unix) - .bind(lifecycle.deactivated_by_health) + .bind(lifecycle.went_down.map(|w| w.with.as_str())) .bind(lifecycle.drain_deadline_unix) .bind(crate::lease::now_unix()) + .bind(lifecycle.went_down.map(|w| w.at_unix)) .execute(&mut *tx) .await?; let unlisten = match signals { - SignalsGoing::Kept => crate::journal::postgres::activation_signals(&mut *tx, project_id, keys).await?, + SignalsGoing::Kept => Vec::new(), SignalsGoing::Project { except } => { crate::journal::postgres::remove_project_signals_except(&mut *tx, project_id, except).await? } @@ -769,20 +898,25 @@ impl ActivationStoreOps for PostgresActivationStore { } async fn list_deactivating(&self) -> anyhow::Result> { - let rows: Vec<(uuid::Uuid, String, Option)> = sqlx::query_as( + let rows = sqlx::query_as( "SELECT project_id, trigger, instance_id FROM trigger_activation WHERE status = 'deactivating'", ) .fetch_all(&self.pool) .await?; - rows.into_iter() - .map(|(project, trigger, instance)| { - let instance = instance.map(InstanceId::new).transpose().map_err(anyhow::Error::msg)?; - Ok((project, ActivationKey::new(trigger, Owner::from_instance(instance)))) - }) - .collect() + project_keys(rows) } } +/// `(project_id, trigger, instance_id)` rows as the activations they key. +fn project_keys(rows: Vec<(uuid::Uuid, String, Option)>) -> anyhow::Result> { + rows.into_iter() + .map(|(project, trigger, instance)| { + let instance = instance.map(InstanceId::new).transpose().map_err(anyhow::Error::msg)?; + Ok((project, ActivationKey::new(trigger, Owner::from_instance(instance)))) + }) + .collect() +} + /// One fake row: the lifecycle, the heartbeat, and the code and version /// its setup recorded. #[cfg(any(test, feature = "test-helpers"))] diff --git a/crates/weft-dispatcher/src/api/execution.rs b/crates/weft-dispatcher/src/api/execution.rs index 53d2be09..68f1aca2 100644 --- a/crates/weft-dispatcher/src/api/execution.rs +++ b/crates/weft-dispatcher/src/api/execution.rs @@ -652,6 +652,17 @@ pub struct ListExecutionsParams { pub instance: Option, /// Only runs carrying this tag. pub tag: Option, + /// Only runs in which this node fired. + pub node: Option, + /// Only finished runs whose recorded values carry every word of this + /// (`crate::run_search`). + pub search: Option, + /// The page after the run that started at `before_started` with id + /// `before_execution` (the last run of the page before): a walk that + /// hands each page's last run back here reaches every run once, + /// whatever starts or ends meanwhile, which `offset` cannot promise. + pub before_started: Option, + pub before_execution: Option, } const DEFAULT_PAGE: u32 = 50; @@ -662,6 +673,11 @@ pub async fn list_executions( caller: CallerTenant, Query(params): Query, ) -> Result, StatusCode> { + let below = match (params.before_started, params.before_execution) { + (Some(started), Some(execution_id)) => Some((started, execution_id)), + (None, None) => None, + _ => return Err(StatusCode::BAD_REQUEST), + }; let query = ExecutionQuery { limit: params.limit.unwrap_or(DEFAULT_PAGE).clamp(1, MAX_PAGE), offset: params.offset.unwrap_or(0), @@ -673,7 +689,9 @@ pub async fn list_executions( status: params.status, instance: params.instance, tag: params.tag, - below: None, + node: params.node.filter(|n| !n.trim().is_empty()), + search: params.search.filter(|s| !s.trim().is_empty()), + below, }; let mut page = state .journal @@ -708,6 +726,8 @@ pub async fn latest_for_project( status: None, instance: None, tag: None, + node: None, + search: None, below: None, }; let mut page = state @@ -895,6 +915,8 @@ pub(crate) fn run_query( status: filter.status, instance: filter.instance.clone(), tag: filter.tag.clone(), + node: None, + search: None, below: None, } } diff --git a/crates/weft-dispatcher/src/api/infra.rs b/crates/weft-dispatcher/src/api/infra.rs index 6d845e82..e4d1bf66 100644 --- a/crates/weft-dispatcher/src/api/infra.rs +++ b/crates/weft-dispatcher/src/api/infra.rs @@ -678,13 +678,95 @@ pub(crate) async fn finish_sync( crate::api::project::reconcile_worker(state, id, running_policy, drain_timeout_secs) .await?; - // No auto-reactivate. An upgrade takes its triggers down before its - // stop leg, and a user-invoked - // upgrade intentionally leaves it deactivated (the user clicks - // Activate when ready). Automatic reactivation lives only in the - // autonomous health-recovery path (the supervisor's AutoRecover - // protocol -> dispatcher lifecycle_claimer -> activate_inner), where - // there is no human to click. + Ok(()) +} + +/// Switch back on, in every project, the triggers an infra verb took down +/// with the copies they read (`DownWith::Infra`), once each of those +/// copies runs again. Woken by any change of a copy's status +/// (`crate::reaper`'s `infra_returns`), so whichever way a copy came back +/// up (a `weft infra start`, an upgrade's start leg, a program's own +/// `ctx.infra(..).start()`) and whichever dispatcher heard it, the marks +/// in the rows are what it acts on: one that died midway leaves them for +/// the next pass. One a person took down since carries no mark and stays +/// down, and one whose other copies are still down waits for their start. +pub(crate) async fn bring_back_triggers_whose_infra_returned(state: DispatcherState) -> anyhow::Result { + // SYNC: 'infra' <-> weft_broker_client::activation::DownWith::Infra + let marked: Vec = sqlx::query_scalar( + "SELECT DISTINCT project_id FROM trigger_activation WHERE went_down_with = 'infra' AND status = 'inactive'", + ) + .fetch_all(&state.pg_pool) + .await?; + for id in marked { + bring_back_what_went_down_with_infra(&state, id).await?; + } + Ok(weft_task_store::drain::DrainStep::Done) +} + +/// [`bring_back_triggers_whose_infra_returned`] for project `id`. Owner by +/// owner, since one setup run is one owner's: one that cannot come back is +/// logged with its owner and the others still come back. +async fn bring_back_what_went_down_with_infra(state: &DispatcherState, id: uuid::Uuid) -> anyhow::Result<()> { + let Some(project) = state.projects.project(id).await? else { return Ok(()) }; + let deps = crate::api::project::compute_trigger_deps(&project); + let copies = crate::infra_node::statuses(&state.pg_pool, id).await?; + let mut by_owner: std::collections::BTreeMap, Vec> = Default::default(); + for activation in state.activations.list(id).await? { + let lifecycle = &activation.lifecycle; + let Some(went_down) = lifecycle.went_down.filter(|w| w.with == crate::take_down::DownWith::Infra) else { continue }; + if lifecycle.status != crate::activation_store::ProjectStatus::Inactive { + continue; + } + let reads: std::collections::HashSet = + deps.iter().filter(|(_, trigger)| *trigger == activation.key.trigger).map(|(infra, _)| infra.clone()).collect(); + match crate::api::project::missing_infra_nodes(state, id, &project, Some(&reads), activation.key.instance()).await { + Ok(missing) if missing.is_empty() => { + // Every copy it reads runs, and one came back up since it + // went down: the verb takes a trigger down before its copy, + // so a copy still running from before may be on its way + // down. Both moments are the database's clock. + let came_back = copies.iter().any(|row| { + row.applied_at_unix.is_some_and(|at| at >= went_down.at_unix) + && crate::api::project::copies_read(&project, Some(&reads), activation.key.instance()) + .iter() + .any(|(place, copy)| &row.node_id == place && Some(row.instance.as_ref()) == *copy) + }); + if came_back { + by_owner.entry(activation.key.instance().cloned()).or_default().push(activation.key.trigger) + } + } + Ok(_) => {} + Err((status, message)) => tracing::error!( + target: "weft_dispatcher::infra", + project_id = %id, trigger = %activation.key.trigger, instance = ?activation.key.instance(), + "could not tell whether a trigger's infra is back, so it stays down for now: {status}: {message}" + ), + } + } + for (instance, triggers) in by_owner { + tracing::info!(target: "weft_dispatcher::infra", project_id = %id, ?instance, ?triggers, "bringing back the triggers that went down with their infra"); + let request = weft_core::activation::ActivateRequest { + scope: weft_core::activation::ActivationScope { triggers: triggers.clone(), instance: instance.clone() }, + ..Default::default() + }; + match crate::api::project::activate_with(state, id, request, crate::api::project::ActivateAsker::Outside).await { + Ok(_) => {} + // Another dispatcher heard the same copy come back and is + // switching them on. + Err(crate::api::project::ActivateError::Refused(crate::activation_store::ClaimRefused::Claimed)) => { + tracing::info!(target: "weft_dispatcher::infra", project_id = %id, ?instance, ?triggers, "already being brought back"); + } + Err(error) => { + let (status, message) = <(axum::http::StatusCode, String)>::from(error); + tracing::error!( + target: "weft_dispatcher::infra", + project_id = %id, ?instance, ?triggers, + "the infra the triggers read is up, but they did not come back; tried again at the next change of a copy, \ + or `weft activate` switches them on: {status}: {message}" + ); + } + } + } Ok(()) } diff --git a/crates/weft-dispatcher/src/api/project.rs b/crates/weft-dispatcher/src/api/project.rs index 0d120b11..7363e3f3 100644 --- a/crates/weft-dispatcher/src/api/project.rs +++ b/crates/weft-dispatcher/src/api/project.rs @@ -975,27 +975,8 @@ pub(crate) async fn missing_infra_nodes( within: Option<&HashSet>, instance: Option<&weft_core::instance::InstanceId>, ) -> Result, (StatusCode, String)> { - // Per PLACE, spelled: an infra node inside a file included twice is - // two instances with two rows, and `within` names places the same - // way, so a run cut to one call waits on that call's instance alone. - // A per-instance node is looked up in `instance`'s copy: that is the - // one the run or the instance's triggers read. - // Which copy of which place (`None` for a per-instance place with no - // instance named, which has no copy to look at), then the project's - // copies as this dispatcher holds them (`crate::held`). - let mut wanted: Vec<(String, Option>)> = Vec::new(); - for place in weft_core::project::infra_places(project) { - let spelled = weft_core::project::address_of(project, &place.id, &place.path); - if within.is_some_and(|set| !set.contains(&spelled)) { - continue; - } - let per_instance = weft_core::project::is_per_instance(project, &place.id); - // Nobody named, so there is no copy to look at: activate's note - // over the whole project lands here (it leaves these out), and a - // run never does ([`require_run_infra`] refuses it first). - let copy = if per_instance { instance.map(Some) } else { Some(None) }; - wanted.push((spelled, copy)); - } + // The project's copies as this dispatcher holds them (`crate::held`). + let wanted = copies_read(project, within, instance); let missing_of = |copies: &[crate::infra_node::CopyStatus]| -> Vec { let mut missing = Vec::new(); for (place, copy) in &wanted { @@ -1032,6 +1013,34 @@ pub(crate) async fn missing_infra_nodes( Ok(missing_of(&fresh)) } +/// Which copy of which infra place a run or a trigger reads, the places +/// narrowed to `within` (see [`missing_infra_nodes`]): each place spelled +/// (an infra node inside a file included twice is two places with two +/// rows, and `within` names places the same way), with its copy: the +/// shared one (`Some(None)`), `instance`'s for a per-instance place, or +/// `None` for a per-instance place with no instance named, which has no +/// copy to look at. +pub(crate) fn copies_read<'a>( + project: &ProjectDefinition, + within: Option<&HashSet>, + instance: Option<&'a weft_core::instance::InstanceId>, +) -> Vec<(String, Option>)> { + let mut wanted = Vec::new(); + for place in weft_core::project::infra_places(project) { + let spelled = weft_core::project::address_of(project, &place.id, &place.path); + if within.is_some_and(|set| !set.contains(&spelled)) { + continue; + } + let per_instance = weft_core::project::is_per_instance(project, &place.id); + // Nobody named, so there is no copy to look at: activate's note + // over the whole project lands here (it leaves these out), and a + // run never does ([`require_run_infra`] refuses it first). + let copy = if per_instance { instance.map(Some) } else { Some(None) }; + wanted.push((spelled, copy)); + } + wanted +} + /// One infra copy a run or a trigger needs and that is not running. pub(crate) struct MissingCopy { /// The node, spelled. @@ -1704,6 +1713,8 @@ pub async fn status( status: None, instance: None, tag: None, + node: None, + search: None, below: None, }; let (program_runs, every_run) = (runs_of(Some(weft_core::context::Phase::Fire)), runs_of(None)); @@ -3153,9 +3164,12 @@ pub(crate) async fn activate_with( } let keys = resolve_scope(&project, &scope)?; // An activate turns on the triggers that are off and leaves the ones - // already on as they are: after an infra stop took down only the - // triggers reading it, this is what brings them back. A re-arm is the - // other way round: its triggers are on, and it sets them up again. + // already on (or being switched on) as they are: after an infra stop + // took down only the triggers reading it, this is what brings them + // back. One being switched off is not on: an activate on it is the + // change of mind that rolls its drain back (`resume_active`). A re-arm + // is the other way round: its triggers are on, and it sets them up + // again. let keys = match rearm { Some(_) => keys, None => { @@ -3165,7 +3179,12 @@ pub(crate) async fn activate_with( .await .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("activations: {e}")))? .into_iter() - .filter(|a| !is_down(&a.lifecycle)) + .filter(|a| { + matches!( + a.lifecycle.status, + crate::project_store::ProjectStatus::Active | crate::project_store::ProjectStatus::Activating + ) + }) .map(|a| a.key) .collect(); let off: Vec = keys.into_iter().filter(|k| !on.contains(k)).collect(); @@ -3940,7 +3959,8 @@ pub async fn execute_trigger_deactivation( spec.validate() .map_err(|m| (StatusCode::BAD_REQUEST, format!("triggerDeactivation: {m}")))?; let target = crate::take_down::TakeDownTarget::Activations(keys); - let existed = crate::take_down::take_down(state, id, &target, spec, false, None).await?; + // Down with the copies the verb takes down: their start brings it back. + let existed = crate::take_down::take_down(state, id, &target, spec, Some(crate::take_down::DownWith::Infra), None).await?; if !existed { return Err(( StatusCode::INTERNAL_SERVER_ERROR, @@ -3986,7 +4006,7 @@ pub async fn deactivate( let keys = keys_to_take_down(&rows, &scope)?; let target = crate::take_down::TakeDownTarget::Activations(keys); // user-initiated (the standalone Deactivate verb) - let existed = crate::take_down::take_down(&state, id, &target, &body.spec, false, None).await?; + let existed = crate::take_down::take_down(&state, id, &target, &body.spec, None, None).await?; if !existed { // Vanished between the gate and the deactivate (a concurrent // rm): same "no project I may see" answer, marker included. @@ -4331,7 +4351,7 @@ pub async fn deactivate_project( running_policy: crate::infra_lifecycle_command::RunningPolicy::Cancel, drain_timeout_secs: None, }; - crate::take_down::take_down(state, id, &crate::take_down::TakeDownTarget::WholeProject, &spec, false, None).await + crate::take_down::take_down(state, id, &crate::take_down::TakeDownTarget::WholeProject, &spec, None, None).await } /// `POST /projects/{id}/quiesce`: take the whole project down (every diff --git a/crates/weft-dispatcher/src/api/signal.rs b/crates/weft-dispatcher/src/api/signal.rs index 7a107eaa..38ba1f48 100644 --- a/crates/weft-dispatcher/src/api/signal.rs +++ b/crates/weft-dispatcher/src/api/signal.rs @@ -415,7 +415,7 @@ async fn fire_checked( if !routing.is_resume { refuse_gated_entry(&routing.auth_kind, "its /connect address")?; } - apply_lifecycle_gate(state, token, routing, payload).await + apply_lifecycle_gate(state, token, routing, payload, None).await } /// Refuse to start a run through an entry gated by a connection (its @@ -453,135 +453,84 @@ pub(crate) async fn fire_registered_signal( return Err((StatusCode::TOO_MANY_REQUESTS, format!("fire dropped: {} is reached", refused.reason.describe()))); } } - apply_lifecycle_gate(state, token, &routing, payload).await + apply_lifecycle_gate(state, token, &routing, payload, None).await } -/// One chokepoint for every external fire. Reads the project's -/// lifecycle (status + accepting/visible/deadline) and decides: +/// One chokepoint for every fire that reaches a signal (a call at the +/// door, a provider's push, an answer to a waiting run, an event a +/// listener picked up itself): the rule `crate::arrival` states, applied. /// -/// - **Live**: dispatch to listener for immediate processing. -/// - **Park**: append `payload` to `signal.parked_fires`. Drained on -/// reactivate by `drain_parked_fires`, which calls the exact same -/// `dispatch_listener_outcome` a live fire would. Entry signals -/// append on every fire; resume signals append iff the queue is -/// empty (first submission answers the suspension, later ones are -/// dropped as duplicates). -/// - **Refuse**: 410 Gone. Used when the project is wiped, the -/// hibernate deadline has expired, or status is fully Inactive -/// with `accepting_fires=false`. +/// - **Live**: dispatch to the listener for immediate processing. +/// - **Wait**: append `payload` to `signal.parked_fires`, under `id` (the +/// fire's identity when it can come back, a fresh one otherwise). +/// Drained when the trigger is back by `drain_parked_fires`, which +/// calls the exact same `dispatch_listener_outcome` a live fire would, +/// so the run is born on the version the trigger came back with. Entry +/// signals append on every fire; resume signals append iff the queue +/// is empty (the first submission answers the suspension, later ones +/// are dropped as duplicates). +/// - **Refused**: 410 Gone. /// -/// The function is signal-kind agnostic past the resume vs entry -/// queue-cap rule: park / refuse / dispatch work uniformly across -/// webhook, form, resume tokens. Stateful kinds (timer, sse) bypass -/// entirely via internal fire paths. -async fn apply_lifecycle_gate( +/// The function is signal-kind agnostic past the resume vs entry queue-cap +/// rule. +pub(crate) async fn apply_lifecycle_gate( state: &DispatcherState, token: &str, routing: &FireGateInfo, payload: Value, + id: Option<&str>, ) -> Result { - use crate::activation_store::ProjectStatus; - - // The governing activation is live: live fire. A signal no - // activation governs (a wait of a run started by hand) reads as live - // too: it has no activate/drain moment ever, so a fire parked here - // would strand the suspended run forever. - if matches!( - routing.status, - ProjectStatus::Active | ProjectStatus::Registered - ) { - return dispatch_listener_outcome( - state, - token, - routing.project_id, - &routing.tenant_id, - payload, - None, - ) - .await; - } - - // Past the deadline (hibernate-style grace expired): refuse, - // even if accepting_fires is still true on the row. We could - // also lazily flip accepting_fires=false when this triggers, - // but the gate is the cheapest place to evaluate the deadline - // and avoids a write per fire. - if let Some(deadline) = routing.fires_deadline_unix { - if (crate::lease::now_unix()) > deadline { - return Err(( - StatusCode::GONE, - "Project is not accepting requests. Please contact the project administrator.".into(), - )); + let parked = id.map(|id| ParkedRef { id, attempts: 0 }); + match routing.standing().arrival(crate::lease::now_unix()) { + crate::arrival::Arrival::Live => { + dispatch_listener_outcome(state, token, routing.project_id, &routing.tenant_id, payload, parked).await } - } - - // Not Active but still accepting fires. Append to the queue. - // Cases: - // - Activating: TriggerSetup is mid-flight; the listener may - // not have every signal registered yet. The drain at the - // end of activate replays everything queued here. - // - Inactive in park / hibernate-in-grace mode. - // - Deactivating toward park / hibernate. - // Reactivate's drain replays each element through - // dispatch_listener_outcome in FIFO order. - if routing.accepting_fires { - // `id` distinguishes this queued fire from any other fire on - // the same token, even when bodies are identical. The drain - // uses it as the task-table dedup nonce so a crash between - // task-insert and head-pop collapses the retry back to one - // task (same id, same dedup_key) while two genuinely - // distinct fires (different ids) produce two executions. - let entry = ParkedFire { - id: uuid::Uuid::new_v4().to_string(), - payload, - received_at_unix: crate::lease::now_unix(), - attempts: 0, - not_before_unix: 0, - instance_gap: None, - }; - // The shared append names its refusal; never swallow one under a - // 200. A fresh-UUID id can't hit the dedup guard on a live fire, so - // that arm is a contract violation if it ever fires. - match append_parked_fire(&state.pg_pool, token, &entry) - .await - .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("park: {e}")))? - { - ParkAppend::Parked => {} - ParkAppend::Refused(ParkRefusal::ResumeAlreadyAnswered) => { - return Err(( + crate::arrival::Arrival::Refused => Err((StatusCode::GONE, crate::arrival::REFUSED_FOR_CALLER.into())), + crate::arrival::Arrival::Wait => { + // The id tells this queued fire apart from any other on the + // same token, even when bodies are identical. The drain uses + // it as the task-table dedup nonce so a crash between + // task-insert and head-pop collapses the retry back to one + // task (same id, same dedup_key) while two genuinely distinct + // fires (different ids) produce two executions. + let entry = ParkedFire { + id: id.map_or_else(|| uuid::Uuid::new_v4().to_string(), str::to_string), + payload, + received_at_unix: crate::lease::now_unix(), + attempts: 0, + not_before_unix: 0, + instance_gap: None, + }; + // The shared append names its refusal; never swallow one under + // a 200. + match append_parked_fire(&state.pg_pool, token, &entry) + .await + .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("park: {e}")))? + { + ParkAppend::Parked => Ok(StatusCode::OK), + ParkAppend::Refused(ParkRefusal::ResumeAlreadyAnswered) => Err(( StatusCode::CONFLICT, "suspension already answered; duplicate submission ignored".into(), - )); - } - ParkAppend::Refused(ParkRefusal::QueueFull) => { - return Err(( + )), + ParkAppend::Refused(ParkRefusal::QueueFull) => Err(( StatusCode::TOO_MANY_REQUESTS, "this entry has too many pending fires queued; wait for the project \ to process them (it is currently parked) or retry later" .into(), - )); - } - ParkAppend::Refused(ParkRefusal::RowGone) => { - return Err(( + )), + ParkAppend::Refused(ParkRefusal::RowGone) => Err(( StatusCode::GONE, "this signal is no longer registered; the fire was not accepted".into(), - )); - } - ParkAppend::Refused(ParkRefusal::AlreadyQueued) => { - return Err(( + )), + // The same fire queued again (a retried task): it waits once. + ParkAppend::Refused(ParkRefusal::AlreadyQueued) if id.is_some() => Ok(StatusCode::OK), + ParkAppend::Refused(ParkRefusal::AlreadyQueued) => Err(( StatusCode::INTERNAL_SERVER_ERROR, format!("park: a fresh fire id {} is already queued; dispatcher contract broken", entry.id), - )); + )), } } - return Ok(StatusCode::OK); } - - // Wiped or hibernate-post-grace fully off: refuse. - Err(( - StatusCode::GONE, - "Project is not accepting requests. Please contact the project administrator.".into(), - )) } // `crate::lease::now_unix` is the canonical wall-clock reader. @@ -613,6 +562,17 @@ pub(crate) struct FireGateInfo { pub limits: weft_core::signal::ResolvedLimits, } +impl FireGateInfo { + /// How the governing activation stands, for the arrival rule. + pub(crate) fn standing(&self) -> crate::arrival::Standing { + crate::arrival::Standing { + status: self.status, + accepting_fires: self.accepting_fires, + fires_deadline_unix: self.fires_deadline_unix, + } + } +} + /// The gate columns of a signal, read through the activation that /// governs it (`signal.activation_trigger` + `signal.instance_id`, the one /// join `SIGNAL_ACTIVATION_JOIN`). No governing activation reads as @@ -1509,7 +1469,7 @@ pub async fn fire_public_entry( { return refused; } - apply_lifecycle_gate(&state, &token, &routing, payload).await.into_response() + apply_lifecycle_gate(&state, &token, &routing, payload, None).await.into_response() } /// Which open entry a bare-path fire reaches, with its gate info and @@ -1823,7 +1783,9 @@ pub async fn connect_live( // How long each step took, in the line that says the run was born: // what to read first when a call is slow. let began = std::time::Instant::now(); - let (route, token, params) = live_route(&state, &tenant_segment, only_project, &method_name, &path).await?; + // A route whose trigger is parked holds the caller here until it is + // back on, then serves them as it is armed then (`crate::arrival`). + let (route, token, params) = route_taking_calls(&state, &tenant_segment, only_project, &method_name, &path).await?; let ArmedRoute { project_id, node_id, protocol, live_config, auth_kind, auth_config, program, .. } = route.as_ref(); // Every header the caller sent, repeats included, less the one the @@ -1880,9 +1842,6 @@ pub async fn connect_live( ) }); - // Project must be Active to accept a live connection. - route.require_active()?; - // The entry's limits are checked as the run is born, in the same // call to the database: the caller is who the gate established when // the route has auth, else the instance the run is for, else the @@ -2014,11 +1973,77 @@ pub(crate) struct HeldRoute { type LiveRoute = (Arc, String, std::collections::BTreeMap); /// The live route of `tenant` (of `only_project`, when the call came by a -/// project's API domain) serving `method` on `path`: the route as it is -/// armed, its token and the path's captures. Matched against the routes -/// this dispatcher holds; a refusal (no such route, a half-armed or -/// inactive one) is made on the rows themselves, since a route activated a -/// moment ago may not have been heard yet. +/// project's API domain) serving `method` on `path`, once it takes calls: +/// the route as it is armed then, its token and the path's captures. A +/// route whose trigger is parked (or hibernating within its grace window, +/// or being set up) holds the caller until its trigger is back on, with +/// no deadline of weft's own: the caller's own connection bounds the wait. +/// One that takes no work is refused, in the caller's terms. +async fn route_taking_calls( + state: &DispatcherState, + tenant: &str, + only_project: Option, + method: &str, + path: &str, +) -> Result { + // Subscribed before the first read, so a route switched back on + // between the read and the wait still ends the wait. + let mut heard = state.signals.subscribe(); + let mut found = live_route(state, tenant, only_project, method, path).await?; + let mut held_since: Option = None; + loop { + let standing = found.0.standing(); + match standing.arrival(crate::lease::now_unix()) { + crate::arrival::Arrival::Live => { + if let Some(since) = held_since { + tracing::info!(target: "weft_dispatcher::signal", node = %found.0.node_id, held_ms = since.elapsed().as_millis() as u64, "a caller held while the route's trigger was off is served"); + } + return Ok(found); + } + crate::arrival::Arrival::Refused => { + tracing::info!( + target: "weft_dispatcher::signal", + node = %found.0.node_id, status = %found.0.status.as_str(), + "a caller refused: the route's trigger takes no work (`weft status` shows it; `weft activate` switches it back on)" + ); + return Err((StatusCode::SERVICE_UNAVAILABLE, crate::arrival::REFUSED_FOR_CALLER.into())); + } + crate::arrival::Arrival::Wait => {} + } + if held_since.is_none() { + held_since = Some(std::time::Instant::now()); + tracing::info!(target: "weft_dispatcher::signal", node = %found.0.node_id, status = %found.0.status.as_str(), "a caller held until the route's trigger is back on"); + } + // Until the tenant's routes change, or the grace window ends. + let until = standing.waits_until().map_or_else(far_future, |deadline| { + let left = (deadline + 1 - crate::lease::now_unix()).max(0) as u64; + tokio::time::Instant::now() + std::time::Duration::from_secs(left) + }); + heard + .woken_before(until, |channel, payload| channel == crate::held::ROUTES_CHANNEL && payload == tenant) + .await + .map_err(|e| { + tracing::error!( + target: "weft_dispatcher::signal", + node = %found.0.node_id, error = %format!("{e:#}"), + "a caller held until the route's trigger is back on lost the wait: this dispatcher stopped hearing the database" + ); + (StatusCode::SERVICE_UNAVAILABLE, crate::arrival::REFUSED_FOR_CALLER.to_string()) + })?; + found = match_live_route(&fresh_tenant_routes(state, tenant).await?, tenant, only_project, method, path)?; + } +} + +/// A deadline nothing reaches, for a wait that ends on a signal alone. +fn far_future() -> tokio::time::Instant { + tokio::time::Instant::now() + std::time::Duration::from_secs(86_400 * 365) +} + +/// The live route of `tenant` serving `method` on `path` as it is armed +/// now, whether or not it takes calls. Matched against the routes this +/// dispatcher holds; one that does not take calls there is read again from +/// the rows, since a route activated a moment ago may not have been heard +/// yet. async fn live_route( state: &DispatcherState, tenant: &str, @@ -2028,7 +2053,7 @@ async fn live_route( ) -> Result { if let Some(held) = state.held.routes.held(&tenant.to_string()) { if let Ok(found) = match_live_route(&held, tenant, only_project, method, path) { - if found.0.require_active().is_ok() { + if found.0.standing().arrival(crate::lease::now_unix()) == crate::arrival::Arrival::Live { return Ok(found); } } @@ -2106,39 +2131,23 @@ pub(crate) struct ArmedRoute { program: weft_core::project::hash::ProgramIdentity, source_version: String, /// The status of the activation governing the route at the read. - status: String, + status: crate::activation_store::ProjectStatus, + /// Whether that activation takes work while off, and until when + /// (`crate::arrival`). + accepting_fires: bool, + fires_deadline_unix: Option, /// Whose route it is: the instance whose trigger registered it, `None` /// for a shared one. Its gate's connection is that instance's. instance: Option, } impl ArmedRoute { - /// A live connection is accepted only while the route's activation is - /// Active. - fn require_active(&self) -> Result<(), (StatusCode, String)> { - use crate::project_store::ProjectStatus; - let refused = |why: String| Err((StatusCode::SERVICE_UNAVAILABLE, why)); - let node = &self.node_id; - match crate::project_store::project_status_from_str(&self.status) { - Ok(ProjectStatus::Active) => Ok(()), - Ok(ProjectStatus::Activating) => { - refused(format!("this route's trigger '{node}' is being switched on right now, so it takes no calls yet; call again in a moment")) - } - Ok(ProjectStatus::Deactivating) => refused(format!( - "this route's trigger '{node}' is being switched off right now, so it takes no calls; \ - a caller is never held while it is off" - )), - Ok(ProjectStatus::Registered | ProjectStatus::Inactive) => { - let activate = match &self.instance { - Some(instance) => format!("weft activate --instance {instance}"), - None => "weft activate".to_string(), - }; - refused(format!( - "this route is switched off (its trigger '{node}' is off), so it takes no calls; a caller \ - is never held while it is off. Whoever runs the program turns it back on with `{activate}`" - )) - } - Err(_) => refused(format!("this route's trigger '{node}' reads as '{}', which this install does not know", self.status)), + /// How the route's activation stands, for the arrival rule. + fn standing(&self) -> crate::arrival::Standing { + crate::arrival::Standing { + status: self.status, + accepting_fires: self.accepting_fires, + fires_deadline_unix: self.fires_deadline_unix, } } } @@ -2150,7 +2159,9 @@ impl ArmedRoute { /// inactive. const ARMED_COLUMNS: &str = "s.project_id, s.node_id, s.spec_json, s.auth_kind, s.auth_config, \ s.port_snapshot, s.program_json, s.source_version, s.instance_id, \ - CASE WHEN p.id IS NULL THEN 'inactive' ELSE COALESCE(a.status, 'active') END AS status"; + CASE WHEN p.id IS NULL THEN 'inactive' ELSE COALESCE(a.status, 'active') END AS status, \ + CASE WHEN p.id IS NULL THEN FALSE ELSE COALESCE(a.accepting_fires, TRUE) END AS accepting_fires, \ + a.fires_deadline_unix"; /// The armed route a signal row holds ([`ARMED_COLUMNS`]), or the HTTP /// answer when it is half-armed. @@ -2159,6 +2170,10 @@ fn armed_route_of(row: &sqlx::postgres::PgRow) -> Result = row.try_get("fires_deadline_unix").map_err(row_err)?; let auth_kind: String = row.try_get("auth_kind").map_err(row_err)?; let auth_config: Option = row.try_get("auth_config").map_err(row_err)?; let port_snapshot: Option = row.try_get("port_snapshot").map_err(row_err)?; @@ -2188,7 +2203,7 @@ fn armed_route_of(row: &sqlx::postgres::PgRow) -> Result::from) + .map_err(|gap| match gap { + // Infra the run reads is down: the fix is the operator's + // (logged), and the caller is told what is true for them. + crate::api::project::RunGap::Other((StatusCode::PRECONDITION_REQUIRED, why)) => { + tracing::info!(target: "weft_dispatcher::signal", %project_id, node = %route.node_id, "a caller refused: {why}"); + (StatusCode::SERVICE_UNAVAILABLE, crate::arrival::INFRA_DOWN_FOR_CALLER.to_string()) + } + gap => gap.into(), + }) }, crate::api::project::picks_for_run(state, project_id, &project_def, &subgraph), )?; diff --git a/crates/weft-dispatcher/src/app.rs b/crates/weft-dispatcher/src/app.rs index 9b21f583..573057a7 100644 --- a/crates/weft-dispatcher/src/app.rs +++ b/crates/weft-dispatcher/src/app.rs @@ -47,6 +47,9 @@ pub async fn apply_core_schema(pool: &sqlx::PgPool) -> anyhow::Result<()> { /// applies it whole, the migration generator writes its origins from it, /// and the agreement test builds from it. pub static ALL_GROUPS: &[&weft_task_store::SchemaGroup] = &[ + // What the writes a run makes announce, sent in batches: every + // trigger on a run's path writes to it. + &weft_task_store::announce::GROUP, &crate::journal::postgres::GROUP, // The exclusive infra ownership leases. &crate::infra_owner::GROUP, @@ -76,6 +79,9 @@ pub static ALL_GROUPS: &[&weft_task_store::SchemaGroup] = &[ // The public edge's counters: per-minute buckets and the runs each // entry has going (no FK: a slot outlives nothing it names). &crate::entry_limits::GROUP, + // Every finished run's words, for finding a run by what went through + // it (`run_search`). + &crate::run_search::GROUP, // The image builds running and done, by image ref (`build::ledger`). &crate::build::ledger::GROUP, // The wakes a local install has set and not yet delivered (a cloud @@ -230,7 +236,7 @@ pub async fn build_state(settings: DispatcherSettings<'_>, defaults: Defaults) - domains, holder_pool, holder_settings: config.holders, - worker_defaults: config.workers.clone(), + worker_defaults: config.worker_settings(), builder, install_info: install_info(config), projects, @@ -353,6 +359,8 @@ pub fn loop_wakes() -> Vec<(&'static str, &'static [WakeOn])> { ("infra_event_bridge", crate::infra_event_bridge::ON_INFRA_EVENT), ("parked_fires", crate::reaper::ON_PARKED_FIRE), ("storage_sweep", crate::reaper::ON_STORAGE_SWEEP), + ("hibernations", crate::reaper::ON_HIBERNATION), + ("infra_returns", crate::reaper::ON_INFRA_STATUS), ("holders", crate::holders::ON_HELD_SIGNALS), ("domains_door", crate::domains::ON_DOMAINS), ] diff --git a/crates/weft-dispatcher/src/arrival.rs b/crates/weft-dispatcher/src/arrival.rs new file mode 100644 index 00000000..22159871 --- /dev/null +++ b/crates/weft-dispatcher/src/arrival.rs @@ -0,0 +1,115 @@ +//! What happens to work arriving for a trigger, by how its activation +//! stands: the one rule every way work arrives goes through. +//! +//! Work arrives four ways: a call from outside at weft's door (a webhook, +//! a form, a provider's push), an answer to a run waiting on a person, an +//! event a listener picks up itself (a timer's tick, a message on a +//! connection it holds), and a caller on the line of a live route. While +//! the trigger is on, each becomes a run (or the answer reaches its run) +//! at once. While it is off, the way it went down decides, the same for +//! all four: +//! +//! - **park**: everything happens as usual except that nothing becomes a +//! run: the work waits, and runs once the trigger is back on, on the +//! version it comes back with. Listeners keep listening, so a trigger +//! that fires by itself keeps firing and each fire waits too. +//! - **hibernate**: the same until its grace window ends; after it, new +//! work is refused and the listeners stop. +//! - **wipe**: work is refused at once. +//! +//! An activation still being set up takes work the way a parked one does: +//! its listeners may not all be ready, and the setup's end runs what +//! waited. A live caller cannot be put in a queue, so a caller who arrives +//! while the trigger is parked is held on the line instead, and served the +//! moment the trigger is back (`crate::api::signal::connect_live`). +//! +//! What a person SEES while a trigger is off is a separate rule, kept on +//! the activation as `fires_visible_to_consumers`: a parked trigger's +//! questions stay listed for the people who answer them, a hibernated +//! one's are hidden. + +use crate::activation_store::ProjectStatus; + +/// What becomes of work arriving now. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Arrival { + /// The trigger is on: the work becomes its run now. + Live, + /// The trigger is parked (or hibernating within its grace window, or + /// being set up): the work waits for it to be back on. + Wait, + /// The trigger takes no work: wiped, never switched on, or past its + /// hibernation's grace window. + Refused, +} + +/// How an activation stands, as far as arriving work is concerned: its +/// status, whether it takes fires at all, and until when. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Standing { + pub status: ProjectStatus, + pub accepting_fires: bool, + pub fires_deadline_unix: Option, +} + +impl Standing { + /// What becomes of work arriving at `now_unix`. + // SYNC: arrival <-> weft_broker_client::protocol::ACTIVATION_LISTENS, IN_GRACE_WINDOW_SQL (crates/weft-dispatcher/src/activation_store.rs) + pub fn arrival(&self, now_unix: i64) -> Arrival { + match self.status { + // A signal no activation governs (a wait of a run started by + // hand) reads as active: it has no moment it comes back at, + // so work for it never waits. + ProjectStatus::Active => Arrival::Live, + ProjectStatus::Registered => Arrival::Refused, + ProjectStatus::Activating | ProjectStatus::Deactivating | ProjectStatus::Inactive => { + let past_grace = self.fires_deadline_unix.is_some_and(|deadline| now_unix > deadline); + if self.accepting_fires && !past_grace { + Arrival::Wait + } else { + Arrival::Refused + } + } + } + } + + /// When waiting work stops being taken (the end of a hibernation's + /// grace window), if it ever does. + pub fn waits_until(&self) -> Option { + self.fires_deadline_unix + } +} + +/// What a caller is told when the trigger takes no work, in the caller's +/// own terms: they cannot switch it back on, so the answer says what is +/// true for them and nothing about how the program is run. +pub const REFUSED_FOR_CALLER: &str = "This is not taking requests right now."; + +/// What a caller is told when what their request needs is not running (an +/// instance's infra stopped): the same reasoning as [`REFUSED_FOR_CALLER`]. +/// The operator's fix (which copy, the command that starts it) goes to the +/// log and `weft status`. +pub const INFRA_DOWN_FOR_CALLER: &str = "What this request needs is not running right now."; + +#[cfg(test)] +mod tests { + use super::*; + + fn standing(status: ProjectStatus, accepting_fires: bool, fires_deadline_unix: Option) -> Standing { + Standing { status, accepting_fires, fires_deadline_unix } + } + + #[test] + fn work_waits_while_parked_or_hibernating_and_is_refused_once_wiped_or_past_grace() { + assert_eq!(standing(ProjectStatus::Active, true, None).arrival(10), Arrival::Live); + assert_eq!(standing(ProjectStatus::Inactive, true, None).arrival(10), Arrival::Wait, "parked"); + assert_eq!(standing(ProjectStatus::Inactive, true, Some(20)).arrival(10), Arrival::Wait, "hibernating, in grace"); + assert_eq!(standing(ProjectStatus::Inactive, true, Some(20)).arrival(21), Arrival::Refused, "hibernating, past grace"); + assert_eq!(standing(ProjectStatus::Inactive, false, Some(20)).arrival(21), Arrival::Refused, "a hibernation whose end was marked"); + assert_eq!(standing(ProjectStatus::Inactive, false, None).arrival(10), Arrival::Refused, "wiped"); + assert_eq!(standing(ProjectStatus::Activating, true, None).arrival(10), Arrival::Wait, "being set up"); + assert_eq!(standing(ProjectStatus::Deactivating, true, None).arrival(10), Arrival::Wait, "going down to park"); + assert_eq!(standing(ProjectStatus::Deactivating, false, None).arrival(10), Arrival::Refused, "going down to wipe"); + assert_eq!(standing(ProjectStatus::Registered, true, None).arrival(10), Arrival::Refused, "never switched on"); + } +} diff --git a/crates/weft-dispatcher/src/holders.rs b/crates/weft-dispatcher/src/holders.rs index 159c4277..0fe9fde8 100644 --- a/crates/weft-dispatcher/src/holders.rs +++ b/crates/weft-dispatcher/src/holders.rs @@ -2,16 +2,17 @@ //! from the held signals and set on the platform, so the holders start as //! the first held signal is registered and stop with the last. -use weft_broker_client::protocol::{LISTENER_HELD_STATUSES, SIGNAL_ACTIVATION_JOIN}; +use weft_broker_client::protocol::{ACTIVATION_LISTENS, SIGNAL_ACTIVATION_JOIN}; use weft_platform_traits::HolderPool; use weft_task_store::drain::{DrainLoop, DrainStep, WakeOn}; use crate::state::DispatcherState; /// Announced when a held signal comes or goes (the `signal` table's -/// triggers), and when an activation is parked or woken (the -/// `trigger_activation` table's), which changes which signals are held -/// without a write to their rows. +/// triggers), and when an activation changes status, whether it takes +/// work, or until when (the `trigger_activation` table's): a trigger +/// switched on, wiped, or past its hibernation's grace window changes which +/// signals are held without a write to their rows. // SYNC: HELD_SIGNALS_CHANNEL <-> crates/weft-dispatcher/src/journal/postgres.rs (signal_held_notify) pub const HELD_SIGNALS_CHANNEL: &str = "weft_held_signals"; @@ -22,9 +23,8 @@ pub(crate) static ON_HELD_SIGNALS: &[WakeOn] = &[WakeOn::any(HELD_SIGNALS_CHANNE pub async fn held_count(pool: &sqlx::PgPool) -> anyhow::Result { let n: i64 = sqlx::query_scalar(&format!( "SELECT count(*) FROM signal s {SIGNAL_ACTIVATION_JOIN} \ - WHERE s.holds AND COALESCE(a.status, 'active') = ANY($1)" + WHERE s.holds AND {ACTIVATION_LISTENS}" )) - .bind(&LISTENER_HELD_STATUSES[..]) .fetch_one(pool) .await?; Ok(u64::try_from(n).unwrap_or(0)) @@ -40,7 +40,7 @@ pub async fn size(pool: &sqlx::PgPool, holders: &dyn HolderPool, per_copy: u32) } /// The holder sizing as a loop of the dispatcher: woken when a held signal -/// comes or goes or its activation is parked or woken. +/// comes or goes or its activation changes ([`HELD_SIGNALS_CHANNEL`]). pub fn drain_loop(state: &DispatcherState) -> DrainLoop { crate::reaper::woken(state, ON_HELD_SIGNALS, "holders", |state| async move { size(&state.pg_pool, state.holder_pool.as_ref(), state.holder_settings.signals_per_copy).await diff --git a/crates/weft-dispatcher/src/infra_node.rs b/crates/weft-dispatcher/src/infra_node.rs index b2398db0..b8755eff 100644 --- a/crates/weft-dispatcher/src/infra_node.rs +++ b/crates/weft-dispatcher/src/infra_node.rs @@ -225,17 +225,19 @@ pub struct CopyStatus { pub node_id: String, pub instance: Option, pub status: InfraNodeStatus, + /// When its last apply landed (a start, an upgrade), if one did. + pub applied_at_unix: Option, } /// Every copy of `project_id`'s infra nodes and its status. pub async fn statuses(pool: &PgPool, project_id: uuid::Uuid) -> Result> { - let rows: Vec<(String, Option, String)> = - sqlx::query_as("SELECT node_id, instance_id, status FROM infra_node WHERE project_id = $1") + let rows: Vec<(String, Option, String, Option)> = + sqlx::query_as("SELECT node_id, instance_id, status, applied_at_unix FROM infra_node WHERE project_id = $1") .bind(project_id) .fetch_all(pool) .await?; rows.into_iter() - .map(|(node_id, instance, status)| { + .map(|(node_id, instance, status, applied_at_unix)| { let instance = instance .map(weft_core::instance::InstanceId::new) .transpose() @@ -243,7 +245,7 @@ pub async fn statuses(pool: &PgPool, project_id: uuid::Uuid) -> Result bool { + self.inner.lock().unwrap().events.iter().any(|e| { + matches!(e, ExecEvent::NodeStarted { execution_id: of, node_id, .. } if *of == execution_id && node_id == node) + }) + } + pub fn new() -> Self { Self::default() } @@ -562,6 +569,10 @@ impl Journal for FakeJournal { tenant: &str, query: &ExecutionQuery, ) -> anyhow::Result { + // The words of a run are Postgres's full-text search, which a fake + // would only imitate: `search` is tested against Postgres + // (`tests/db_tags.rs`). + assert!(query.search.is_none(), "the listing's `search` is tested against Postgres, not the fake journal"); // Every summary for this tenant, newest first, then apply the same // project + start-time filters the Postgres query does, then page. // A run parked on a wait (a resume signal registered for it) reads @@ -583,6 +594,7 @@ impl Journal for FakeJournal { .filter(|s| query.status.is_none_or(|st| st.reaches(honest(s)))) .filter(|s| query.instance.as_ref().is_none_or(|m| s.instance.as_ref() == Some(m))) .filter(|s| query.tag.as_deref().is_none_or(|t| s.tags.iter().any(|x| x == t))) + .filter(|s| query.node.as_deref().is_none_or(|node| self.ran_node(s.execution_id, node))) .collect(); all.sort_by(|a, b| b.started_at.cmp(&a.started_at).then(b.execution_id.cmp(&a.execution_id))); let total = all.len() as u64; diff --git a/crates/weft-dispatcher/src/journal/mod.rs b/crates/weft-dispatcher/src/journal/mod.rs index c4d82dcd..98dc185b 100644 --- a/crates/weft-dispatcher/src/journal/mod.rs +++ b/crates/weft-dispatcher/src/journal/mod.rs @@ -768,6 +768,12 @@ pub struct ExecutionQuery { pub instance: Option, /// Only runs carrying this tag (`ctx.tag_execution`). pub tag: Option, + /// Only runs in which this node fired (it started at least once), + /// spelled the way the journal names it. + pub node: Option, + /// Only finished runs whose recorded values carry every word of this + /// (`crate::run_search`; a quoted phrase as written). + pub search: Option, /// Keyset cursor: only runs strictly after this `(started_at, /// execution)` in the listing's order (newest first, then execution /// descending). A walk that hands each page's last run back here diff --git a/crates/weft-dispatcher/src/journal/postgres.rs b/crates/weft-dispatcher/src/journal/postgres.rs index 19aa7c65..c20a5070 100644 --- a/crates/weft-dispatcher/src/journal/postgres.rs +++ b/crates/weft-dispatcher/src/journal/postgres.rs @@ -306,9 +306,11 @@ impl PostgresJournal { dedup_key: Option<&str>, ) -> anyhow::Result<()> { if !matches!(event, ExecEvent::ExecutionStarted { .. }) { - return weft_journal::record_event_in(&self.pool, event, None, dedup_key) + weft_journal::record_event_in(&self.pool, event, None, dedup_key) .await - .map_err(|e| anyhow::anyhow!("{e}")); + .map_err(|e| anyhow::anyhow!("{e}"))?; + weft_task_store::announce::committed(&self.pool); + return Ok(()); } let started = StartedRow::of(event, dedup_key, crate::lease::now_unix())?; sqlx::query("SELECT weft_execution_started($1)") @@ -316,6 +318,7 @@ impl PostgresJournal { .execute(&self.pool) .await .map_err(birth_refusal)?; + weft_task_store::announce::committed(&self.pool); Ok(()) } @@ -350,6 +353,7 @@ impl PostgresJournal { .fetch_one(&self.pool) .await .map_err(birth_refusal)?; + weft_task_store::announce::committed(&self.pool); match answer.get("outcome").and_then(serde_json::Value::as_str) { Some("started" | "already_started") => Ok(Ok(())), Some("refused") => { @@ -509,23 +513,25 @@ pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { r#"CREATE INDEX IF NOT EXISTS idx_exec_event_kind ON exec_event(kind, id DESC)"#, r#"CREATE UNIQUE INDEX IF NOT EXISTS idx_exec_event_dedup ON exec_event(dedup_key) WHERE dedup_key IS NOT NULL"#, - // Announce every row, with its execution, when it commits: the - // dispatcher's event bridge wakes on any, and a worker waiting - // on its run's journal (through the broker) on its own execution's. - // Many rows of one execution in one transaction collapse to one - // notification, since Postgres drops a duplicate payload within - // a transaction. + // Announce the executions a write added rows to, once each, when + // it commits: the dispatcher's event bridge wakes on any, and a + // worker waiting on its run's journal (through the broker) on its + // own execution's. Once per statement, over the rows it added, + // and through the announcement outbox (`weft_task_store::announce`), + // since every run writes here. // SYNC: 'weft_exec_event' <-> weft_journal::EXEC_EVENT_CHANNEL r#"CREATE OR REPLACE FUNCTION exec_event_notify() RETURNS trigger AS $$ BEGIN - PERFORM pg_notify('weft_exec_event', NEW.execution_id); + PERFORM weft_announce('weft_exec_event', a.execution_id) + FROM (SELECT DISTINCT execution_id FROM added) a; RETURN NULL; END; $$ LANGUAGE plpgsql"#, r#"DROP TRIGGER IF EXISTS exec_event_notify_on_insert ON exec_event"#, r#"CREATE TRIGGER exec_event_notify_on_insert AFTER INSERT ON exec_event - FOR EACH ROW + REFERENCING NEW TABLE AS added + FOR EACH STATEMENT EXECUTE FUNCTION exec_event_notify()"#, // signal_token: token-scoped enumeration credential. Allow // sets are TEXT[] arrays so parameterized binding gives no @@ -901,6 +907,7 @@ pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { // rows of one execution are numbered and committed in the same // order; taking it again in a transaction that already holds it is // free. + // SYNC: execution.owner_replica <-> weft_bind_execution_id_owner, bind_execution_id_owner (crates/weft-task-store/src/tasks.rs) r#"CREATE OR REPLACE FUNCTION weft_journal_append( p_execution_id TEXT, p_kinds TEXT[], p_payloads TEXT[], p_created_at BIGINT, p_replica TEXT, p_owner TEXT, p_dedup_key TEXT @@ -1188,6 +1195,7 @@ impl Journal for PostgresJournal { } } tx.commit().await?; + weft_task_store::announce::committed(&self.pool); Ok(write) } @@ -1402,9 +1410,9 @@ impl Journal for PostgresJournal { // SQL so a tenant with a huge history never truncates blindly. // // Bind order is fixed ($1 tenant, $2 project filter, $3 after, $4 before, - // $5 phase, $6 entry node, $7 status, $8 instance, $9 tag) and every optional filter is a - // `($n IS NULL OR ...)` clause so one prepared statement serves every - // filter combination. + // $5 phase, $6 entry node, $7 status, $8 instance, $9 tag, $10 node, + // $11 search) and every optional filter is a `($n IS NULL OR ...)` + // clause so one prepared statement serves every filter combination. // The `execution` row (seeded at start) carries the real, indexed // columns the filters key on: `tenant_id` (the wall), `project_id`, and // `started_at_unix`. `exec_event` only has `execution_id`/`kind`/`payload_json`, @@ -1456,6 +1464,14 @@ impl Journal for PostgresJournal { AND ($8::text IS NULL OR ec.instance_id = $8) \ AND ($9::text IS NULL OR EXISTS ( \ SELECT 1 FROM execution_tag et WHERE et.execution_id = ec.execution_id AND et.tag = $9 \ + )) \ + AND ($10::text IS NULL OR EXISTS ( \ + SELECT 1 FROM exec_event WHERE execution_id = ec.execution_id \ + AND kind = 'node_started' AND payload_json::jsonb->>'node_id' = $10 \ + )) \ + AND ($11::text IS NULL OR EXISTS ( \ + SELECT 1 FROM execution_search es WHERE es.execution_id = ec.execution_id \ + AND es.words @@ websearch_to_tsquery('simple', $11) \ ))"); // The count carries the SAME started-event predicate as the row @@ -1480,6 +1496,8 @@ impl Journal for PostgresJournal { .bind(status) .bind(query.instance.as_ref().map(|m| m.as_str())) .bind(query.tag.as_deref()) + .bind(query.node.as_deref()) + .bind(query.search.as_deref()) .fetch_one(&self.pool) .await?; @@ -1500,8 +1518,8 @@ impl Journal for PostgresJournal { ORDER BY id DESC LIMIT 1 \ ) t ON TRUE \ WHERE {where_clause} \ - AND ($12::bigint IS NULL OR (ec.started_at_unix, ec.execution_id) < ($12, $13::text)) \ - ORDER BY ec.started_at_unix DESC, ec.execution_id DESC LIMIT $10 OFFSET $11" + AND ($14::bigint IS NULL OR (ec.started_at_unix, ec.execution_id) < ($14, $15::text)) \ + ORDER BY ec.started_at_unix DESC, ec.execution_id DESC LIMIT $12 OFFSET $13" )) .bind(tenant) .bind(project) @@ -1512,6 +1530,8 @@ impl Journal for PostgresJournal { .bind(status) .bind(query.instance.as_ref().map(|m| m.as_str())) .bind(query.tag.as_deref()) + .bind(query.node.as_deref()) + .bind(query.search.as_deref()) .bind(query.limit as i64) .bind(query.offset as i64) .bind(query.below.map(|(started, _)| started as i64)) @@ -2037,6 +2057,8 @@ async fn erase_execution_ids( // execution forever as non-terminal (its NOT EXISTS terminal check // passes vacuously once every event is gone), so wipe and // cancel_running re-sweep a ghost. + sqlx::query("DELETE FROM execution_search WHERE execution_id = ANY($1)") + .bind(&ids).execute(&mut *tx).await?; sqlx::query("DELETE FROM execution WHERE execution_id = ANY($1)") .bind(&ids).execute(&mut *tx).await?; // The run's row in the version tree belongs to the version store, diff --git a/crates/weft-dispatcher/src/journal_bridge.rs b/crates/weft-dispatcher/src/journal_bridge.rs index 344f07c0..1d1a8129 100644 --- a/crates/weft-dispatcher/src/journal_bridge.rs +++ b/crates/weft-dispatcher/src/journal_bridge.rs @@ -260,6 +260,8 @@ async fn process_one_row( } } terminal_cleanup(state, execution_id).await?; + // Its words, for finding it by what went through it. + crate::run_search::index_finished_run(&state.pg_pool, execution_id).await?; // Storage terminate sweep: queue the un-kept exec-file // sweep DURABLY (workers stall-then-die, so worker-side // cleanup is only an eager optimization; this row is the diff --git a/crates/weft-dispatcher/src/lib.rs b/crates/weft-dispatcher/src/lib.rs index e2058296..80fa235e 100644 --- a/crates/weft-dispatcher/src/lib.rs +++ b/crates/weft-dispatcher/src/lib.rs @@ -17,6 +17,7 @@ pub mod activation_store; pub mod api; pub mod app; +pub mod arrival; pub mod authenticator; pub mod build; pub mod delivery; @@ -46,6 +47,7 @@ pub mod install_picks; pub mod projection; pub mod project_store; pub mod reaper; +pub mod run_search; pub mod reclaim; pub mod settled; pub mod state; diff --git a/crates/weft-dispatcher/src/lifecycle_claimer.rs b/crates/weft-dispatcher/src/lifecycle_claimer.rs index 39b529a5..bf69cc62 100644 --- a/crates/weft-dispatcher/src/lifecycle_claimer.rs +++ b/crates/weft-dispatcher/src/lifecycle_claimer.rs @@ -359,7 +359,8 @@ async fn run_deactivate( project_id, &crate::take_down::TakeDownTarget::Activations(targets), &take_down.spec, - true, // health-loop autonomous park: its auto-recover MAY reactivate this + // The health loop's own park: its auto-recover may bring it back. + Some(crate::take_down::DownWith::Health), None, ) .await @@ -464,7 +465,7 @@ fn health_restore_targets( ) -> Vec { activations .iter() - .filter(|a| a.lifecycle.deactivated_by_health && a.lifecycle.status == crate::activation_store::ProjectStatus::Inactive) + .filter(|a| a.lifecycle.parked_by_health()) .filter(|a| { !deps .iter() @@ -502,7 +503,7 @@ mod tests { /// An activation the health loop took down. fn by_health(trigger: &str, owner: Owner) -> crate::activation_store::Activation { let mut a = activation(trigger, owner, crate::activation_store::ProjectStatus::Inactive); - a.lifecycle.deactivated_by_health = true; + a.lifecycle.went_down = Some(weft_broker_client::activation::WentDown { with: crate::take_down::DownWith::Health, at_unix: 0 }); a } diff --git a/crates/weft-dispatcher/src/listener.rs b/crates/weft-dispatcher/src/listener.rs index 17195cf8..f1cec307 100644 --- a/crates/weft-dispatcher/src/listener.rs +++ b/crates/weft-dispatcher/src/listener.rs @@ -229,3 +229,91 @@ pub async fn project_is_listening(pool: &sqlx::PgPool, project: uuid::Uuid) -> R .fetch_one(pool) .await?) } + +/// How long one call to the listener while letting go may take: a +/// teardown can call out to a provider, so this is generous; one that +/// runs past it is tried again on the next pass. +const LET_GO_WAIT: std::time::Duration = std::time::Duration::from_secs(60); + +/// How many signals are let go of at once. +const LET_GO_AT_ONCE: usize = 16; + +/// Have the listener let go of `signals`, whose activations took no work +/// when the caller looked (a hibernation past its grace window), each +/// answered once its teardown is over ([`ListenerClient::unregister_replaced`], +/// which also passes a row no listener can ever read). Then any of them +/// listening by now (a reactivation landed meanwhile, and its own bring-up +/// may have been undone by the teardown) is torn down and brought up again +/// whole (what fails there is logged with its recovery, never stopping the +/// others). Answers the tokens the listener did not let go of, for the +/// caller to try again; fails only when the look after could not be made. +pub async fn let_go_of_stopped( + pool: &sqlx::PgPool, + listener: &ListenerClient, + signals: &[crate::journal::SignalRegistration], +) -> Result> { + use futures::StreamExt; + if signals.is_empty() { + return Ok(Vec::new()); + } + let kept: Vec = futures::stream::iter(signals.iter().cloned()) + .map(|sig| { + let listener = listener.clone(); + async move { + match tokio::time::timeout(LET_GO_WAIT, listener.unregister_replaced(&sig)).await { + Ok(Ok(_)) => None, + Ok(Err(e)) => { + tracing::warn!(target: "weft_dispatcher::listener", token = %sig.token, error = %format!("{e:#}"), "the listener did not let go of a signal; tried again later"); + Some(sig.token.clone()) + } + Err(_) => { + tracing::warn!(target: "weft_dispatcher::listener", token = %sig.token, "the listener did not answer in time when told to let go of a signal; tried again later"); + Some(sig.token.clone()) + } + } + } + }) + .buffer_unordered(LET_GO_AT_ONCE) + .filter_map(|kept| async move { kept }) + .collect() + .await; + let tokens: Vec<&str> = signals.iter().map(|s| s.token.as_str()).collect(); + let listening: Vec<(String, uuid::Uuid)> = sqlx::query_as(&format!( + "SELECT s.token, s.project_id FROM signal s {} WHERE s.token = ANY($1) AND {}", + weft_broker_client::protocol::SIGNAL_ACTIVATION_JOIN, + weft_broker_client::protocol::ACTIVATION_LISTENS, + )) + .bind(&tokens) + .fetch_all(pool) + .await?; + // Every one of them is brought up, whatever happens to another: one + // left torn down would read as listening with nothing behind it, and + // nothing would look at it again. + let mut projects = std::collections::BTreeSet::new(); + let mut failed = Vec::new(); + for (token, project) in &listening { + if let Some(sig) = signals.iter().find(|s| &s.token == token) { + match tokio::time::timeout(LET_GO_WAIT, listener.unregister_replaced(sig)).await { + Ok(Ok(_)) => {} + Ok(Err(e)) => failed.push(format!("{token}: {e:#}")), + Err(_) => failed.push(format!("{token}: no answer within {}s", LET_GO_WAIT.as_secs())), + } + } + projects.insert(*project); + } + for project in projects { + match tokio::time::timeout(LET_GO_WAIT, listener.rehydrate(project, &[])).await { + Ok(Ok(())) => {} + Ok(Err(e)) => failed.push(format!("project {project}: {e:#}")), + Err(_) => failed.push(format!("project {project}: no answer within {}s", LET_GO_WAIT.as_secs())), + } + } + if !failed.is_empty() { + tracing::error!( + target: "weft_dispatcher::listener", + failed = %failed.join("; "), + "signals switched back on while being let go of could not all be brought up again; `weft activate` on their project brings them up" + ); + } + Ok(kept) +} diff --git a/crates/weft-dispatcher/src/reaper.rs b/crates/weft-dispatcher/src/reaper.rs index 3473b861..0e878a3e 100644 --- a/crates/weft-dispatcher/src/reaper.rs +++ b/crates/weft-dispatcher/src/reaper.rs @@ -42,6 +42,12 @@ fn parked_fire_longest_sleep() -> Duration { const NOTHING: &[WakeOn] = &[]; pub(crate) static ON_PARKED_FIRE: &[WakeOn] = &[WakeOn::any(PARKED_FIRE_CHANNEL)]; pub(crate) static ON_STORAGE_SWEEP: &[WakeOn] = &[WakeOn::any(STORAGE_SWEEP_CHANNEL)]; +/// A trigger's activation changing status, whether it takes work, or until +/// when: a hibernation starting has a grace window to end. +pub(crate) static ON_HIBERNATION: &[WakeOn] = &[WakeOn::any(crate::holders::HELD_SIGNALS_CHANNEL)]; +/// An infra copy coming, going or changing status: one running again +/// brings back the triggers that went down with it. +pub(crate) static ON_INFRA_STATUS: &[WakeOn] = &[WakeOn::any(crate::held::INFRA_STATUS_CHANNEL)]; /// Safety tick of the reapers that are woken by a write: 60 seconds in /// real time, at this install's pace (`weft_core::time_scale`). @@ -50,7 +56,8 @@ fn woken_reaper_safety() -> Duration { } /// Whether anything in the install is in motion: a claim held, work -/// waiting for a worker, an activation or a build under way, a slot at an +/// waiting for a worker (a cancel waiting for the worker driving its run +/// is not: that drive's own claim counts), an activation or a build under way, a slot at an /// entry for a run that has not started yet, a lifecycle command not /// finished. While nothing is, no lease can lapse and no driver can die /// mid-way, so the loops that watch for that have nothing to watch. @@ -61,7 +68,8 @@ fn woken_reaper_safety() -> Duration { /// a person does not keep the install awake. pub async fn in_motion(pool: &sqlx::PgPool) -> anyhow::Result { Ok(sqlx::query_scalar( - "SELECT EXISTS (SELECT 1 FROM task WHERE status = 'claimed' OR (status = 'pending' AND target = 'worker')) \ + "SELECT EXISTS (SELECT 1 FROM task WHERE status = 'claimed' \ + OR (status = 'pending' AND target = 'worker' AND kind <> 'cancel_execution')) \ OR EXISTS (SELECT 1 FROM trigger_activation WHERE status IN ('activating', 'deactivating')) \ OR EXISTS (SELECT 1 FROM project WHERE transition <> 'none') \ OR EXISTS (SELECT 1 FROM entry_slot s \ @@ -135,9 +143,65 @@ pub fn drain_loops(state: &DispatcherState) -> Vec { // it and asks for another look soon. The kept-file expiry sweep is // the broker's own loop (it owns the bucket + metadata). woken(state, ON_STORAGE_SWEEP, "storage_sweep", crate::storage::process_sweep_queue), + // A hibernation's grace window ending: its triggers stop listening + // and taking work. Sleeps until the next one ends. + woken(state, ON_HIBERNATION, "hibernations", end_hibernations), + // An infra copy running again: the triggers its stop took down + // with it come back. + woken(state, ON_INFRA_STATUS, "infra_returns", crate::api::infra::bring_back_triggers_whose_infra_returned), ] } +/// End every hibernation whose grace window has passed: the listener lets +/// go of its signals (`crate::listener::let_go_of_stopped`), and then its +/// activation stops taking work (`crate::activation_store::end_grace_windows`). +/// In that order, so one whose letting go failed (the listener down, a +/// signal it cannot tear down yet) stays for the next pass, which lets go +/// again, and holds back no other. Then sleep until the next window ends, +/// or until an activation changes. +async fn end_hibernations(state: DispatcherState) -> anyhow::Result { + let mut by_project: std::collections::BTreeMap> = Default::default(); + for (project_id, key) in crate::activation_store::grace_windows_over(&state.pg_pool).await? { + by_project.entry(project_id).or_default().push(key); + } + let mut kept_any = false; + for (project_id, keys) in by_project { + let signals = crate::journal::postgres::activation_signals(&state.pg_pool, project_id, &keys).await?; + let kept = crate::listener::let_go_of_stopped(&state.pg_pool, &state.listener, &signals).await?; + let let_go: Vec = keys + .into_iter() + .filter(|key| { + !signals.iter().any(|s| { + kept.contains(&s.token) + && s.activation_trigger.as_deref() == Some(key.trigger.as_str()) + && s.instance.as_ref() == key.instance() + }) + }) + .collect(); + kept_any |= !kept.is_empty(); + let ended = crate::activation_store::end_grace_windows(&state.pg_pool, project_id, &let_go).await?; + if ended > 0 { + tracing::info!( + target: "weft_dispatcher::reaper", + %project_id, triggers = ended, + "a hibernation's grace window ended: its triggers stopped listening and take no more work" + ); + } + } + if kept_any { + return Ok(DrainStep::RetryIn(HIBERNATION_LET_GO_RETRY)); + } + let now = crate::lease::now_unix(); + Ok(match crate::activation_store::next_grace_end(&state.pg_pool).await? { + None => DrainStep::Done, + Some(deadline) => DrainStep::RetryIn(Duration::from_secs((deadline + 1 - now).max(1) as u64)), + }) +} + +/// How soon a hibernation whose signals the listener did not let go of is +/// tried again. +const HIBERNATION_LET_GO_RETRY: Duration = Duration::from_secs(30); + /// A sweep that runs on its interval alone. `interval` is given in real /// time and runs at this install's pace (`weft_core::time_scale`), like /// the leases these sweeps judge. diff --git a/crates/weft-dispatcher/src/run_search.rs b/crates/weft-dispatcher/src/run_search.rs new file mode 100644 index 00000000..afd0b690 --- /dev/null +++ b/crates/weft-dispatcher/src/run_search.rs @@ -0,0 +1,102 @@ +//! Finding a run by what went through it. +//! +//! When a person traces an incident ("this customer says their order +//! failed at noon"), what they hold is a word that went through the run: +//! an email, an order id, a phrase from an error. Each finished run gets +//! one search document, the words of every value its journal recorded +//! (the trigger's input, what every node sent on, every error and log +//! line), built once when the run ends, off the path any run takes. The +//! runs listing matches it (`GET /executions?search=...`), beside its +//! other filters. +//! +//! Words, not substrings: `ada@example.com` finds the runs that carried that +//! address, and several words find the runs carrying all of them (a +//! quoted phrase, the runs carrying it as written). A run still going has +//! no document yet. + +/// How much of a run's text its document keeps, in characters, and how +/// much of any one value. The values are read in the order the run +/// recorded them and reading stops at the cap, so a run with a huge journal +/// costs no more than a small one. A document holds each word once with +/// its positions and Postgres refuses one past a megabyte: at four bytes a +/// character, every word distinct and each hyphenated one also kept in +/// parts, this many characters stays well under it. +const MAX_TEXT: i32 = 50_000; +const MAX_VALUE: i32 = 2_000; + +pub static GROUP: weft_task_store::SchemaGroup = weft_task_store::SchemaGroup { + name: "execution_search", + tables: &["execution_search"], + ddl: &[ + // One row per finished recorded run: its words, in the `simple` + // configuration (every word as written, lowercased, no language's + // stemming or stop words: an id or a name is found as it is). + r#"CREATE TABLE IF NOT EXISTS execution_search ( + execution_id TEXT PRIMARY KEY, + words TSVECTOR NOT NULL + )"#, + r#"CREATE INDEX IF NOT EXISTS idx_execution_search_words ON execution_search USING GIN (words)"#, + // The text of a run's document: every string and number its + // journal recorded (never a field's name), row by row in the order + // recorded, each cut to `p_value_max` characters, up to `p_max` in + // all. A + // value holding a NUL character cannot be read as JSONB, so a row + // with one is read without it; one that still does not read (a + // NUL spelled out with an escaped backslash before it) adds + // nothing, and the rest of the run still does. + r#"CREATE OR REPLACE FUNCTION weft_run_search_text(p_execution_id TEXT, p_max INTEGER, p_value_max INTEGER) RETURNS TEXT AS $$ + DECLARE + -- Collected, then joined once: growing one string would + -- copy all of it at every value. + v_parts TEXT[] := '{}'; + v_length INTEGER := 0; + v_payload TEXT; + v_json JSONB; + v_value TEXT; + BEGIN + FOR v_payload IN SELECT payload_json FROM exec_event WHERE execution_id = p_execution_id ORDER BY id LOOP + IF strpos(v_payload, '\u0000') > 0 THEN + BEGIN + v_json := replace(v_payload, '\u0000', '')::jsonb; + EXCEPTION WHEN untranslatable_character OR invalid_text_representation THEN + v_json := NULL; + END; + CONTINUE WHEN v_json IS NULL; + ELSE + v_json := v_payload::jsonb; + END IF; + -- One row's values at once; the cap is looked at row by + -- row, and the last row is cut to it. + SELECT string_agg(left(j #>> '{}', p_value_max), ' ') INTO v_value + FROM jsonb_path_query(v_json, 'strict $.** ? (@.type() == "string" || @.type() == "number")') j; + CONTINUE WHEN v_value IS NULL; + v_parts := array_append(v_parts, v_value); + v_length := v_length + length(v_value) + CASE WHEN v_length = 0 THEN 0 ELSE 1 END; + IF v_length >= p_max THEN + RETURN left(array_to_string(v_parts, ' '), p_max); + END IF; + END LOOP; + RETURN array_to_string(v_parts, ' '); + END; + $$ LANGUAGE plpgsql"#, + ], + seed: &[], +}; + +/// Build the search document of the finished run `execution_id` +/// (`weft_run_search_text`). Idempotent: a second build of the same run +/// (another dispatcher's bridge, a retried row) leaves the first. +pub async fn index_finished_run(pool: &sqlx::PgPool, execution_id: weft_core::ExecutionId) -> anyhow::Result<()> { + sqlx::query( + "INSERT INTO execution_search (execution_id, words) \ + SELECT ec.execution_id, to_tsvector('simple', weft_run_search_text(ec.execution_id, $2, $3)) \ + FROM execution ec WHERE ec.execution_id = $1 \ + ON CONFLICT (execution_id) DO NOTHING", + ) + .bind(execution_id.to_string()) + .bind(MAX_TEXT) + .bind(MAX_VALUE) + .execute(pool) + .await?; + Ok(()) +} diff --git a/crates/weft-dispatcher/src/take_down.rs b/crates/weft-dispatcher/src/take_down.rs index 00930ecd..1cd90c91 100644 --- a/crates/weft-dispatcher/src/take_down.rs +++ b/crates/weft-dispatcher/src/take_down.rs @@ -24,6 +24,7 @@ use weft_core::running_policy::{DeactivateSpec, DeactivationMode, RunningPolicy} use weft_core::ExecutionId; use crate::activation_store::{ActivationLifecycle, LifecycleWrite, ProjectStatus, SignalsGoing}; +pub use weft_broker_client::activation::{DownWith, WentDown}; use crate::events::DispatcherEvent; use crate::state::DispatcherState; @@ -88,13 +89,13 @@ pub fn runs_using_copies<'a>( } /// The lifecycle a spec lands its activations at, before any drain. -pub fn landing_lifecycle(spec: &DeactivateSpec, now_unix: i64, by_health: bool) -> ActivationLifecycle { +pub fn landing_lifecycle(spec: &DeactivateSpec, now_unix: i64, went_down: Option) -> ActivationLifecycle { let target = match spec.mode { DeactivationMode::Wipe => ActivationLifecycle::wiped(), DeactivationMode::Hibernate => ActivationLifecycle::hibernating(now_unix + (spec.grace_minutes as i64) * 60), DeactivationMode::Park => ActivationLifecycle::parked(), }; - let target = ActivationLifecycle { deactivated_by_health: by_health, ..target }; + let target = ActivationLifecycle { went_down, ..target }; if spec.drains() { let cap = spec.drain_timeout_secs.unwrap_or(weft_broker_client::protocol::DEFAULT_DRAIN_TIMEOUT_SECS); ActivationLifecycle::deactivating_to(target, now_unix + cap as i64) @@ -139,8 +140,8 @@ pub(crate) async fn live_runs(state: &DispatcherState, project_id: uuid::Uuid) - /// left half done: refuse a target that is mid-activation (cancel that /// first) or a project mid-build; then wipe drops what the target's /// activations had registered and cancels every run it reaches, while -/// hibernate and park keep the rows and take the listeners' copies away -/// (reactivating restores them from the rows); `cancel` stops what is +/// hibernate and park keep the rows and the listeners keep holding them +/// (what they hear waits for the trigger: `crate::arrival`); `cancel` stops what is /// running now (a parked run survives hibernate and park), `wait` leaves /// it to finish and lands the activations once it has (the reaper cancels /// what is left at the cap). The lifecycle write comes before any of @@ -150,7 +151,7 @@ pub async fn take_down( project_id: uuid::Uuid, target: &TakeDownTarget, spec: &DeactivateSpec, - by_health: bool, + went_down_with: Option, asked_by: Option, ) -> Result { spec.validate().map_err(|m| (StatusCode::BAD_REQUEST, m.to_string()))?; @@ -182,10 +183,22 @@ pub async fn take_down( // instance's wiped row goes) and their signals leave together: a // failure after it leaves nothing registered for a row that is gone, // and the cancels below are safe to repeat. A park or a hibernate - // keeps the rows (the parking gate needs them) and reads them in that - // transaction: a reactivation landing right after registers fresh - // signals this cleanup never sees. - let landing = landing_lifecycle(spec, crate::lease::now_unix(), by_health); + // keeps the rows, and the listeners keep holding them: what they hear + // waits for the trigger to be back on (`crate::arrival`). + // When it went down is the database's moment, the clock an infra + // copy's apply is stamped with, which is what it is compared with + // (`crate::api::infra::bring_back_triggers_whose_infra_returned`). + let went_down = match went_down_with { + None => None, + Some(with) => { + let at_unix: i64 = sqlx::query_scalar("SELECT EXTRACT(EPOCH FROM NOW())::BIGINT") + .fetch_one(&state.pg_pool) + .await + .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("read the database's clock: {e}")))?; + Some(WentDown { with, at_unix }) + } + }; + let landing = landing_lifecycle(spec, crate::lease::now_unix(), went_down); let signals = match (spec.mode, target) { (DeactivationMode::Wipe, TakeDownTarget::WholeProject) => SignalsGoing::Project { except: asked_by }, (DeactivationMode::Wipe, TakeDownTarget::Activations(_)) => SignalsGoing::Activations, @@ -203,8 +216,8 @@ pub async fn take_down( return Err((StatusCode::CONFLICT, format!("cannot deactivate: {blocker}; cancel it first"))); } }; - // Take the listeners' copies away; a reactivation restores the - // kept ones. A retry reads the kept rows again and repeats this. + // Take the listeners' copies of the wiped signals away. A retry finds + // the rows already gone and repeats nothing. if !unlisten.is_empty() { state.listener.unregister_many(&unlisten).await; } @@ -357,14 +370,18 @@ mod tests { running_policy: RunningPolicy::Wait, drain_timeout_secs: Some(30), }; - let landing = landing_lifecycle(&spec, 100, false); + let landing = landing_lifecycle(&spec, 100, None); assert_eq!(landing.status, ProjectStatus::Deactivating); assert_eq!(landing.drain_deadline_unix, Some(130)); assert_eq!(landing.mode().as_str(), "deactivating"); let spec = DeactivateSpec { running_policy: RunningPolicy::Cancel, ..spec }; - assert_eq!(landing_lifecycle(&spec, 100, true).mode().as_str(), "park"); - assert!(landing_lifecycle(&spec, 100, true).deactivated_by_health); + let health = WentDown { with: DownWith::Health, at_unix: 90 }; + assert_eq!(landing_lifecycle(&spec, 100, Some(health)).mode().as_str(), "park"); + assert_eq!( + landing_lifecycle(&spec, 100, Some(WentDown { with: DownWith::Infra, at_unix: 90 })).went_down, + Some(WentDown { with: DownWith::Infra, at_unix: 90 }) + ); let hibernate = DeactivateSpec { mode: DeactivationMode::Hibernate, ..spec }; - assert_eq!(landing_lifecycle(&hibernate, 100, false).fires_deadline_unix, Some(100 + 15 * 60)); + assert_eq!(landing_lifecycle(&hibernate, 100, None).fires_deadline_unix, Some(100 + 15 * 60)); } } diff --git a/crates/weft-dispatcher/src/task_kinds/execute.rs b/crates/weft-dispatcher/src/task_kinds/execute.rs index ea668b74..479be35b 100644 --- a/crates/weft-dispatcher/src/task_kinds/execute.rs +++ b/crates/weft-dispatcher/src/task_kinds/execute.rs @@ -1,8 +1,8 @@ //! Producer helpers for `execute`, `resume`, and `cancel_execution` //! tasks. The first two are the work a worker is called for (the //! dispatcher's `delivery` hands each to the project's workers); a cancel -//! is taken by the worker driving its execution, through its cancel wait. The -//! handlers live in `weft_engine::worker`. +//! is heard by the worker driving its execution, announced on its line, +//! which asks for it. The handlers live in `weft_engine::worker`. use anyhow::Result; @@ -119,7 +119,7 @@ pub fn execution_task_spec(task: ExecutionTask<'_>) -> Result { /// Enqueue a `cancel_execution` task for `execution_id` when a worker is driving /// it right now (its execute or resume task holds a claim that is being /// renewed). The cancel flag lives in that worker's memory; the worker -/// takes the task through its cancel wait and fires the flag. +/// hears the task announced on its line, asks for it, and fires the flag. /// /// `cause` rides in the payload so the worker flips the execution's flag WITH /// it: when the worker's terminal write beats the dispatcher's, the diff --git a/crates/weft-dispatcher/src/task_kinds/fire_signal.rs b/crates/weft-dispatcher/src/task_kinds/fire_signal.rs index 92e4bda6..bc4b55ce 100644 --- a/crates/weft-dispatcher/src/task_kinds/fire_signal.rs +++ b/crates/weft-dispatcher/src/task_kinds/fire_signal.rs @@ -1,8 +1,8 @@ //! `fire_signal` task: a held-event signal fired inside a pooled //! listener process (Timer expiry, SSE event delivery, future //! browser-session resolution). The listener enqueues a row through the -//! broker; a dispatcher claims it and runs the same -//! `dispatch_listener_outcome` path a stateless fire goes through. +//! broker; a dispatcher claims it and runs it through the same gate and +//! `dispatch_listener_outcome` path a fire at the door goes through. //! //! The listener never speaks HTTP to the dispatcher (only the other way //! round): listener → dispatcher coordination goes through the task @@ -26,52 +26,80 @@ pub struct FireSignalExecutor; impl TaskExecutor for FireSignalExecutor { async fn execute(&self, state: &DispatcherState, task: &Task) -> Result { let payload: FireSignalPayload = serde_json::from_value(task.payload.clone())?; - // Held-event fires bypass the public lifecycle gate. Even - // though the listener process holding this signal may have been - // reaped between when the event was held and when this task - // runs, `ensure_placed_handle` (inside `dispatch_listener_ - // outcome`) re-places the signal from its durable row onto a - // live process to honor the fire; the reaper retires the empty process - // again on the next sweep. - let signal = state - .journal - .signal_get(&payload.token) - .await? - .ok_or_else(|| anyhow::anyhow!("signal {} not found", payload.token))?; + // A fire a listener picked up itself passes the same rule as one + // that came in at the door (`crate::arrival`): it becomes a run + // while its trigger is on, waits while it is parked, and is + // dropped once it takes no work. Even though the listener process + // holding this signal may have been reaped between when the event + // was held and when this task runs, `ensure_placed_handle` (inside + // `dispatch_listener_outcome`) re-places the signal from its + // durable row onto a live process to honor the fire. + use axum::http::StatusCode; + let routing = match crate::api::signal::lookup_signal_routing(state, &payload.token).await { + Ok(routing) => routing, + // Its row went after the listener heard it (a wipe, which told + // the listener as it removed it): nothing is left to fire. + Err((StatusCode::NOT_FOUND, _)) => { + tracing::info!(target: "weft_dispatcher::fire_signal", token = %payload.token, "fire dropped: its signal is gone"); + return Ok(serde_json::json!({ "status": StatusCode::GONE.as_u16() })); + } + Err((code, msg)) => anyhow::bail!("signal {} {code}: {msg}", payload.token), + }; // An entry's fire counts against its per-minute limit here, the // one place every listener-held fire passes (a resume answers a // run already going and is no entry). Past it the fire is // dropped: nobody called, so there is nobody to answer 429 to. // The task id is the fire's identity, so a retried task reuses // its first decision instead of counting the fire again. - if !signal.is_resume { - let limits = signal.spec()?.limits.resolve(); + if !routing.is_resume { let now = crate::lease::now_unix(); - if let Err(refused) = crate::entry_limits::admit_fire(&state.pg_pool, &payload.token, Some(&task.id.to_string()), &limits, now).await? { + if let Err(refused) = crate::entry_limits::admit_fire(&state.pg_pool, &payload.token, Some(&task.id.to_string()), &routing.limits, now).await? { tracing::info!( target: "weft_dispatcher::fire_signal", - token = %payload.token, project_id = %signal.project_id, + token = %payload.token, project_id = %routing.project_id, "fire dropped: {} is reached", refused.reason.describe() ); return Ok(serde_json::json!({ "status": axum::http::StatusCode::TOO_MANY_REQUESTS.as_u16() })); } } - // Use the FireSignal task id as the dedup nonce: an executor - // retry of this same task re-calls dispatch with the same - // nonce, so any RouteEntry task inserted on the first - // attempt collapses on retry instead of producing a - // duplicate execution. + // The task id is the fire's identity: a retry of this same task + // collapses onto what the first attempt did, whether it started + // a run (the route task's dedup nonce) or queued the fire. let nonce = task.id.to_string(); - let status = crate::api::signal::dispatch_listener_outcome( - state, - &payload.token, - signal.project_id, - &signal.tenant_id, - payload.payload, - Some(crate::api::signal::ParkedRef { id: &nonce, attempts: 0 }), - ) - .await - .map_err(|(code, msg)| anyhow::anyhow!("dispatch_listener_outcome {code}: {msg}"))?; + let status = match crate::api::signal::apply_lifecycle_gate(state, &payload.token, &routing, payload.payload, Some(&nonce)).await { + Ok(status) => status, + // Nobody called, so there is nobody to answer: a fire the + // trigger takes no work for (wiped, past its hibernation), one + // its full parked queue cannot hold, or an answer a run already + // has is dropped, and says why. + Err((code @ (StatusCode::GONE | StatusCode::TOO_MANY_REQUESTS | StatusCode::CONFLICT), msg)) => { + tracing::info!( + target: "weft_dispatcher::fire_signal", + token = %payload.token, project_id = %routing.project_id, + "fire dropped: {msg}" + ); + // The trigger takes no work, so its listener should not be + // holding the signal: it lets go (and holds it again if a + // reactivation lands meanwhile). A row gone was let go of by + // whatever removed it. + if code == StatusCode::GONE { + if let Some(signal) = state.journal.signal_get(&payload.token).await? { + // What it could not let go of is said where it failed; + // the next fire tries again. + if let Err(e) = crate::listener::let_go_of_stopped(&state.pg_pool, &state.listener, std::slice::from_ref(&signal)).await { + tracing::warn!( + target: "weft_dispatcher::fire_signal", + token = %payload.token, error = %format!("{e:#}"), + "could not look whether a signal let go of is wanted again; its next fire tries again" + ); + } + } + } + code + } + Err((code, msg)) => anyhow::bail!("fire of {} {code}: {msg}", payload.token), + }; Ok(serde_json::json!({ "status": status.as_u16() })) } } + diff --git a/crates/weft-dispatcher/src/task_kinds/program_call.rs b/crates/weft-dispatcher/src/task_kinds/program_call.rs index fb2f3e2a..5c900f6a 100644 --- a/crates/weft-dispatcher/src/task_kinds/program_call.rs +++ b/crates/weft-dispatcher/src/task_kinds/program_call.rs @@ -268,18 +268,18 @@ async fn infra_down( // no row, or one mid-terminate, may still hold the disks an earlier // terminate kept, and only the supervisor's pass over the host can // tell. - let already_down = match (row, take_down) { - (_, TakeDown::Terminate { disks: weft_core::infra::TerminateDisks::DeleteAll }) => false, - (None, _) => true, - (Some(InfraNodeStatus::Stopped | InfraNodeStatus::Stopping), TakeDown::Stop { .. }) => true, - (Some(InfraNodeStatus::Terminating), TakeDown::Terminate { .. }) => true, - (Some(_), _) => false, + use weft_core::program::InfraDownAnswer; + let answered = |answer: InfraDownAnswer| -> Result { + Ok(ProgramCallOutcome { value: serde_json::to_value(answer).map_err(internal("answer"))?, stops_asker: false }) }; - if already_down { - return Ok(ProgramCallOutcome { - value: serde_json::json!({ "issued": false, "cancelled_start": cancelled_start }), - stops_asker: false, - }); + match (row, take_down) { + (_, TakeDown::Terminate { disks: weft_core::infra::TerminateDisks::DeleteAll }) => {} + // A start of it still in its setup run was the copy, and is gone. + (None, _) if cancelled_start => return answered(InfraDownAnswer::TakenDown), + (None, _) => return answered(InfraDownAnswer::NoCopy), + (Some(InfraNodeStatus::Stopped | InfraNodeStatus::Stopping), TakeDown::Stop { .. }) + | (Some(InfraNodeStatus::Terminating), TakeDown::Terminate { .. }) => return answered(InfraDownAnswer::AlreadyDown), + (Some(_), _) => {} } let copies = Copies::of(instance.cloned()); let nodes = std::collections::BTreeSet::from([node.to_string()]); @@ -297,7 +297,13 @@ async fn infra_down( project_id, &crate::take_down::TakeDownTarget::Activations(live_readers), spec, - false, + // Down with the copy: the copy's start brings it back. A wipe + // takes the copy away for good (its owner is going), so nothing + // is left to bring them back. + match take_down { + TakeDown::Terminate { disks: weft_core::infra::TerminateDisks::DeleteAll } => None, + TakeDown::Stop { .. } | TakeDown::Terminate { .. } => Some(crate::take_down::DownWith::Infra), + }, Some(asker), ) .await?; @@ -326,7 +332,8 @@ async fn infra_down( .await?; let stops_asker = stop_self == StopSelf::Include && asker_uses_it; stop_asker(state, asker, stops_asker).await?; - Ok(ProgramCallOutcome { value: serde_json::json!({ "issued": true, "command": command_id }), stops_asker }) + tracing::info!(target: "weft_dispatcher::program_call", %project_id, node, command = %command_id, "a program took an infra copy down"); + Ok(ProgramCallOutcome { value: serde_json::to_value(InfraDownAnswer::TakenDown).map_err(internal("answer"))?, stops_asker }) } /// Activate the activations `scope` names. Those already active answer @@ -376,7 +383,7 @@ async fn triggers_deactivate( let target = crate::take_down::TakeDownTarget::Activations(keys); let runs = crate::take_down::live_runs(state, project_id).await.map_err(internal("live runs"))?; let asker_fired = crate::take_down::affected_runs(&target, &runs, None).iter().any(|r| r.execution_id == asker); - crate::take_down::take_down(state, project_id, &target, spec, false, Some(asker)).await?; + crate::take_down::take_down(state, project_id, &target, spec, None, Some(asker)).await?; let stops_asker = stop_self == StopSelf::Include && asker_fired; stop_asker(state, asker, stops_asker).await?; Ok(ProgramCallOutcome { value: serde_json::json!({ "deactivated": true }), stops_asker }) diff --git a/crates/weft-dispatcher/tests/db_lifecycle.rs b/crates/weft-dispatcher/tests/db_lifecycle.rs index f27ef47f..ba1eb6d8 100644 --- a/crates/weft-dispatcher/tests/db_lifecycle.rs +++ b/crates/weft-dispatcher/tests/db_lifecycle.rs @@ -289,11 +289,11 @@ async fn an_instances_wipe_takes_its_row_and_its_signals_in_one_write(pool: PgPo assert!(journal.signal_get("ada-wiped-entry").await.unwrap().is_none(), "and its signal went with it"); } -/// A park keeps the signals and hands back, from its own transaction, the -/// ones its activations govern: the listener cleanup never reads them -/// after a reactivation could have registered fresh ones. +/// A park keeps the signals and leaves them listening: what they hear +/// waits for the trigger to be back on, so the write hands nothing back +/// for the listener to let go of. #[sqlx::test] -async fn a_park_hands_back_the_signals_it_keeps(pool: PgPool) { +async fn a_park_keeps_its_signals_listening(pool: PgPool) { let (journal, projects) = setup(&pool).await; let activations = PostgresActivationStore::new(pool.clone()); let id = Uuid::new_v4(); @@ -308,9 +308,45 @@ async fn a_park_hands_back_the_signals_it_keeps(pool: PgPool) { .await .unwrap(); let LifecycleWrite::Applied { unlisten } = written else { panic!("applied: {written:?}") }; - assert_eq!(unlisten.iter().map(|s| s.token.as_str()).collect::>(), vec!["parked-entry"], - "the kept signals are handed back for the listener cleanup"); - assert!(journal.signal_get("parked-entry").await.unwrap().is_some(), "and stay stored for the reactivation"); + assert!(unlisten.is_empty(), "a parked trigger goes on listening: {unlisten:?}"); + assert!(journal.signal_get("parked-entry").await.unwrap().is_some(), "and its signal stays stored"); +} + +/// A hibernation takes work until its grace window ends, then stops: the +/// end flips only the windows that are over, keeps them hibernating (the +/// row stays, so `weft activate` brings it back), and says when the next +/// window still open ends. +#[sqlx::test] +async fn a_hibernation_stops_taking_work_once_its_window_ends(pool: PgPool) { + use weft_dispatcher::activation_store::{end_grace_windows, grace_windows_over, next_grace_end}; + let (_journal, projects) = setup(&pool).await; + let activations = PostgresActivationStore::new(pool.clone()); + let id = Uuid::new_v4(); + seed_project(&projects, id, "bin-A").await; + let (over, open) = (feed_key(), ActivationKey::new("reads", Owner::Shared)); + let setup_execution_id = Uuid::new_v4(); + assert!(activations.try_begin_activating(id, &[over.clone(), open.clone()], setup_execution_id, None).await.unwrap().is_ok()); + activations.end_activating(id, setup_execution_id, &ActivationLifecycle::active(), false, None).await.unwrap().expect("owned"); + for (key, deadline) in [(&over, 100), (&open, i64::MAX / 2)] { + activations + .set_lifecycle_guarded(id, std::slice::from_ref(key), &ActivationLifecycle::hibernating(deadline), SignalsGoing::Kept) + .await + .unwrap(); + } + assert_eq!(next_grace_end(&pool).await.unwrap(), Some(100)); + + // The windows are over by the database's clock: one in the past, one + // far ahead. + assert_eq!(grace_windows_over(&pool).await.unwrap(), vec![(id, over.clone())]); + assert_eq!(end_grace_windows(&pool, id, &[over.clone(), open.clone()]).await.unwrap(), 1, "only the window that is over ends"); + assert!(grace_windows_over(&pool).await.unwrap().is_empty(), "taking no work, it is over no longer"); + assert_eq!(end_grace_windows(&pool, id, std::slice::from_ref(&over)).await.unwrap(), 0, "a window ends once"); + let listed = activations.list(id).await.unwrap(); + let lifecycle = |key: &ActivationKey| listed.iter().find(|a| &a.key == key).expect("kept").lifecycle.clone(); + assert!(!lifecycle(&over).accepting_fires, "the window that is over takes no more work"); + assert_eq!(lifecycle(&over).mode(), weft_core::activation::ActivationMode::Down(weft_core::DeactivationMode::Hibernate), "and is still a hibernation"); + assert!(lifecycle(&open).accepting_fires, "the window still open takes work"); + assert_eq!(next_grace_end(&pool).await.unwrap(), Some(i64::MAX / 2)); } #[sqlx::test] @@ -899,6 +935,8 @@ async fn quiesce_waits_until_no_run_of_the_project_is_live(pool: PgPool) { let mut tx = pool.begin().await.unwrap(); weft_journal::unrecorded::forget_in(&mut tx, execution_id).await.unwrap(); tx.commit().await.unwrap(); + // What every caller of `forget_in` does once its transaction commits. + weft_task_store::announce::committed(&pool); tokio::time::timeout(std::time::Duration::from_secs(5), waiter) .await .expect("the ending wakes the quiesce at once") @@ -957,6 +995,8 @@ async fn forgetting_an_unrecorded_run_announces_its_ending_at_the_commit(pool: P let before = tokio::time::timeout(std::time::Duration::from_millis(300), heard_next(&mut heard)).await; assert!(before.is_err(), "nothing is announced before the commit"); tx.commit().await.unwrap(); + // What every caller of `forget_in` does once its transaction commits. + weft_task_store::announce::committed(&pool); let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5); let announced = loop { let got = tokio::time::timeout_at(deadline, heard_next(&mut heard)).await.expect("the ending is announced"); @@ -2124,14 +2164,19 @@ async fn the_holders_count_the_held_rows_of_live_activations(pool: PgPool) { assert_eq!(weft_dispatcher::holders::held_count(&pool).await.unwrap(), 2); sqlx::query( "INSERT INTO trigger_activation (project_id, trigger, status, accepting_fires, fires_visible_to_consumers, updated_at) \ - VALUES ($1, 'parked', 'inactive', FALSE, FALSE, 0)", + VALUES ($1, 'wiped', 'inactive', FALSE, FALSE, 0)", ) .bind(id) .execute(&pool) .await .unwrap(); - sqlx::query("UPDATE signal SET activation_trigger = 'parked' WHERE token = 'two'").execute(&pool).await.unwrap(); - assert_eq!(weft_dispatcher::holders::held_count(&pool).await.unwrap(), 1, "a parked activation's row holds nothing"); + sqlx::query("UPDATE signal SET activation_trigger = 'wiped' WHERE token = 'two'").execute(&pool).await.unwrap(); + assert_eq!(weft_dispatcher::holders::held_count(&pool).await.unwrap(), 1, "a wiped activation's row holds nothing"); + sqlx::query("UPDATE trigger_activation SET accepting_fires = TRUE, fires_visible_to_consumers = TRUE WHERE trigger = 'wiped'") + .execute(&pool) + .await + .unwrap(); + assert_eq!(weft_dispatcher::holders::held_count(&pool).await.unwrap(), 2, "a parked one's goes on holding: what it hears waits"); } /// The holders run as many copies as the held signals need, none for none, diff --git a/crates/weft-dispatcher/tests/db_tags.rs b/crates/weft-dispatcher/tests/db_tags.rs index 7515266f..b6c06113 100644 --- a/crates/weft-dispatcher/tests/db_tags.rs +++ b/crates/weft-dispatcher/tests/db_tags.rs @@ -453,3 +453,78 @@ async fn an_instance_token_is_one_project_and_expires(pool: PgPool) { assert_eq!(left.len(), 2, "{left:?}"); assert!(left.contains(&"bob-1".to_string()) && left.contains(&"author".to_string())); } + +/// The listing's `node` and `search` filters: `node` finds the runs a +/// node started in, `search` the finished runs whose recorded values carry +/// every word asked for. A run still going has no search document yet, and +/// a run's document is built once, whatever asks twice. +#[sqlx::test] +async fn runs_are_found_by_a_node_they_ran_and_by_what_went_through_them(pool: PgPool) { + use std::sync::Arc; + let (journal, projects) = setup(&pool).await; + let project = Uuid::new_v4(); + seed_project(&projects, project).await; + let ran = |execution_id, value: &'static str| { + let journal = &journal; + async move { + for event in [ + ExecEvent::NodeStarted { execution_id, node_id: "wait".into(), frames: vec![], at_unix: 1 }, + ExecEvent::PortEmitted { + execution_id, + emission_id: Uuid::new_v4(), + node_id: "wait".into(), + frames: vec![], + port: "out".into(), + value: Arc::new(json!({ "email": value, "order": 4217 })), + provided: false, + at_unix: 2, + }, + ] { + journal.record_event(&event).await.expect("node event"); + } + } + }; + let ada = start_execution(&journal, project, weft_core::exec::RunKind::Execution).await; + let bob = start_execution(&journal, project, weft_core::exec::RunKind::Execution).await; + let idle = start_execution(&journal, project, weft_core::exec::RunKind::Execution).await; + let going = start_execution(&journal, project, weft_core::exec::RunKind::Execution).await; + ran(ada, "ada@example.com").await; + ran(bob, "bob@example.com").await; + ran(going, "ada@example.com").await; + for finished in [ada, bob, idle] { + journal + .record_event(&ExecEvent::ExecutionCompleted { execution_id: finished, at_unix: 3 }) + .await + .expect("ExecutionCompleted"); + weft_dispatcher::run_search::index_finished_run(&pool, finished).await.expect("index"); + } + weft_dispatcher::run_search::index_finished_run(&pool, ada).await.expect("a second index of the same run"); + + let listed = |node: Option<&'static str>, search: Option<&'static str>| { + let journal = &journal; + async move { + let query = weft_dispatcher::journal::ExecutionQuery { + limit: 10, + node: node.map(Into::into), + search: search.map(Into::into), + ..Default::default() + }; + let page = journal.list_executions(TENANT, &query).await.unwrap(); + let mut ids: Vec<_> = page.executions.into_iter().map(|s| s.execution_id).collect(); + assert_eq!(page.total, ids.len() as u64, "the count agrees with the list"); + ids.sort(); + ids + } + }; + let sorted = |mut ids: Vec| { + ids.sort(); + ids + }; + assert_eq!(listed(Some("wait"), None).await, sorted(vec![ada, bob, going])); + assert_eq!(listed(Some("elsewhere"), None).await, Vec::::new()); + assert_eq!(listed(None, Some("ada@example.com")).await, vec![ada], "the run still going has no document yet"); + assert_eq!(listed(None, Some("4217")).await, sorted(vec![ada, bob]), "numbers are words too"); + assert_eq!(listed(None, Some("ada@example.com 4217")).await, vec![ada], "every word asked for"); + assert_eq!(listed(None, Some("email")).await, Vec::::new(), "a field's name is not one of its values"); + assert_eq!(listed(Some("wait"), Some("bob@example.com")).await, vec![bob]); +} diff --git a/crates/weft-e2e/tests/api_reply.rs b/crates/weft-e2e/tests/api_reply.rs index 991ce7c3..7c7697f9 100644 --- a/crates/weft-e2e/tests/api_reply.rs +++ b/crates/weft-e2e/tests/api_reply.rs @@ -9,7 +9,7 @@ use reqwest::Method; use serde_json::{json, Value}; -use weft_e2e::{ensure, live, project::Project, run::SettledRun}; +use weft_e2e::{ensure, live, project::Project, run::SettledRun, status}; #[path = "common/mod.rs"] mod common; @@ -216,3 +216,41 @@ async fn firing_a_get_route_needs_no_body() -> anyhow::Result<()> { project.finish().await } + +/// A parked route keeps its callers: a call that arrives while the +/// project is parked waits, and is answered by the program once it is +/// back, the same way a parked trigger keeps its fires. A wipe is the +/// other answer: the route is gone with its trigger's signals, and a +/// caller is refused at once. +#[tokio::test] +async fn a_parked_route_holds_its_caller_until_it_is_back() -> anyhow::Result<()> { + let disp = ensure::up().await?; + let mut project = Project::prepare("api_reply", disp.clone()).await?; + let base = project.unique_live_path()?; + project.activate().await?; + + project.weft(&["deactivate", "--mode", "park", "--running-policy", "cancel"]).await?; + status::wait_until_status(&disp, &project.id(), "inactive", status::STATUS_DEADLINE).await?; + + let call = { + let disp = disp.clone(); + let path = format!("{base}/users/42"); + tokio::spawn(async move { live::http_request(&disp, Method::GET, &path, &[], None).await }) + }; + tokio::time::sleep(std::time::Duration::from_secs(2)).await; + anyhow::ensure!(!call.is_finished(), "a parked route answered at once: {:?}", call.await); + + project.activate().await?; + let (status, _, body) = call.await??; + assert_eq!(status, 200, "{}", String::from_utf8_lossy(&body)); + let body: Value = serde_json::from_slice(&body)?; + assert_eq!(body["name"], "Ada", "{body}"); + + project.weft(&["deactivate", "--mode", "wipe", "--running-policy", "cancel"]).await?; + status::wait_until_status(&disp, &project.id(), "inactive", status::STATUS_DEADLINE).await?; + let (status, _, body) = + live::http_request(&disp, Method::GET, &format!("{base}/users/42"), &[], None).await?; + assert_eq!(status, 404, "{}", String::from_utf8_lossy(&body)); + + project.finish().await +} diff --git a/crates/weft-e2e/tests/lifecycle_transitions.rs b/crates/weft-e2e/tests/lifecycle_transitions.rs index 73572167..e8dffe43 100644 --- a/crates/weft-e2e/tests/lifecycle_transitions.rs +++ b/crates/weft-e2e/tests/lifecycle_transitions.rs @@ -368,11 +368,11 @@ async fn an_infra_stop_takes_down_only_what_reads_it_and_recovers() -> anyhow::R } /// An infra stop takes down only the triggers reading that infra, so the -/// project is left partly on: once the infra runs again, `weft activate` -/// turns back on the trigger that is off and leaves the other as it is, and -/// the trigger it turned back on fires. +/// project is left partly on; once the infra runs again, the trigger it took +/// down comes back on by itself and fires. One a person switched off +/// themselves stays off through the next stop and start. #[tokio::test] -async fn activate_turns_back_on_only_the_triggers_an_infra_stop_took_down() -> anyhow::Result<()> { +async fn an_infra_start_brings_back_the_triggers_its_stop_took_down() -> anyhow::Result<()> { let disp = ensure::up().await?; let mut project = Project::prepare("lifecycle", disp.clone()).await?; let pid = project.id(); @@ -412,14 +412,13 @@ async fn activate_turns_back_on_only_the_triggers_an_infra_stop_took_down() -> a .await? .assert_actions_exactly(&["deactivate", "infra_start", "infra_terminate"])?; + // The start brings `reads` back with its infra: nothing is left off. infra::start_and_wait_running(&mut project, INFRA_NODE).await?; - status::wait_until_status(&disp, &pid, "active", STATUS_DEADLINE) - .await? - .assert_actions_exactly(&["run", "deactivate", "activate", "infra_stop", "infra_terminate"])?; - project.activate().await?; - status::wait_until_status(&disp, &pid, "active", STATUS_DEADLINE) - .await? - .assert_actions_exactly(&["run", "deactivate", "infra_stop", "infra_terminate"])?; + status::wait_until(&disp, &pid, "the start brought the trigger back", STATUS_DEADLINE, |s| { + s.status() == "active" && !s.available_actions().contains(&"activate") + }) + .await? + .assert_actions_exactly(&["run", "deactivate", "infra_stop", "infra_terminate"])?; // `plain` never let go of the feed, so one subscriber says nothing about // `reads`: wait until both are listening before pushing. @@ -431,6 +430,20 @@ async fn activate_turns_back_on_only_the_triggers_an_infra_stop_took_down() -> a .completed()? .assert_input("heard", "data", &json!("go"))?; + // Switched off by a person, `reads` stays off through a stop and a + // start: only what the stop itself took down comes back. + project + .weft(&["deactivate", "--trigger", "reads", "--mode", "park", "--running-policy", "cancel"]) + .await?; + project + .weft(&["infra", "stop", "--mode", "park", "--running-policy", "cancel"]) + .await?; + status::wait_until(&disp, &pid, "the infra stopped", STATUS_DEADLINE, |s| s.infra_rollup() == "stopped").await?; + infra::start_and_wait_running(&mut project, INFRA_NODE).await?; + status::wait_until_status(&disp, &pid, "active", STATUS_DEADLINE) + .await? + .assert_actions_exactly(&["run", "deactivate", "activate", "infra_stop", "infra_terminate"])?; + project .weft(&["deactivate", "--mode", "wipe", "--running-policy", "cancel"]) .await?; diff --git a/crates/weft-engine/src/context.rs b/crates/weft-engine/src/context.rs index 46bae29b..aa91cef7 100644 --- a/crates/weft-engine/src/context.rs +++ b/crates/weft-engine/src/context.rs @@ -108,6 +108,60 @@ pub struct EngineClients { /// Charges opened by a call whose amount only a later response /// states, held process-wide until that response lands. pub open_charges: Arc, + /// What the process's line to the broker pushes, which is how a worker + /// hears that a run it drives is asked to stop. + pub line: Arc, +} + +/// What a worker reads off its line to the broker besides answers: the +/// notices the broker pushes, and a hold that keeps the line open while +/// the worker must hear them (`weft_broker_client::line`'s module doc). +pub trait WorkerLine: Send + Sync { + fn subscribe(&self) -> weft_task_store::pg_signal::Subscription; + fn stay_open(&self) -> Box; +} + +impl WorkerLine for weft_broker_client::BrokerLink { + fn subscribe(&self) -> weft_task_store::pg_signal::Subscription { + weft_broker_client::BrokerLink::subscribe(self) + } + + fn stay_open(&self) -> Box { + Box::new(weft_broker_client::BrokerLink::stay_open(self)) + } +} + +/// A line whose notices are what a test sends into [`TestLine::pushed`]. +#[cfg(test)] +pub(crate) struct TestLine { + pub pushed: tokio::sync::broadcast::Sender, + /// How many holds are taken right now. + pub held: Arc, +} + +#[cfg(test)] +impl TestLine { + pub(crate) fn new() -> Arc { + Arc::new(Self { pushed: tokio::sync::broadcast::channel(64).0, held: Arc::default() }) + } +} + +#[cfg(test)] +impl WorkerLine for TestLine { + fn subscribe(&self) -> weft_task_store::pg_signal::Subscription { + self.pushed.subscribe().into() + } + + fn stay_open(&self) -> Box { + struct Hold(Arc); + impl Drop for Hold { + fn drop(&mut self) { + self.0.fetch_sub(1, std::sync::atomic::Ordering::SeqCst); + } + } + self.held.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + Box::new(Hold(self.held.clone())) + } } impl EngineClients { @@ -134,6 +188,7 @@ impl EngineClients { steering: weft_broker_client::BrokerExecutionClient::new(link.clone()), pending_costs: crate::metering::PendingCostRecords::new(), open_charges: crate::metering::OpenCharges::new(), + line: Arc::new(link.clone()), } } } @@ -1575,13 +1630,12 @@ impl TaskStoreClient for JournalFirst { self.inner.fail(task_id, replica, error).await } - async fn wait_cancels( + async fn cancels_asked( &self, project_id: uuid::Uuid, execution_ids: Vec, - wait: Duration, ) -> anyhow::Result> { - self.inner.wait_cancels(project_id, execution_ids, wait).await + self.inner.cancels_asked(project_id, execution_ids).await } } @@ -3728,6 +3782,7 @@ mod replay_tests { access_broker: FakeAccessBroker::new(), pending_costs: crate::metering::PendingCostRecords::new(), open_charges: crate::metering::OpenCharges::new(), + line: TestLine::new(), steering: Arc::new(NoopSteering), }; RunnerHandle::new( @@ -3799,6 +3854,7 @@ mod replay_tests { access_broker, pending_costs: crate::metering::PendingCostRecords::new(), open_charges: crate::metering::OpenCharges::new(), + line: TestLine::new(), steering: Arc::new(NoopSteering), }; RunnerHandle::new( @@ -4065,6 +4121,7 @@ mod replay_tests { access_broker: worker_broker.clone(), pending_costs: crate::metering::PendingCostRecords::new(), open_charges: crate::metering::OpenCharges::new(), + line: TestLine::new(), steering: Arc::new(NoopSteering), }; let execution_id = uuid::Uuid::from_u128(0xC0); @@ -4105,6 +4162,7 @@ mod replay_tests { access_broker: test_broker.clone(), pending_costs: crate::metering::PendingCostRecords::new(), open_charges: crate::metering::OpenCharges::new(), + line: TestLine::new(), steering: Arc::new(NoopSteering), }; let runner = crate::test_rig::LiveTestRunner::new( @@ -4299,11 +4357,10 @@ mod replay_tests { struct NoopTaskStore; #[async_trait] impl TaskStoreClient for NoopTaskStore { - async fn wait_cancels( + async fn cancels_asked( &self, _project_id: uuid::Uuid, _execution_ids: Vec, - _wait: std::time::Duration, ) -> anyhow::Result> { Ok(Vec::new()) } @@ -5072,7 +5129,7 @@ mod drive_journal_tests { async fn fail(&self, _: uuid::Uuid, _: &str, _: String) -> anyhow::Result<()> { unimplemented!("not asked") } - async fn wait_cancels(&self, _: uuid::Uuid, _: Vec, _: Duration) -> anyhow::Result> { + async fn cancels_asked(&self, _: uuid::Uuid, _: Vec) -> anyhow::Result> { unimplemented!("not asked") } } diff --git a/crates/weft-engine/src/execution_driver_tests/bus_comm.rs b/crates/weft-engine/src/execution_driver_tests/bus_comm.rs index e094905e..c17830f3 100644 --- a/crates/weft-engine/src/execution_driver_tests/bus_comm.rs +++ b/crates/weft-engine/src/execution_driver_tests/bus_comm.rs @@ -1210,11 +1210,10 @@ } #[async_trait] impl weft_task_store::TaskStoreClient for AwaitTasks { - async fn wait_cancels( + async fn cancels_asked( &self, _project_id: uuid::Uuid, _execution_ids: Vec, - _wait: std::time::Duration, ) -> anyhow::Result> { Ok(Vec::new()) } diff --git a/crates/weft-engine/src/execution_driver_tests/rig.rs b/crates/weft-engine/src/execution_driver_tests/rig.rs index 62d78e11..06edb871 100644 --- a/crates/weft-engine/src/execution_driver_tests/rig.rs +++ b/crates/weft-engine/src/execution_driver_tests/rig.rs @@ -123,11 +123,10 @@ pub(super) struct NoopTasks; #[async_trait] impl weft_task_store::TaskStoreClient for NoopTasks { - async fn wait_cancels( + async fn cancels_asked( &self, _project_id: uuid::Uuid, _execution_ids: Vec, - _wait: std::time::Duration, ) -> anyhow::Result> { Ok(Vec::new()) } @@ -580,6 +579,7 @@ access_broker: crate::context::FakeAccessBroker::new(), pending_costs: crate::metering::PendingCostRecords::new(), open_charges: crate::metering::OpenCharges::new(), + line: crate::context::TestLine::new(), steering: Arc::new(NoopSteering), } } diff --git a/crates/weft-engine/src/lib.rs b/crates/weft-engine/src/lib.rs index d3d151f2..f6759ce9 100644 --- a/crates/weft-engine/src/lib.rs +++ b/crates/weft-engine/src/lib.rs @@ -10,6 +10,7 @@ pub(crate) mod context; pub(crate) mod execution_driver; pub(crate) mod fired_caller; pub(crate) mod held; +pub(crate) mod memory_guard; pub(crate) mod metering; pub(crate) mod socket; pub(crate) mod stream_runtime; diff --git a/crates/weft-engine/src/memory_guard.rs b/crates/weft-engine/src/memory_guard.rs new file mode 100644 index 00000000..3705303d --- /dev/null +++ b/crates/weft-engine/src/memory_guard.rs @@ -0,0 +1,136 @@ +//! Turning new work away while the worker's memory is nearly full. +//! +//! A worker takes many runs at once (`workers.concurrency`), and what one +//! run holds depends on the program: a run that loads a large file, or a +//! node that keeps a big value in memory, can take far more than another. +//! Past its memory the platform kills the whole worker, and every run on +//! it dies with it. So a worker refuses to start more work once its memory +//! is nearly full, answering `503` with a `Retry-After`: an outside caller +//! is told to try again in a moment, and a run weft hands it is handed +//! again on the next delivery. Runs already going are never touched. +//! +//! The limit is the container's own (its cgroup's `memory.max`); a +//! container whose cgroup sets none refuses nothing, and one that shows no +//! cgroup at all (a sandbox) reads its own total from `/proc/meminfo`. + +use axum::extract::Request; +use axum::http::{header, HeaderValue, StatusCode}; +use axum::middleware::Next; +use axum::response::{IntoResponse, Response}; + +/// How full the memory may be before new work is turned away. +const FULL_ABOVE: f64 = 0.90; + +/// What a refused caller is told to wait before asking again. +const RETRY_AFTER_SECS: u64 = 1; + +/// How much memory is in use and how much there is, in bytes. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Memory { + pub used: u64, + pub limit: u64, +} + +impl Memory { + /// Whether new work is turned away at this reading. + pub fn full(&self) -> bool { + self.limit > 0 && (self.used as f64) > (self.limit as f64) * FULL_ABOVE + } +} + +/// The container's memory as its cgroup (v2, then v1) counts it; `None` +/// when the cgroup sets no limit. Only where there is no cgroup at all (a +/// sandbox that shows none) is the machine's own count read, which there +/// is the sandbox's. +pub fn read() -> Option { + let file = |path: &str| std::fs::read_to_string(path).ok(); + for (current, max) in [ + ("/sys/fs/cgroup/memory.current", "/sys/fs/cgroup/memory.max"), + ("/sys/fs/cgroup/memory/memory.usage_in_bytes", "/sys/fs/cgroup/memory/memory.limit_in_bytes"), + ] { + if std::path::Path::new(max).exists() { + let (current, max) = (file(current), file(max)); + if current.is_none() || max.is_none() { + // The cgroup is there and cannot be read: the guard cannot + // see the memory, which is said once rather than per request. + static SAID: std::sync::Once = std::sync::Once::new(); + SAID.call_once(|| { + tracing::warn!( + target: "weft_engine::memory_guard", + "this container's memory cgroup cannot be read, so a worker nearly out of memory does not turn new work away" + ) + }); + } + return from_cgroup(current, max); + } + } + file("/proc/meminfo").as_deref().and_then(from_meminfo) +} + +/// A cgroup's reading: its usage and its limit, when it sets one (`max`, +/// or cgroup v1's huge "no limit" number, sets none). +fn from_cgroup(current: Option, max: Option) -> Option { + let used = current?.trim().parse::().ok()?; + let limit = max?.trim().parse::().ok()?; + // cgroup v1 says "no limit" with a number near the largest it holds. + (limit < 1 << 60).then_some(Memory { used, limit }) +} + +/// The machine's reading from `/proc/meminfo`: all of it, and all of it +/// less what is available. +fn from_meminfo(meminfo: &str) -> Option { + let kib = |name: &str| { + meminfo + .lines() + .find_map(|line| line.strip_prefix(name)?.trim().strip_suffix("kB")?.trim().parse::().ok()) + }; + let total = kib("MemTotal:")? * 1024; + let available = kib("MemAvailable:")? * 1024; + Some(Memory { used: total.saturating_sub(available), limit: total }) +} + +/// Middleware: refuse a request that would start work while the memory is +/// full (see the module doc). Every request a worker serves but its health +/// check starts work: a run weft hands it, or a live caller attaching, +/// which starts its run. The health check always answers. +pub async fn refuse_when_full(request: Request, next: Next) -> Response { + if request.uri().path() != "/_weft/healthz" { + if let Some(memory) = read().filter(Memory::full) { + tracing::warn!( + target: "weft_engine::memory_guard", + used = memory.used, limit = memory.limit, path = %request.uri().path(), + "the worker's memory is nearly full; new work is turned away until runs end" + ); + let mut answer = (StatusCode::SERVICE_UNAVAILABLE, "This worker is busy right now; try again in a moment.").into_response(); + answer.headers_mut().insert(header::RETRY_AFTER, HeaderValue::from(RETRY_AFTER_SECS)); + return answer; + } + } + next.run(request).await +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_cgroup_with_a_limit_is_read_and_one_without_is_not() { + assert_eq!(from_cgroup(Some("100\n".into()), Some("1000\n".into())), Some(Memory { used: 100, limit: 1000 })); + assert_eq!(from_cgroup(Some("100\n".into()), Some("max\n".into())), None); + assert_eq!(from_cgroup(Some("100".into()), Some("9223372036854771712".into())), None, "cgroup v1's no limit"); + assert_eq!(from_cgroup(None, Some("1000".into())), None); + } + + #[test] + fn meminfo_reads_total_and_available() { + let meminfo = "MemTotal: 1000 kB\nMemFree: 100 kB\nMemAvailable: 250 kB\n"; + assert_eq!(from_meminfo(meminfo), Some(Memory { used: 750 * 1024, limit: 1000 * 1024 })); + } + + #[test] + fn work_is_turned_away_only_past_the_line() { + assert!(!Memory { used: 900, limit: 1000 }.full()); + assert!(Memory { used: 901, limit: 1000 }.full()); + assert!(!Memory { used: 5, limit: 0 }.full(), "no limit read refuses nothing"); + } +} diff --git a/crates/weft-engine/src/metering.rs b/crates/weft-engine/src/metering.rs index 829e7643..390b6ff0 100644 --- a/crates/weft-engine/src/metering.rs +++ b/crates/weft-engine/src/metering.rs @@ -964,11 +964,10 @@ mod tests { #[async_trait::async_trait] impl weft_task_store::TaskStoreClient for RecordingTaskStore { - async fn wait_cancels( + async fn cancels_asked( &self, _project_id: uuid::Uuid, _execution_ids: Vec, - _wait: std::time::Duration, ) -> anyhow::Result> { Ok(Vec::new()) } diff --git a/crates/weft-engine/src/socket.rs b/crates/weft-engine/src/socket.rs index c64e6185..2b9fab0d 100644 --- a/crates/weft-engine/src/socket.rs +++ b/crates/weft-engine/src/socket.rs @@ -431,11 +431,10 @@ mod tests { #[async_trait::async_trait] impl weft_task_store::TaskStoreClient for RecordingTaskStore { - async fn wait_cancels( + async fn cancels_asked( &self, _project_id: uuid::Uuid, _execution_ids: Vec, - _wait: std::time::Duration, ) -> anyhow::Result> { Ok(Vec::new()) } diff --git a/crates/weft-engine/src/worker.rs b/crates/weft-engine/src/worker.rs index 59b7f7e9..f6542889 100644 --- a/crates/weft-engine/src/worker.rs +++ b/crates/weft-engine/src/worker.rs @@ -163,8 +163,8 @@ struct ExecutionResidue { /// it: the exact failure this guard exists to prevent, moved into /// the gap between the guard and the task. flag: Arc, - /// Poked once the flag is gone, so the cancel wait stops asking - /// for this execution. + /// Poked once the flag is gone, so the cancel watch lets the line go + /// once nothing is driven. driving_changed: Arc, caller_registry: crate::caller_conn::CallerRegistry, live_configs: LiveConfigMap, @@ -224,6 +224,13 @@ impl Drop for ExecutionResidue { } } +/// A claimed execute or resume task, with how many rechecks the cancel +/// watch had made before the claim ([`Worker::rechecks`]). +struct Claim { + claimed: ClaimedExecution, + rechecks: u64, +} + /// One worker process: what every drive on it shares. /// /// The `ProjectDefinition` is not held here: each execution names its @@ -237,8 +244,20 @@ struct Worker { tenant_id: String, cancel_registry: CancelRegistry, /// Poked whenever the set of executions this worker drives changes, so - /// the cancel wait re-asks for exactly those. + /// the cancel watch holds the line open exactly while there are some. driving_changed: Arc, + /// Cancels heard for executions this worker does not drive (yet): a + /// cancel can be announced between a claim and the moment its drive + /// registers its flag, which then takes it. Read and written only + /// under the cancel registry's lock, so a cancel heard and a flag + /// registered never miss each other. + early_cancels: Arc>>, + /// How many times the cancel watch asked for the cancels of every + /// execution driven, because the line may have missed some. A drive + /// that registers after one that came since its claim asks for its own + /// cancels: that ask could not include it, and what it missed is not + /// announced again. + rechecks: Arc, project_cache: ProjectCache, /// The live caller registry: the connection server attaches an /// accepted socket here keyed by execution; the execute path awaits it. @@ -301,25 +320,31 @@ impl Worker { /// `execution_id`'s execute or resume task, claimed by this worker /// with the execution's journal, or `None` when there is nothing here /// to claim. - async fn claim_execution_id(&self, execution_id: ExecutionId) -> Result> { - self.clients + async fn claim_execution_id(&self, execution_id: ExecutionId) -> Result> { + // Read before the claim: a recheck after it may have come before + // the drive registers (`register_cancel_flag`). + let rechecks = self.rechecks.load(std::sync::atomic::Ordering::SeqCst); + Ok(self + .clients .tasks .claim_execution(&self.replica, self.project_id, &execution_id.to_string()) .await - .context("claim the execution's task") + .context("claim the execution's task")? + .map(|claimed| Claim { claimed, rechecks })) } /// Drive a claimed task to its end on a task of its own, so whoever /// handed it here can go away without stopping it. - fn drive_detached(&self, claimed: ClaimedExecution) -> tokio::task::JoinHandle { + fn drive_detached(&self, claim: Claim) -> tokio::task::JoinHandle { let worker = self.clone(); let token = self.background.token(); tokio::spawn(async move { let _token = token; let store = worker.clients.tasks.clone(); let replica = worker.replica.clone(); - let ClaimedExecution { task, journal } = claimed; - let end = weft_task_store::run_claimed_worker_task(store, &replica, &task, worker.drive(&task, journal)).await; + let Claim { claimed: ClaimedExecution { task, journal }, rechecks } = claim; + let end = + weft_task_store::run_claimed_worker_task(store, &replica, &task, worker.drive(&task, journal, rechecks)).await; RunAnswer::from(end) }) } @@ -328,7 +353,7 @@ impl Worker { /// claim read and run the loop driver. The two are identical here (the /// journal carries the lifecycle truth); the dispatcher distinguishes /// them so the editor can label the event. - async fn drive(&self, task: &Task, journal: Vec) -> Result<()> { + async fn drive(&self, task: &Task, journal: Vec, rechecks: u64) -> Result<()> { let ctx = self; let payload: ExecutionPayload = serde_json::from_value(task.payload.clone())?; let execution_id: ExecutionId = payload @@ -389,9 +414,7 @@ impl Worker { }; let flag = CancellationFlag::new_arc(); - ctx.cancel_registry.lock().await.insert(execution_id, flag.clone()); - // The cancel wait now asks for this execution too. - ctx.driving_changed.notify_one(); + register_cancel_flag(ctx, execution_id, flag.clone(), rechecks).await; // From here on every exit path, including a panic, releases what // this execution registered on the worker. let _residue = ExecutionResidue { @@ -1103,44 +1126,158 @@ impl crate::caller_conn::ExecutionCanceller for RegistryCanceller { } -/// The cancel wait: one held call to the broker for the executions this worker -/// drives, re-asked whenever that set changes. A cancel it hears fires the -/// execution's flag at once. -fn spawn_cancel_wait(worker: Worker) { +/// How long a cancel heard for an execution this worker does not drive is +/// kept for a drive that registers just after: far longer than the moment +/// between a claim and its drive's registration, short enough that the +/// cancels of other workers' executions do not pile up. +const EARLY_CANCEL_KEPT: std::time::Duration = std::time::Duration::from_secs(120); + +/// The cancel watch: the broker pushes every cancel of this worker's +/// project down its line (`weft_task_store::tasks::CANCEL_CHANNEL`), and +/// the one for an execution this worker drives is asked for and fires its +/// flag at once. When the line may have missed some (it opened again, or +/// fell behind), the cancels of every execution driven are asked for in +/// one call. The line is held open while anything is driven, so a cancel +/// is heard however long a run goes without calling the broker. +fn spawn_cancel_watch(worker: Worker) { + // Subscribed before anything is driven, so no cancel is announced + // before the watch hears. + let mut heard = worker.clients.line.subscribe(); tokio::spawn(async move { + let mut held_open: Option> = None; loop { // Armed before the set is read, so a change between the read - // and the wait still ends the wait. + // and the wait still wakes it. let changed = worker.driving_changed.notified(); tokio::pin!(changed); changed.as_mut().enable(); - let execution_ids: Vec = worker.cancel_registry.lock().await.keys().map(|c| c.to_string()).collect(); - if execution_ids.is_empty() { - changed.await; - continue; + let driving = !worker.cancel_registry.lock().await.is_empty(); + match (driving, held_open.is_some()) { + (true, false) => held_open = Some(worker.clients.line.stay_open()), + (false, true) => held_open = None, + _ => {} } - let heard = tokio::select! { + let next = tokio::select! { _ = &mut changed => continue, - heard = worker.clients.tasks.wait_cancels(worker.project_id, execution_ids, weft_task_store::pg_signal::MAX_HOLD) => heard, + next = heard.next() => next, }; - match heard { - Ok(cancels) => { - for asked in cancels { - match asked.execution_id.parse::() { - Ok(execution_id) => cancel_execution_id(&worker.cancel_registry, execution_id, asked.cause).await, - Err(e) => tracing::error!(target: "weft_engine::worker", execution_id = %asked.execution_id, error = %e, "a cancel named no execution"), - } + match next { + Ok(weft_task_store::pg_signal::Heard::Signal { channel, payload }) if channel == weft_task_store::tasks::CANCEL_CHANNEL => { + let Some((project, execution)) = weft_task_store::tasks::parse_cancel_payload(&payload) else { + tracing::error!(target: "weft_engine::worker", %payload, "a cancel announcement that names no execution"); + continue; + }; + if project != worker.project_id { + continue; + } + match execution.parse::() { + Ok(execution_id) => heard_cancel(&worker, execution_id).await, + Err(e) => tracing::error!(target: "weft_engine::worker", %payload, error = %e, "a cancel announcement that names no execution"), + } + } + Ok(weft_task_store::pg_signal::Heard::Recheck) => { + // Counted before the registry is read: a drive that + // registers after the read sees the count moved and asks + // for its own (`register_cancel_flag`). + worker.rechecks.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + let driven: Vec = worker.cancel_registry.lock().await.keys().copied().collect(); + if !driven.is_empty() { + fire_cancels_asked(&worker, driven).await; } } + Ok(_) => {} Err(e) => { - tracing::warn!(target: "weft_engine::worker", error = %format!("{e:#}"), "cancel wait failed; asking again"); - tokio::time::sleep(std::time::Duration::from_secs(1)).await; + tracing::error!(target: "weft_engine::worker", error = %format!("{e:#}"), "the cancel watch stopped hearing the line"); + return; } } } }); } +/// Register `flag` as `execution_id`'s, so its cancel reaches it, and ask +/// for its cancels when one may have been missed: heard before it +/// registered, or announced while the line could not hear, which the +/// watch's ask for every execution driven (a recheck since `rechecks`, the +/// count read before the claim) did not include. +async fn register_cancel_flag(worker: &Worker, execution_id: ExecutionId, flag: Arc, rechecks: u64) { + let missed = { + let mut registry = worker.cancel_registry.lock().await; + registry.insert(execution_id, flag); + let heard_early = worker.early_cancels.lock().expect("early cancels").remove(&execution_id).is_some(); + heard_early || worker.rechecks.load(std::sync::atomic::Ordering::SeqCst) != rechecks + }; + // The cancel watch now holds the line open for this execution too. + worker.driving_changed.notify_one(); + if missed { + fire_cancels_asked(worker, vec![execution_id]).await; + } +} + +/// A cancel was announced for `execution_id`: asked for now if this worker +/// drives it, kept for a drive about to register otherwise. +async fn heard_cancel(worker: &Worker, execution_id: ExecutionId) { + let driven = { + let registry = worker.cancel_registry.lock().await; + let driven = registry.contains_key(&execution_id); + if !driven { + let mut early = worker.early_cancels.lock().expect("early cancels"); + early.retain(|_, heard| heard.elapsed() < EARLY_CANCEL_KEPT); + early.insert(execution_id, std::time::Instant::now()); + } + driven + }; + if driven { + fire_cancels_asked(worker, vec![execution_id]).await; + } +} + +/// How long an ask that failed waits before it is made again, at first and +/// at most. +const REASK_FIRST: std::time::Duration = std::time::Duration::from_secs(1); +const REASK_LONGEST: std::time::Duration = std::time::Duration::from_secs(30); + +/// Ask for the cancels of `execution_ids` and fire their flags. An ask that +/// fails (the broker could not answer, or its answer was lost) is made +/// again for as long as any of them is still driven here: asking only +/// reads, so the cancel is still there to find, and nothing would +/// announce it again. +async fn fire_cancels_asked(worker: &Worker, execution_ids: Vec) { + if let Err(e) = fire_cancels_asked_once(worker, &execution_ids).await { + tracing::warn!(target: "weft_engine::worker", error = %format!("{e:#}"), "could not ask for the cancels heard; asking again"); + let worker = worker.clone(); + tokio::spawn(async move { + let mut wait = REASK_FIRST; + loop { + tokio::time::sleep(wait).await; + let still: Vec = { + let registry = worker.cancel_registry.lock().await; + execution_ids.iter().copied().filter(|id| registry.contains_key(id)).collect() + }; + if still.is_empty() { + return; + } + match fire_cancels_asked_once(&worker, &still).await { + Ok(()) => return, + Err(e) => tracing::warn!(target: "weft_engine::worker", error = %format!("{e:#}"), "could not ask for the cancels heard; asking again"), + } + wait = (wait * 2).min(REASK_LONGEST); + } + }); + } +} + +async fn fire_cancels_asked_once(worker: &Worker, execution_ids: &[ExecutionId]) -> Result<()> { + let ids: Vec = execution_ids.iter().map(ToString::to_string).collect(); + for asked in worker.clients.tasks.cancels_asked(worker.project_id, ids).await? { + match asked.execution_id.parse::() { + Ok(execution_id) => cancel_execution_id(&worker.cancel_registry, execution_id, asked.cause).await, + Err(e) => tracing::error!(target: "weft_engine::worker", execution_id = %asked.execution_id, error = %e, "a cancel named no execution"), + } + } + Ok(()) +} + /// A short run's self-imposed end, just before the platform's cap: the /// execution is cancelled with a cause that names the lever, instead of /// being cut mid-node with no ending written. A run with a live caller has @@ -1192,6 +1329,8 @@ fn new_worker(catalog: Arc, clients: EngineClients, config: &Wo tenant_id: config.tenant_id.clone(), cancel_registry: Arc::new(Mutex::new(HashMap::new())), driving_changed: Arc::new(tokio::sync::Notify::new()), + early_cancels: Arc::default(), + rechecks: Arc::default(), project_cache: Arc::new(Mutex::new(BoundedProjectCache::new())), caller_registry: crate::caller_conn::CallerRegistry::new(), live_configs: Arc::new(std::sync::Mutex::new(HashMap::new())), @@ -1227,7 +1366,7 @@ pub async fn serve(catalog: Arc, clients: EngineClients, config weft_core::net::install_crypto_provider(); weft_core::time_scale::announce(); let worker = new_worker(catalog, clients, &config); - spawn_cancel_wait(worker.clone()); + spawn_cancel_watch(worker.clone()); // SYNC: the `/_weft` prefix <-> weft_core::route::RESERVED_SEGMENT let own = Router::new() .route("/_weft/run/{execution_id}", post(run_handler)) @@ -1251,6 +1390,7 @@ pub async fn serve(catalog: Arc, clients: EngineClients, config own } }; + let app = app.layer(axum::middleware::from_fn(crate::memory_guard::refuse_when_full)); crate::caller_conn::serve(app.layer(axum::middleware::map_response(mark_worker_answer)), config.port, shutdown_signal()).await?; wind_down(&worker).await; Ok(()) @@ -1270,7 +1410,7 @@ pub async fn run_long(catalog: Arc, clients: EngineClients, con weft_core::net::install_crypto_provider(); weft_core::time_scale::announce(); let worker = new_worker(catalog, clients, &WorkerConfig { short_run_cap: None, ..config }); - spawn_cancel_wait(worker.clone()); + spawn_cancel_watch(worker.clone()); let answer = tokio::select! { answer = worker.run_execution_id(execution_id) => answer?, _ = shutdown_signal() => { @@ -1434,22 +1574,22 @@ mod tests { assert_eq!(serde_json::to_value(RunAnswer::NothingToRun).unwrap(), serde_json::json!({ "ended": "nothing_to_run" })); } - /// Answers one scripted cancel for the first execution it is asked about, - /// then holds each later wait for its full length. - struct OneCancel { + /// Answers the cancels it holds for the executions it is asked about + /// (asking only reads, so every ask finds them), and records every + /// ask. + #[derive(Default)] + struct Cancels { asked: std::sync::Mutex>>, - given: std::sync::atomic::AtomicBool, + waiting: std::sync::Mutex>, } #[async_trait] - impl weft_task_store::TaskStoreClient for OneCancel { - async fn wait_cancels(&self, _p: uuid::Uuid, execution_ids: Vec, wait: std::time::Duration) -> anyhow::Result> { + impl weft_task_store::TaskStoreClient for Cancels { + async fn cancels_asked(&self, _p: uuid::Uuid, execution_ids: Vec) -> anyhow::Result> { self.asked.lock().unwrap().push(execution_ids.clone()); - if !self.given.swap(true, std::sync::atomic::Ordering::SeqCst) { - return Ok(vec![weft_task_store::tasks::CancelAsked { execution_id: execution_ids[0].clone(), cause: weft_core::exec::CancelCause::User }]); - } - tokio::time::sleep(wait).await; - Ok(Vec::new()) + let waiting = self.waiting.lock().unwrap(); + let asked: Vec = waiting.iter().filter(|id| execution_ids.contains(id)).cloned().collect(); + Ok(asked.into_iter().map(|execution_id| weft_task_store::tasks::CancelAsked { execution_id, cause: weft_core::exec::CancelCause::User }).collect()) } async fn enqueue_dedup(&self, _s: weft_task_store::tasks::NewTask) -> anyhow::Result { unreachable!() @@ -1474,9 +1614,10 @@ mod tests { } } - fn worker_over(tasks: Arc) -> Worker { + fn worker_over(tasks: Arc, line: Arc) -> Worker { let mut clients = clients(Arc::new(MemJournal::default())); clients.tasks = tasks; + clients.line = line; struct NoCatalog; impl NodeCatalog for NoCatalog { fn lookup(&self, _t: &str) -> Option<&'static dyn weft_core::Node> { @@ -1501,36 +1642,102 @@ mod tests { ) } - // A cancel the broker answers reaches the flag of the execution being - // driven, and the wait asks for exactly the executions driven: it starts - // only once an execution is registered, whichever of the two comes first. + fn cancel_heard(execution_id: ExecutionId) -> weft_task_store::pg_signal::Heard { + weft_task_store::pg_signal::Heard::Signal { + channel: weft_task_store::tasks::CANCEL_CHANNEL, + payload: weft_task_store::tasks::cancel_payload(uuid::Uuid::from_u128(1), &execution_id.to_string()).into(), + } + } + + async fn cancelled(flag: &CancellationFlag) { + tokio::time::timeout(std::time::Duration::from_secs(10), async { + while !flag.is_cancelled() { + tokio::task::yield_now().await; + } + }) + .await + .expect("the cancel reached the flag"); + } + + // A cancel the line announces reaches the flag of the execution driven, + // whether it is heard after the drive registered or just before; the + // line is held open exactly while something is driven; a line that + // opens again asks for the cancels of every execution driven; and a + // drive that registers after such an ask asks for its own. weft_core::stress_test! { name: a_heard_cancel_fires_the_flag_of_the_execution_id_driven, runs: 20, worker_threads: 4, async fn body() { - let tasks = Arc::new(OneCancel { asked: std::sync::Mutex::new(Vec::new()), given: false.into() }); - let worker = worker_over(tasks.clone()); - spawn_cancel_wait(worker.clone()); - let execution_id = ExecutionId::new_v4(); + let tasks = Arc::new(Cancels::default()); + let line = crate::context::TestLine::new(); + let worker = worker_over(tasks.clone(), line.clone()); + spawn_cancel_watch(worker.clone()); + + let after = ExecutionId::new_v4(); + let flag = CancellationFlag::new_arc(); + register_cancel_flag(&worker, after, flag.clone(), 0).await; + tokio::time::timeout(std::time::Duration::from_secs(10), async { + while line.held.load(std::sync::atomic::Ordering::SeqCst) == 0 { + tokio::task::yield_now().await; + } + }) + .await + .expect("driving, the line is held open"); + tasks.waiting.lock().unwrap().push(after.to_string()); + line.pushed.send(cancel_heard(after)).unwrap(); + cancelled(&flag).await; + assert_eq!(tasks.asked.lock().unwrap()[0], vec![after.to_string()], "only the execution heard is asked about"); + + let before = ExecutionId::new_v4(); + tasks.waiting.lock().unwrap().push(before.to_string()); + line.pushed.send(cancel_heard(before)).unwrap(); + let flag = CancellationFlag::new_arc(); + // Heard before it registered: the watch may not have read it + // yet, so the registration finds it or the watch does. + tokio::task::yield_now().await; + register_cancel_flag(&worker, before, flag.clone(), 0).await; + cancelled(&flag).await; + + let missed = ExecutionId::new_v4(); let flag = CancellationFlag::new_arc(); - worker.cancel_registry.lock().await.insert(execution_id, flag.clone()); + register_cancel_flag(&worker, missed, flag.clone(), 0).await; + tasks.waiting.lock().unwrap().push(missed.to_string()); + line.pushed.send(weft_task_store::pg_signal::Heard::Recheck).unwrap(); + cancelled(&flag).await; + + // Claimed before a recheck, registered after it: the recheck + // could not include it, so the registration asks. + let late = ExecutionId::new_v4(); + let claimed_at = worker.rechecks.load(std::sync::atomic::Ordering::SeqCst); + tasks.waiting.lock().unwrap().push(late.to_string()); + line.pushed.send(weft_task_store::pg_signal::Heard::Recheck).unwrap(); + tokio::time::timeout(std::time::Duration::from_secs(10), async { + while worker.rechecks.load(std::sync::atomic::Ordering::SeqCst) == claimed_at { + tokio::task::yield_now().await; + } + }) + .await + .expect("the watch heard the recheck"); + let flag = CancellationFlag::new_arc(); + register_cancel_flag(&worker, late, flag.clone(), claimed_at).await; + cancelled(&flag).await; + + worker.cancel_registry.lock().await.clear(); worker.driving_changed.notify_one(); tokio::time::timeout(std::time::Duration::from_secs(10), async { - while !flag.is_cancelled() { + while line.held.load(std::sync::atomic::Ordering::SeqCst) != 0 { tokio::task::yield_now().await; } }) .await - .expect("the cancel reached the flag"); - assert_eq!(tasks.asked.lock().unwrap()[0], vec![execution_id.to_string()]); + .expect("with nothing driven, the line is let go of"); } } #[tokio::test] async fn a_call_for_an_execution_with_nothing_to_claim_says_so() { - let tasks = Arc::new(OneCancel { asked: std::sync::Mutex::new(Vec::new()), given: true.into() }); - let worker = worker_over(tasks); + let worker = worker_over(Arc::new(Cancels::default()), crate::context::TestLine::new()); assert_eq!(worker.run_execution_id(ExecutionId::new_v4()).await.unwrap(), RunAnswer::NothingToRun); } } diff --git a/crates/weft-infra-supervisor/src/broker_ops/fake.rs b/crates/weft-infra-supervisor/src/broker_ops/fake.rs index 5ca769bd..665ab2cf 100644 --- a/crates/weft-infra-supervisor/src/broker_ops/fake.rs +++ b/crates/weft-infra-supervisor/src/broker_ops/fake.rs @@ -280,18 +280,29 @@ impl Inner { } /// Whether a supervisor may own `project_id`, as the broker's - /// `ownable_project` answers it: infra to manage, copies on the host - /// (`held`), or an uncompleted command waiting on it. + /// `ownable_project` answers it: infra to manage, infra copies, copies + /// on the host (`held`), or an uncompleted command waiting on it. fn ownable(&self, project_id: uuid::Uuid, held: &[uuid::Uuid]) -> bool { - !self.infraless.contains(&project_id) || held.contains(&project_id) || self.has_work(project_id) + !self.infraless.contains(&project_id) || self.has_infra_nodes(project_id) || held.contains(&project_id) || self.has_work(project_id) } - /// Whether `project_id` gives its owner something to look at now, as - /// the broker's `supervisor_work` answers it: infra nodes, or an - /// uncompleted command. - fn has_work(&self, project_id: uuid::Uuid) -> bool { + /// Whether `project_id` has infra copies, as the broker's + /// `has_infra_nodes` answers it. + fn has_infra_nodes(&self, project_id: uuid::Uuid) -> bool { self.infra_nodes.keys().any(|k| k.0 == project_id) - || self.commands.iter().any(|c| c.project_id == project_id && !self.completed(c.id)) + } + + /// Whether `project_id` has infra copies expected to run, as the + /// broker's `runs_infra` answers it. + fn runs_infra(&self, project_id: uuid::Uuid) -> bool { + self.infra_nodes.iter().any(|(k, n)| k.0 == project_id && n.status.expects_running_units()) + } + + /// Whether `project_id` gives its owner something to do now that + /// nothing will announce, as the broker's `supervisor_work` answers it: + /// an uncompleted command. + fn has_work(&self, project_id: uuid::Uuid) -> bool { + self.commands.iter().any(|c| c.project_id == project_id && !self.completed(c.id)) } /// Whether an uncompleted command waits on a project nobody holds. @@ -743,10 +754,11 @@ impl BrokerSupervisorOps for FakeBroker { let others_lapse_in_secs = inner .projects .keys() - .filter(|id| inner.has_work(**id) || held_projects.contains(id)) + .filter(|id| inner.has_work(**id) || inner.runs_infra(**id) || held_projects.contains(id)) .filter_map(|id| match inner.owners.get(id) { Some(Owner::Other) => Some(weft_broker_client::lifecycle_command::infra_owner_lease_secs()), Some(Owner::Nobody) => Some(0), + // The fake's default owner is this supervisor (`owns`). Some(Owner::Us) | None => None, }) .min(); diff --git a/crates/weft-infra-supervisor/src/health.rs b/crates/weft-infra-supervisor/src/health.rs index 7f5bbfab..fe94f22c 100644 --- a/crates/weft-infra-supervisor/src/health.rs +++ b/crates/weft-infra-supervisor/src/health.rs @@ -11,13 +11,18 @@ //! flight, subsequent matches queue (next look re-checks once //! the current one settles). //! -//! When it looks: every `health_interval`, every owned project (the -//! flaky and recovery windows are about TIME passing with nothing -//! changing, so a look on a clock is what they need). What this -//! supervisor owns is followed as the ownership loop changes it: a lost -//! project's health state is dropped the moment the loss is known (a look -//! already running finishes; its status write is fenced by ownership), -//! and a claim runs a tick right away. +//! When it looks: in one process that lives as long as the install (a +//! local one), every `health_interval`, every owned project. A supervisor +//! that scales to zero looks when told to (a machine running a project's +//! infra says how its unit stands changed, or the cloud says a machine +//! went away), and then again at the health interval only while something +//! it saw is not settled: a unit inside its flaky or recovery window, which +//! only TIME passing closes, or an action under way or backing off. A +//! unit that runs fine, or one broken and already declared flaky, needs no +//! look until it changes. What this supervisor owns is followed as the ownership loop +//! changes it: a lost project's health state is dropped the moment the +//! loss is known (a look already running finishes; its status write is +//! fenced by ownership), and a claim runs a tick right away. use std::collections::{HashMap, HashSet}; use std::time::{Duration, Instant}; @@ -163,16 +168,23 @@ pub async fn on_ownership_change(state: &SupervisorState, change: &crate::owners /// One tick of the health loop: evaluate every owned project. Exposed /// (rather than only running inside `run_loop`) so integration tests can -/// step the loop one tick at a time. -pub async fn tick(state: &SupervisorState) -> Result<()> { +/// step the loop one tick at a time. Answers whether anything it saw is +/// not settled yet (see the module doc): a project whose look failed +/// counts as unsettled, so it is looked at again. +pub async fn tick(state: &SupervisorState) -> Result { let projects = state.broker.owned_projects(&state.replica).await?; + let mut unsettled = false; for project in &projects { - if let Err(e) = tick_project(state, project).await { - tracing::warn!( - project_id = %project.project_id, - error = %e, - "health tick (project) failed" - ); + match tick_project(state, project).await { + Ok(project_unsettled) => unsettled |= project_unsettled, + Err(e) => { + unsettled = true; + tracing::warn!( + project_id = %project.project_id, + error = %e, + "health tick (project) failed" + ); + } } } @@ -184,13 +196,18 @@ pub async fn tick(state: &SupervisorState) -> Result<()> { projects.iter().map(|p| p.project_id).collect(); state.health.lock().await.keep_only(|project| live.contains(&project)); } - Ok(()) + Ok(unsettled) } +/// Look at one project's health; answers whether anything there is not +/// settled yet: a unit whose latch time passing alone can change +/// (`NodeHealthState::waits_on_time`), or a protocol's action under way or +/// waiting out its backoff. A unit broken and already declared flaky is +/// settled: nothing changes until the unit does, which its machine says. async fn tick_project( state: &SupervisorState, project: &weft_broker_client::protocol::SupervisorProject, -) -> Result<()> { +) -> Result { // Stand down, copy by copy, while a user infra action runs on it: // an uncompleted supervisor command (apply / stop / terminate) that // reaches a copy means the lifecycle handler owns that copy's status @@ -255,6 +272,8 @@ async fn tick_project( String, )> = Vec::new(); let mut units: Vec = Vec::new(); + // Whether time passing alone can change what a unit's latch says. + let mut units_unsettled = false; { let mut registry = state.health.lock().await; for n in &nodes { @@ -290,11 +309,13 @@ async fn tick_project( ) else { continue; }; + let ready = observed.is_some_and(|o| o.ready); + units_unsettled |= decision.next.waits_on_time(ready); units.push(UnitView { node_id: n.node_id.clone(), instance: n.instance.clone(), unit: unit.clone(), - ready: observed.is_some_and(|o| o.ready), + ready, flaky: decision.next.declared_flaky, }); registry.state.insert(key, decision.next.clone()); @@ -431,7 +452,7 @@ async fn tick_project( error = %e, "health_protocols_json malformed; skipping protocol eval until fixed" ); - return Ok(()); + return Ok(units_unsettled); } }, None => protocol::default_protocols(), @@ -466,7 +487,8 @@ async fn tick_project( }; let Some(matched) = evaluate_protocols(&protocols, &fired_snap, in_flight_snap, &inputs) else { - return Ok(()); + let backing_off = state.health.lock().await.backoff.keys().any(|(proj, _)| *proj == project.project_id); + return Ok(units_unsettled || in_flight_snap || backing_off); }; let matched_name = matched.protocol.name.clone(); let matched_proto = matched.protocol.clone(); @@ -483,7 +505,7 @@ async fn tick_project( let mut registry = state.health.lock().await; if let Some(b) = registry.backoff.get(&backoff_key) { if state.clock.now() < b.next_retry_at { - return Ok(()); + return Ok(true); } } registry.in_flight.insert(project.project_id); @@ -564,7 +586,8 @@ async fn tick_project( "HealthProtocol action failed; un-latched from fired set to retry next tick" ); } - Ok(()) + // An action just ran: what it changed is looked at next. + Ok(true) } /// What `plan_action` decided. The I/O wrapper turns each variant diff --git a/crates/weft-infra-supervisor/src/health_engine.rs b/crates/weft-infra-supervisor/src/health_engine.rs index f4df9535..2480e184 100644 --- a/crates/weft-infra-supervisor/src/health_engine.rs +++ b/crates/weft-infra-supervisor/src/health_engine.rs @@ -100,6 +100,20 @@ impl NodeHealthState { pub fn has_seen(&self) -> bool { self.last_ready_at.is_some() || self.last_not_ready_at.is_some() } + + /// Whether time passing alone can change what this latch says, the unit + /// last seen `ready` or not: a unit seen ready before and not ready now + /// turns flaky once its window has passed, and a flaky one ready again + /// recovers once its window has passed. Anything else stays as it is + /// until the unit itself changes, which its machine says + /// (`weft_runtime::unit_agent`). + pub fn waits_on_time(&self, ready: bool) -> bool { + if ready { + self.declared_flaky + } else { + !self.declared_flaky && self.last_ready_at.is_some() + } + } } /// Pure: one unit's decision from what this look saw of it (`None`: the @@ -413,6 +427,22 @@ mod tests { NodeObservation { ready } } + /// Only a latch inside one of its windows waits on time: a unit never + /// seen ready (its apply owns it), one running fine, and one declared + /// flaky and still down stay as they are until the unit changes. + #[test] + fn only_a_latch_inside_a_window_waits_on_time() { + let now = t0(); + let fresh = NodeHealthState::default(); + assert!(!fresh.waits_on_time(false), "never seen ready: the apply owns it"); + let was_ready = NodeHealthState { last_ready_at: Some(now), ..NodeHealthState::default() }; + assert!(!was_ready.waits_on_time(true), "running fine"); + assert!(was_ready.waits_on_time(false), "inside the flaky window"); + let flaky = NodeHealthState { declared_flaky: true, last_not_ready_at: Some(now), ..was_ready.clone() }; + assert!(!flaky.waits_on_time(false), "flaky and still down"); + assert!(flaky.waits_on_time(true), "inside the recovery window"); + } + // ---------- latch seeding from the row ---------- /// A latch seeded from a `Flaky` row keeps saying Flaky while the diff --git a/crates/weft-infra-supervisor/src/lib.rs b/crates/weft-infra-supervisor/src/lib.rs index fc86c30b..2f48e49c 100644 --- a/crates/weft-infra-supervisor/src/lib.rs +++ b/crates/weft-infra-supervisor/src/lib.rs @@ -139,13 +139,16 @@ pub async fn run_loops(state: SupervisorState) -> anyhow::Result<()> { /// all along as they are there: a start or a stop takes the cloud a minute /// or two, and a lease left to lapse meanwhile hands the project to a /// sibling, which runs the command again from the start. Answers when to -/// look again: at the health interval while a project it owns has -/// something to look at (infra nodes, a command), or while the host -/// listing or the gone-copy sweep left something; otherwise when the -/// soonest lease a sibling holds over such a project, or over one the -/// host holds copies of, lapses (the sibling may be gone, and only a -/// lapsed lease is taken over), and the moment after. `None` when there -/// is nothing to look at anywhere: a command being issued wakes it. +/// look again: at the health interval while a project it owns has a +/// command waiting, while the host listing or the gone-copy sweep left +/// something, or while the health it saw is not settled (`health::tick`); +/// otherwise when the soonest lease a sibling holds over a project with a +/// command waiting, or over one the host holds copies of, lapses (the +/// sibling may be gone, and only a lapsed lease is taken over), and the +/// moment after. `None` when there is nothing to look at anywhere: infra +/// that runs fine needs no look, and a command being issued, a machine +/// saying how its unit stands changed, or the cloud saying a machine went +/// away wakes it. pub async fn tick(state: &SupervisorState) -> anyhow::Result> { // A wake that lands during another pass waits for it and then runs // its own, which finds whatever that one left waiting. @@ -160,14 +163,14 @@ pub async fn tick(state: &SupervisorState) -> anyhow::Result> { anyhow::bail!("the supervisor's ownership renewal ended mid-pass"); } } - health::tick(state).await?; - Ok(next_look(&synced, state.health_interval)) + let unsettled = health::tick(state).await?; + Ok(next_look(&synced, unsettled, state.health_interval)) } /// When a supervisor that scales to zero looks again after a pass that -/// found `synced`. -fn next_look(synced: &ownership::Synced, health_interval: Duration) -> Option { - if synced.owns_work { +/// found `synced`, and health `unsettled` or not. +fn next_look(synced: &ownership::Synced, unsettled: bool, health_interval: Duration) -> Option { + if synced.owns_work || unsettled { return Some(health_interval); } synced.others_lapse_in.map(|lapse| lapse + Duration::from_secs(1)) diff --git a/crates/weft-infra-supervisor/src/testing/rig.rs b/crates/weft-infra-supervisor/src/testing/rig.rs index b29df211..72849323 100644 --- a/crates/weft-infra-supervisor/src/testing/rig.rs +++ b/crates/weft-infra-supervisor/src/testing/rig.rs @@ -58,8 +58,8 @@ impl SupervisorTestRig { Ok(ownership::tick(&self.state, &mut *self.owned.lock().await).await?.change) } - /// Step the health loop once. - pub async fn tick_health(&self) -> Result<()> { + /// Step the health loop once; whether anything it saw is unsettled. + pub async fn tick_health(&self) -> Result { health::tick(&self.state).await } diff --git a/crates/weft-infra-supervisor/tests/health_loop.rs b/crates/weft-infra-supervisor/tests/health_loop.rs index 98aeeb5d..f4da5b3c 100644 --- a/crates/weft-infra-supervisor/tests/health_loop.rs +++ b/crates/weft-infra-supervisor/tests/health_loop.rs @@ -77,6 +77,45 @@ fn restarts(rig: &SupervisorTestRig) -> usize { rig.host.calls().iter().filter(|c| matches!(c, HostCall::Restart { .. })).count() } +// ---------- when a supervisor that scales to zero looks again ---------- + +/// A supervisor that scales to zero looks on a clock only while time +/// passing can change what it decides: a unit inside its flaky window (not +/// ready after having been) or its recovery window (ready again after +/// being flaky). A unit that runs fine is settled, and so is one broken and +/// already declared flaky, once what that set off has run: neither changes +/// until the unit does, which its machine says. +#[tokio::test] +async fn health_is_unsettled_only_while_time_can_change_a_decision() { + let rig = rig(); + rig.broker.add_infra_node(PROJECT, NODE, "inst1", Status::Running); + observe(&rig, PROJECT, &[(NODE, "inst1", "bridge", true)]); + assert!(!rig.tick_health().await.unwrap(), "every unit ready: nothing to look at again"); + + observe(&rig, PROJECT, &[(NODE, "inst1", "bridge", false)]); + assert!(rig.tick_health().await.unwrap(), "not ready inside its flaky window"); + observe(&rig, PROJECT, &[]); + assert!(rig.tick_health().await.unwrap(), "no longer reported reads as not ready, still inside the window"); + + rig.advance(Duration::from_secs(31)); + rig.tick_health().await.unwrap(); + let mut settled = false; + for _ in 0..5 { + if !rig.tick_health().await.unwrap() { + settled = true; + break; + } + rig.advance(Duration::from_secs(1)); + } + assert!(settled, "declared flaky, once what that set off has run: nothing to look at again"); + + observe(&rig, PROJECT, &[(NODE, "inst1", "bridge", true)]); + assert!(rig.tick_health().await.unwrap(), "ready again inside its recovery window"); + rig.advance(Duration::from_secs(31)); + rig.tick_health().await.unwrap(); + assert!(!rig.tick_health().await.unwrap(), "recovered: settled"); +} + // ---------- false-flaky-on-fresh-provision ---------- #[tokio::test] diff --git a/crates/weft-infra-supervisor/tests/ownership_loop.rs b/crates/weft-infra-supervisor/tests/ownership_loop.rs index 5b2253a7..f2355fdb 100644 --- a/crates/weft-infra-supervisor/tests/ownership_loop.rs +++ b/crates/weft-infra-supervisor/tests/ownership_loop.rs @@ -39,17 +39,22 @@ async fn ownership_tick_syncs_under_this_supervisors_identity() { assert!(synced, "ownership tick must sync_ownership(test-supervisor)"); } -/// A pass answers when it next has something to look at: soon while a -/// project it owns has infra running, when a sibling's lease over one -/// lapses while it owns none of them, and never while nothing anywhere -/// has anything to do (a declared infra alone included). +/// A pass answers when it next has something to look at: never over infra +/// of its own that runs fine (its machine says when that changes), when a +/// sibling's lease over a project with infra lapses (the word that one +/// changed may have reached this replica, not the sibling), and never while +/// nothing anywhere has anything to do (a declared infra alone included). #[tokio::test] async fn a_pass_looks_again_only_while_there_is_something_to_look_at() { use weft_broker_client::protocol::InfraNodeStatus; let running = SupervisorTestRig::with_tenant("alice"); running.broker.add_project(P1); running.broker.add_infra_node(P1, "db", "i-1", InfraNodeStatus::Running); - assert_eq!(weft_infra_supervisor::tick(&running.state).await.unwrap(), Some(running.state.health_interval)); + assert_eq!( + weft_infra_supervisor::tick(&running.state).await.unwrap(), + None, + "infra of its own that runs needs no look on a clock: its machine says when it changes" + ); let sibling = SupervisorTestRig::with_tenant("alice"); sibling.broker.add_project(P1); diff --git a/crates/weft-journal/src/lib.rs b/crates/weft-journal/src/lib.rs index cfd2cd5e..6ce2a514 100644 --- a/crates/weft-journal/src/lib.rs +++ b/crates/weft-journal/src/lib.rs @@ -19,9 +19,12 @@ pub mod write; pub use events::{ExecEvent, Seed, EXECUTION_TERMINAL_KINDS_SQL, RUN_PARKED_SQL}; -/// The channel every journal row notifies on when it commits, with its -/// execution as the payload, from the `exec_event_notify_on_insert` trigger -/// in the dispatcher's journal schema group. +/// The channel a write of journal rows is announced on, once per execution +/// it wrote to, with the execution as the payload: from the +/// `exec_event_notify_on_insert` trigger in the dispatcher's journal schema +/// group, through the announcement outbox (`weft_task_store::announce`), +/// sent once the write commits and its writer pokes the flusher. +// SYNC: EXEC_EVENT_CHANNEL <-> 'weft_exec_event' in exec_event_notify (crates/weft-dispatcher/src/journal/postgres.rs) pub const EXEC_EVENT_CHANNEL: &str = "weft_exec_event"; pub use fold::{fold_to_snapshot, FiringView, Fold, FoldEffects}; pub use seed::{fold_seeded, seed_chain, LiveFold, SeedChain}; diff --git a/crates/weft-journal/src/traits.rs b/crates/weft-journal/src/traits.rs index c28a984e..c75838b0 100644 --- a/crates/weft-journal/src/traits.rs +++ b/crates/weft-journal/src/traits.rs @@ -214,6 +214,7 @@ impl JournalClient for PostgresJournalClient { let mut tx = self.pool.begin().await?; crate::unrecorded::forget_in(&mut tx, execution_id).await?; tx.commit().await?; + weft_task_store::announce::committed(&self.pool); Ok(()) } diff --git a/crates/weft-journal/src/unrecorded.rs b/crates/weft-journal/src/unrecorded.rs index 070f263d..bb7744ff 100644 --- a/crates/weft-journal/src/unrecorded.rs +++ b/crates/weft-journal/src/unrecorded.rs @@ -350,6 +350,7 @@ pub async fn record_retroactively( .await?; announce_ended(&mut tx, UnrecordedEnded { execution_id, project_id, fired_by, instance }).await?; tx.commit().await?; + weft_task_store::announce::committed(pool); Ok(()) } @@ -363,9 +364,11 @@ struct HeldCost { } /// Announce an unrecorded run's ending on [`UNRECORDED_ENDED_CHANNEL`], -/// delivered when the caller's transaction commits. +/// through the announcement outbox (`weft_task_store::announce`): it +/// commits with the caller's transaction, whose committer pokes the +/// flusher. async fn announce_ended(tx: &mut sqlx::PgConnection, ended: UnrecordedEnded) -> anyhow::Result<()> { - sqlx::query("SELECT pg_notify($1, $2)") + sqlx::query("SELECT weft_announce($1, $2)") .bind(UNRECORDED_ENDED_CHANNEL) .bind(serde_json::to_string(&ended)?) .execute(&mut *tx) @@ -377,7 +380,9 @@ async fn announce_ended(tx: &mut sqlx::PgConnection, ended: UnrecordedEnded) -> /// when no row of it is in the journal, or stamp it ended when its costs /// keep it (they are addressed by execution; it never lists, only recorded /// runs do), and announce the ending on [`UNRECORDED_ENDED_CHANNEL`], -/// delivered when the transaction commits. True when the row went. +/// sent once the transaction commits and its caller pokes the +/// announcement flusher (`weft_task_store::announce::committed`). True +/// when the row went. pub async fn forget_in(tx: &mut sqlx::PgConnection, execution_id: ExecutionId) -> anyhow::Result { crate::write::lock_execution_ids(&mut *tx, &[execution_id]).await?; let owner: Option<(uuid::Uuid, Option, Option)> = sqlx::query_as( diff --git a/crates/weft-journal/src/write.rs b/crates/weft-journal/src/write.rs index 0f30d7f4..42882dd5 100644 --- a/crates/weft-journal/src/write.rs +++ b/crates/weft-journal/src/write.rs @@ -49,8 +49,9 @@ fn unix_now() -> Result { /// Insert `events`, all of execution `events[0]`'s, in order, in ONE /// statement: one round trip whatever their number. The table's own -/// trigger announces the rows on [`crate::EXEC_EVENT_CHANNEL`] when the -/// write commits, which is what wakes the dispatcher's event bridge and a +/// trigger announces the rows on [`crate::EXEC_EVENT_CHANNEL`] through the +/// announcement outbox (`weft_task_store::announce`), sent once the write +/// commits, which is what wakes the dispatcher's event bridge and a /// worker waiting on its run's journal. /// /// `replica` is the writing worker's replica id, stamped on every row; @@ -67,11 +68,15 @@ pub async fn record_events( replica: Option<&str>, owner: Option<&str>, ) -> Result { - insert(pool, events, replica, owner, None).await + let written = insert(pool, events, replica, owner, None).await?; + weft_task_store::announce::committed(pool); + Ok(written) } /// One event, inside the caller's own transaction (the dispatcher pairs -/// `ExecutionStarted` with its `execution` seed atomically). `dedup_key` +/// `ExecutionStarted` with its `execution` seed atomically). The caller +/// pokes the announcement flusher once it commits +/// (`weft_task_store::announce::committed`). `dedup_key` /// makes it idempotent: a retry of the same write collapses on the /// partial UNIQUE index, so a dispatcher task that re-executes after a /// crash never double-fires `ExecutionStarted` / `NodeKicked`. diff --git a/crates/weft-listener/tests/provider_events.rs b/crates/weft-listener/tests/provider_events.rs index 11fe1d65..58880d14 100644 --- a/crates/weft-listener/tests/provider_events.rs +++ b/crates/weft-listener/tests/provider_events.rs @@ -26,11 +26,10 @@ struct FakeTasks { #[async_trait::async_trait] impl weft_task_store::TaskStoreClient for FakeTasks { - async fn wait_cancels( + async fn cancels_asked( &self, _project_id: uuid::Uuid, _execution_ids: Vec, - _wait: std::time::Duration, ) -> anyhow::Result> { Ok(Vec::new()) } diff --git a/crates/weft-listener/tests/stream_listen.rs b/crates/weft-listener/tests/stream_listen.rs index d611e509..d17e7fd5 100644 --- a/crates/weft-listener/tests/stream_listen.rs +++ b/crates/weft-listener/tests/stream_listen.rs @@ -26,11 +26,10 @@ struct FakeTasks { #[async_trait::async_trait] impl weft_task_store::TaskStoreClient for FakeTasks { - async fn wait_cancels( + async fn cancels_asked( &self, _project_id: uuid::Uuid, _execution_ids: Vec, - _wait: std::time::Duration, ) -> anyhow::Result> { Ok(Vec::new()) } diff --git a/crates/weft-listener/tests/wakes.rs b/crates/weft-listener/tests/wakes.rs index 917a3a14..1358c708 100644 --- a/crates/weft-listener/tests/wakes.rs +++ b/crates/weft-listener/tests/wakes.rs @@ -27,11 +27,10 @@ struct FakeTasks { #[async_trait::async_trait] impl weft_task_store::TaskStoreClient for FakeTasks { - async fn wait_cancels( + async fn cancels_asked( &self, _project_id: uuid::Uuid, _execution_ids: Vec, - _wait: std::time::Duration, ) -> anyhow::Result> { Ok(Vec::new()) } diff --git a/crates/weft-platform-gcp/src/infra_host.rs b/crates/weft-platform-gcp/src/infra_host.rs index 762cfa6a..e6962fd9 100644 --- a/crates/weft-platform-gcp/src/infra_host.rs +++ b/crates/weft-platform-gcp/src/infra_host.rs @@ -39,6 +39,9 @@ pub const MD_GPU: &str = "weft-gpu"; pub const MD_RUNTIME_IMAGE: &str = "weft-runtime-image"; pub const MD_CORE_ACCOUNT: &str = "weft-core-account"; pub const MD_GCP_PROJECT: &str = "weft-gcp-project"; +/// Where the machine's agent reaches the broker, to ask for a look at +/// its project's health when how its unit stands changes. +pub const MD_BROKER_URL: &str = "weft-broker-url"; /// The boot image of every infra machine. const BOOT_IMAGE: &str = "projects/cos-cloud/global/images/family/cos-stable"; @@ -105,11 +108,13 @@ pub struct ComputeInfraHost { google: Google, gcp: GcpPlatform, install: weft_core::infra::Install, + /// Where a machine's agent reaches the broker ([`MD_BROKER_URL`]). + broker_url: String, } impl ComputeInfraHost { - pub fn new(google: Google, gcp: GcpPlatform, install: weft_core::infra::Install) -> Self { - Self { google, gcp, install } + pub fn new(google: Google, gcp: GcpPlatform, install: weft_core::infra::Install, broker_url: String) -> Self { + Self { google, gcp, install, broker_url } } fn zone_base(&self) -> String { @@ -213,6 +218,7 @@ impl ComputeInfraHost { { "key": MD_RUNTIME_IMAGE, "value": self.gcp.runtime_image }, { "key": MD_CORE_ACCOUNT, "value": self.gcp.core_service_account }, { "key": MD_GCP_PROJECT, "value": self.gcp.project }, + { "key": MD_BROKER_URL, "value": self.broker_url }, ]}, }); if let Some(kind) = &shape.accelerator { diff --git a/crates/weft-platform-gcp/src/runner.rs b/crates/weft-platform-gcp/src/runner.rs index e3911eba..01e2ca42 100644 --- a/crates/weft-platform-gcp/src/runner.rs +++ b/crates/weft-platform-gcp/src/runner.rs @@ -714,7 +714,7 @@ mod tests { let t = &body["template"]; assert_eq!(t["scaling"]["minInstanceCount"], 0); assert_eq!(t["scaling"]["maxInstanceCount"], 10); - assert_eq!(t["maxInstanceRequestConcurrency"], 20); + assert_eq!(t["maxInstanceRequestConcurrency"], 80); assert_eq!(t["timeout"], "3600s"); assert_eq!(t["serviceAccount"], "wp-x@acme.iam.gserviceaccount.com"); assert_eq!(t["vpcAccess"]["networkInterfaces"][0]["subnetwork"], "weft-sub", "workers leave from the install's main subnet"); diff --git a/crates/weft-platform-traits/src/config.rs b/crates/weft-platform-traits/src/config.rs index 031ed7d0..8a866844 100644 --- a/crates/weft-platform-traits/src/config.rs +++ b/crates/weft-platform-traits/src/config.rs @@ -14,7 +14,7 @@ use serde::{Deserialize, Serialize}; use weft_core::infra::Install; use crate::roles::{CoreRole, Placement, RoleAddresses, RolePlacement, Vantage, INTERNAL_DOOR}; -use crate::runner::WorkerSettings; +use crate::runner::{WorkerOverrides, WorkerSettings}; // SYNC: InstallConfig <-> deploy/terraform/gcp/serverless.tf (local.install_config) #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] @@ -38,9 +38,11 @@ pub struct InstallConfig { /// service serves the role at. Required for exactly those roles. #[serde(default, rename = "roleUrls", skip_serializing_if = "BTreeMap::is_empty")] pub role_urls: BTreeMap, - /// The worker settings every project starts from. - #[serde(default)] - pub workers: WorkerSettings, + /// The worker levers this install sets for every project; each one it + /// leaves out follows weft's own default ([`Self::worker_settings`]), + /// so a default weft changes reaches the install. + #[serde(default, skip_serializing_if = "WorkerOverrides::is_empty")] + pub workers: WorkerOverrides, /// How the holders share the held connections, where they run in a /// pool. #[serde(default)] @@ -79,9 +81,10 @@ pub struct LocalPlatform { #[serde(rename = "runtimeImage")] pub runtime_image: String, /// How long a worker with nothing to do stays up before it is - /// stopped. Raise it to keep workers warm longer between runs. - #[serde(default = "LocalPlatform::default_worker_idle_stop_seconds", rename = "workerIdleStopSeconds")] - pub worker_idle_stop_seconds: u64, + /// stopped, when this install sets it ([`Self::worker_idle_stop`]). + /// Raise it to keep workers warm longer between runs. + #[serde(default, rename = "workerIdleStopSeconds", skip_serializing_if = "Option::is_none")] + pub worker_idle_stop_seconds: Option, /// The ports the install's one process serves. pub listen: Listen, /// Where the internal port is reached from the machine itself @@ -92,9 +95,16 @@ pub struct LocalPlatform { pub internal_url: String, } +/// How long a local worker with nothing to do stays up, unless the install +/// says otherwise: long enough to serve the next call of someone trying +/// things out without starting again, short enough that a laptop is not +/// left running a container for nothing. +pub const DEFAULT_WORKER_IDLE_STOP: std::time::Duration = std::time::Duration::from_secs(30); + impl LocalPlatform { - fn default_worker_idle_stop_seconds() -> u64 { - 300 + /// How long a worker with nothing to do stays up. + pub fn worker_idle_stop(&self) -> std::time::Duration { + self.worker_idle_stop_seconds.map_or(DEFAULT_WORKER_IDLE_STOP, std::time::Duration::from_secs) } } @@ -366,9 +376,15 @@ impl InstallConfig { Ok(cfg) } + /// The worker settings every project starts from: weft's defaults with + /// the levers this install sets. + pub fn worker_settings(&self) -> WorkerSettings { + WorkerSettings::default().with(&self.workers) + } + /// Refuse a config no install could run, naming the setting. pub fn validate(&self) -> Result<(), String> { - self.workers.validate()?; + self.worker_settings().validate()?; if self.holders.signals_per_copy == 0 { return Err("holders.signalsPerCopy is 0: no holder could take a signal; set it to 1 or more".into()); } @@ -488,7 +504,7 @@ pub(crate) mod tests { data_dir: "/home/u/.local/share/weft".into(), container_internal_url: "http://host.docker.internal:14113".into(), runtime_image: "weft-runtime:dev".into(), - worker_idle_stop_seconds: 300, + worker_idle_stop_seconds: None, listen: Listen { public: "127.0.0.1:14111".parse().unwrap(), internal: "127.0.0.1:14113".parse().unwrap(), outside: None }, internal_url: "http://127.0.0.1:14113".into(), }), @@ -497,7 +513,7 @@ pub(crate) mod tests { internet_url: None, roles: RolePlacement::default(), role_urls: BTreeMap::new(), - workers: WorkerSettings::default(), + workers: WorkerOverrides::default(), holders: HolderSettings::default(), build: BuildConfig { compile_lanes: 4, builder_base_image: "b".into(), runtime_base_image: "r".into() }, edge: EdgeConfig { trusted_proxy_hops: ProxyHops { public: 0, outside: 0, domains: 0 }, invalid_tokens_per_minute: Some(30) }, diff --git a/crates/weft-platform-traits/src/infra_host.rs b/crates/weft-platform-traits/src/infra_host.rs index 7b596b8b..c8211a03 100644 --- a/crates/weft-platform-traits/src/infra_host.rs +++ b/crates/weft-platform-traits/src/infra_host.rs @@ -30,6 +30,21 @@ pub enum UnitRunState { Failed { why: String }, } +impl UnitRunState { + /// Which state it is, without what it says about it: what a watcher + /// compares between two looks, since the words of one state may + /// change from one look to the next. + pub fn kind(&self) -> &'static str { + match self { + UnitRunState::Starting { .. } => "starting", + UnitRunState::Ready => "ready", + UnitRunState::NotReady { .. } => "not_ready", + UnitRunState::Stopped => "stopped", + UnitRunState::Failed { .. } => "failed", + } + } +} + /// One unit the host runs for a project. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct UnitObservation { diff --git a/crates/weft-platform-traits/src/roles.rs b/crates/weft-platform-traits/src/roles.rs index 9edeb190..f5663704 100644 --- a/crates/weft-platform-traits/src/roles.rs +++ b/crates/weft-platform-traits/src/roles.rs @@ -191,7 +191,7 @@ impl RoleAddresses { } /// The path a role's tick answers at, under its internal address. -// SYNC: TICK_PATH <-> deploy/terraform/gcp/builds.tf (the push endpoint) +// SYNC: TICK_PATH <-> deploy/terraform/gcp/builds.tf (the push endpoint), deploy/terraform/gcp/machines.tf (the push endpoint) pub const TICK_PATH: &str = "/_weft/tick"; /// Where the internal routes are reached on a local install's public diff --git a/crates/weft-platform-traits/src/runner.rs b/crates/weft-platform-traits/src/runner.rs index 32782e15..cb8f4602 100644 --- a/crates/weft-platform-traits/src/runner.rs +++ b/crates/weft-platform-traits/src/runner.rs @@ -65,8 +65,13 @@ impl WorkerSettings { fn default_max_instances() -> u32 { 10 } + /// Cloud Run's own default for a service: a run mostly waits (on a + /// model, a database, a person), so one copy serves many, and when its + /// CPU fills the platform adds copies. A copy nearly out of memory turns + /// new work away itself (`weft_engine`'s memory guard) rather than + /// taking a run that would bring the others down. fn default_concurrency() -> u32 { - 20 + 80 } fn default_cpu() -> String { "1".into() @@ -121,6 +126,11 @@ pub struct WorkerOverrides { } impl WorkerOverrides { + /// Whether no lever is set. + pub fn is_empty(&self) -> bool { + *self == Self::default() + } + /// Every lever's name, as the flags, the API and a `weft.toml` spell it. pub const LEVERS: [&'static str; 7] = ["min_instances", "max_instances", "concurrency", "cpu", "memory", "startup_boost", "cpu_always_allocated"]; diff --git a/crates/weft-runtime/src/platform.rs b/crates/weft-runtime/src/platform.rs index 1cf38c92..37da6f41 100644 --- a/crates/weft-runtime/src/platform.rs +++ b/crates/weft-runtime/src/platform.rs @@ -57,7 +57,7 @@ pub async fn build(config: &InstallConfig, pool: Option<&sqlx::PgPool>) -> anyho // A local worker gets the ticket secret from the runner // that starts it; a GCP one reads it from Secret Manager. caller_token_secret: crate::secret("WEFT_CALLER_TOKEN_SECRET")?, - idle_stop: Duration::from_secs(local.worker_idle_stop_seconds), + idle_stop: local.worker_idle_stop(), scratch_dir: scratch.clone(), install: config.install.clone(), time_scale: weft_core::time_scale::factor(), @@ -127,7 +127,12 @@ pub async fn build(config: &InstallConfig, pool: Option<&sqlx::PgPool>) -> anyho Ok(Parts { runner: Arc::new(weft_platform_gcp::CloudRunRunner::new(google.clone(), gcp.clone(), config.role_addresses(Vantage::Private).broker, config.install.clone())), images: Arc::new(weft_platform_gcp::CloudBuildImages::new(google.clone(), gcp.clone())?), - host: Arc::new(weft_platform_gcp::ComputeInfraHost::new(google.clone(), gcp.clone(), config.install.clone())), + host: Arc::new(weft_platform_gcp::ComputeInfraHost::new( + google.clone(), + gcp.clone(), + config.install.clone(), + config.role_addresses(Vantage::Private).broker, + )), alarm: Arc::new(weft_platform_gcp::CloudTasksAlarm::new(google.clone(), gcp.clone(), config.role_addresses(Vantage::Public))), frontends: Arc::new(weft_platform_gcp::CloudRunFrontends::new(google.clone(), gcp.clone())), domains: Arc::new(weft_platform_gcp::LoadBalancerDomains::new(google.clone(), gcp.clone())), diff --git a/crates/weft-runtime/src/role_waker.rs b/crates/weft-runtime/src/role_waker.rs index 89b64817..5dd3f6b2 100644 --- a/crates/weft-runtime/src/role_waker.rs +++ b/crates/weft-runtime/src/role_waker.rs @@ -33,7 +33,7 @@ use std::sync::Arc; use std::time::Duration; use parking_lot::Mutex; -use weft_broker_client::lifecycle_command::ISSUED_WAKE; +use weft_broker_client::lifecycle_command::{ISSUED_WAKE, LOOK_WAKE}; use weft_platform_traits::config::InstallConfig; use weft_platform_traits::roles::TICK_PATH; use weft_platform_traits::{CoreRole, IdentityTokens, Placement, RoleAddresses}; @@ -42,8 +42,9 @@ use weft_task_store::pg_signal::{Heard, Subscription}; use crate::server::TICK_LOOP_PARAM; -/// The supervisor's one pass, woken by a lifecycle command being issued. -const SUPERVISOR_WAKES: &[(&str, &[WakeOn])] = &[("supervisor", &[ISSUED_WAKE])]; +/// The supervisor's one pass, woken by a lifecycle command being issued, or +/// by a machine running a project's infra saying how it stands changed. +const SUPERVISOR_WAKES: &[(&str, &[WakeOn])] = &[("supervisor", &[ISSUED_WAKE, LOOK_WAKE])]; /// What wakes each of `role`'s loops, by loop name. The broker's loops run /// on their own timers, so only its first ring and its alarms wake it; the diff --git a/crates/weft-runtime/src/unit_agent.rs b/crates/weft-runtime/src/unit_agent.rs index 48bfeb8e..481596bc 100644 --- a/crates/weft-runtime/src/unit_agent.rs +++ b/crates/weft-runtime/src/unit_agent.rs @@ -179,6 +179,77 @@ impl Host { } } +impl Host { + /// How the machine's unit stands, as weft is told it (`observe`). + async fn seen(&self) -> anyhow::Result> { + use weft_platform_traits::InfraHost as _; + let a = Self::assignment().await?; + let mut seen = self.local.observe(&a.node.node.tenant, a.node.node.project).await?; + // While an apply runs, that is the unit's state, whatever + // containers are there: one still running from before would read + // as a ready unit that is not the one asked for. After one failed, + // so is the failure, unless the unit runs whole as asked anyway + // (Docker brought its containers back). + let ours = |o: &weft_platform_traits::UnitObservation| o.copy_id == a.node.node.copy_id && o.unit == a.unit; + let wanted = a.node.unit(&a.unit).map(|u| u.hash.clone()).unwrap_or_default(); + let queued = self.queued.load(std::sync::atomic::Ordering::SeqCst) > 0; + let idle = self.applying.try_lock().ok().filter(|_| !queued).map(|last_failure| last_failure.clone()); + let state = match idle { + Some(Some(_)) if self.local.runs_whole(&a.node, &a.unit).await? => None, + Some(last_failure) => last_failure.map(|why| weft_platform_traits::UnitRunState::Failed { why }), + None if !seen.iter().any(ours) => Some(weft_platform_traits::UnitRunState::Starting { step: "its images are downloading".into() }), + None => Some(weft_platform_traits::UnitRunState::Starting { step: "its new version is starting".into() }), + }; + if let Some(state) = state { + seen.retain(|o| !ours(o)); + seen.push(weft_platform_traits::UnitObservation { copy_id: a.node.node.copy_id.clone(), unit: a.unit.clone(), hash: wanted, state }); + } + Ok(seen) + } + + /// Watch how the unit stands, here on its machine where looking costs + /// nothing, and ask for a look at the project's health each time it + /// changes (a container stopped, a check started failing or passing + /// again, an apply ended). The supervisor looks at infra that runs + /// fine only when told to, so an install whose infra runs fine + /// wakes nothing; the death of the whole machine is told by the cloud + /// itself. The first look after the agent starts always asks. + async fn watch(self, broker: std::sync::Arc) { + let mut told: Option> = None; + loop { + match self.seen().await.map(|seen| Self::stands(&seen)) { + Ok(stands) if told.as_ref() == Some(&stands) => {} + Ok(stands) => { + let asked = async { + let a = Self::assignment().await?; + broker.ask_for_a_look(a.node.node.project).await + }; + match asked.await { + Ok(()) => told = Some(stands), + Err(e) => tracing::warn!(target: "weft_runtime::unit_agent", error = %format!("{e:#}"), "could not ask for a look at the unit; asking again at the next look"), + } + } + Err(e) => tracing::warn!(target: "weft_runtime::unit_agent", error = %format!("{e:#}"), "could not see how the unit stands"), + } + tokio::time::sleep(WATCH_EVERY).await; + } + } + + /// What a look compares: each unit and the kind of state it is in + /// (never the words, which may change between looks of one state). + fn stands(seen: &[weft_platform_traits::UnitObservation]) -> Vec<(String, String, String)> { + let mut stands: Vec<(String, String, String)> = + seen.iter().map(|o| (o.copy_id.clone(), o.unit.clone(), o.state.kind().to_string())).collect(); + stands.sort(); + stands + } +} + +/// How often the agent looks at its unit (`Host::watch`). Looking is a +/// call to the machine's own Docker and the unit's checks, and costs +/// nothing off the machine. +const WATCH_EVERY: std::time::Duration = std::time::Duration::from_secs(10); + /// Run the machine's unit and answer weft about it. pub async fn host() -> anyhow::Result<()> { use weft_platform_gcp::infra_host::{MD_CORE_ACCOUNT, MD_GCP_PROJECT, MD_GPU, MD_RUNTIME_IMAGE}; @@ -224,6 +295,26 @@ pub async fn host() -> anyhow::Result<()> { // while its images download (`observe`). host.apply_in_background(); + // Asking for a look goes through the broker, as the project's own + // account: the one the machine runs as. A machine made before its + // agent was told where the broker is cannot ask; it says so, and its + // health is looked at when the supervisor looks for another reason. + match attribute(weft_platform_gcp::infra_host::MD_BROKER_URL).await { + Ok(broker_url) => { + let machine = metadata("name").await?; + let link = weft_broker_client::BrokerLink::new( + broker_url, + weft_broker_client::TokenSource::worker(host.tokens.clone(), format!("infra-{}", machine.trim())), + ); + tokio::spawn(host.clone().watch(weft_broker_client::BrokerInfraClient::new(link))); + } + Err(e) => tracing::warn!( + target: "weft_runtime::unit_agent", + error = %format!("{e:#}"), + "this machine was made by a weft that did not give it the broker's address, so it cannot ask for a look when its unit changes; every machine this weft makes has it" + ), + } + let err = |e: anyhow::Error| (axum::http::StatusCode::INTERNAL_SERVER_ERROR, format!("{e:#}")); let app = Router::new() // Answered at once: pulling a unit's images takes longer than any @@ -246,38 +337,7 @@ pub async fn host() -> anyhow::Result<()> { })) .route(HOST_OBSERVE, axum::routing::get({ let host = host.clone(); - move || async move { - use weft_platform_traits::InfraHost as _; - let a = Host::assignment().await.map_err(err)?; - let mut seen = host.local.observe(&a.node.node.tenant, a.node.node.project).await.map_err(err)?; - // While an apply runs, that is the unit's state, whatever - // containers are there: one still running from before would - // read as a ready unit that is not the one asked for. After - // one failed, so is the failure, unless the unit runs whole - // as asked anyway (Docker brought its containers back). - let ours = |o: &weft_platform_traits::UnitObservation| o.copy_id == a.node.node.copy_id && o.unit == a.unit; - let wanted = a.node.unit(&a.unit).map(|u| u.hash.clone()).unwrap_or_default(); - let queued = host.queued.load(std::sync::atomic::Ordering::SeqCst) > 0; - let idle = host.applying.try_lock().ok().filter(|_| !queued).map(|last_failure| last_failure.clone()); - let state = match idle { - Some(Some(_)) if host.local.runs_whole(&a.node, &a.unit).await.map_err(err)? => None, - Some(last_failure) => last_failure.map(|why| weft_platform_traits::UnitRunState::Failed { why }), - None if !seen.iter().any(ours) => { - Some(weft_platform_traits::UnitRunState::Starting { step: "its images are downloading".into() }) - } - None => Some(weft_platform_traits::UnitRunState::Starting { step: "its new version is starting".into() }), - }; - if let Some(state) = state { - seen.retain(|o| !ours(o)); - seen.push(weft_platform_traits::UnitObservation { - copy_id: a.node.node.copy_id.clone(), - unit: a.unit.clone(), - hash: wanted, - state, - }); - } - Ok::<_, (axum::http::StatusCode, String)>(Json(seen)) - } + move || async move { host.seen().await.map(Json).map_err(err) } })) .route(HOST_LOGS, axum::routing::get({ let host = host.clone(); diff --git a/crates/weft-task-store/migrations/announcement/origin.sql b/crates/weft-task-store/migrations/announcement/origin.sql new file mode 100644 index 00000000..bc12ee4e --- /dev/null +++ b/crates/weft-task-store/migrations/announcement/origin.sql @@ -0,0 +1,25 @@ +CREATE UNLOGGED TABLE IF NOT EXISTS weft_announcement ( + channel TEXT NOT NULL, + payload TEXT NOT NULL + ); +CREATE OR REPLACE FUNCTION weft_announce(p_channel TEXT, p_payload TEXT) RETURNS VOID AS $$ + INSERT INTO weft_announcement (channel, payload) VALUES (p_channel, p_payload) + $$ LANGUAGE sql; +CREATE OR REPLACE FUNCTION weft_flush_announcements(p_limit INTEGER) RETURNS INTEGER AS $$ + DECLARE + v_channels TEXT[]; + v_payloads TEXT[]; + BEGIN + WITH taken AS ( + DELETE FROM weft_announcement a + WHERE a.ctid IN (SELECT ctid FROM weft_announcement FOR UPDATE SKIP LOCKED LIMIT p_limit) + RETURNING channel, payload) + SELECT array_agg(channel), array_agg(payload) INTO v_channels, v_payloads FROM taken; + IF v_channels IS NULL THEN + RETURN 0; + END IF; + PERFORM pg_notify(d.channel, d.payload) + FROM (SELECT DISTINCT u.channel, u.payload FROM unnest(v_channels, v_payloads) AS u(channel, payload)) d; + RETURN array_length(v_channels, 1); + END; + $$ LANGUAGE plpgsql; diff --git a/crates/weft-task-store/migrations/exec_event/20261005T150244_exec_event_line_cancels_run_search_arrival.sql b/crates/weft-task-store/migrations/exec_event/20261005T150244_exec_event_line_cancels_run_search_arrival.sql new file mode 100644 index 00000000..7db6ea43 --- /dev/null +++ b/crates/weft-task-store/migrations/exec_event/20261005T150244_exec_event_line_cancels_run_search_arrival.sql @@ -0,0 +1,14 @@ +CREATE OR REPLACE FUNCTION exec_event_notify() + RETURNS trigger + LANGUAGE plpgsql +AS $function$ + BEGIN + PERFORM weft_announce('weft_exec_event', a.execution_id) + FROM (SELECT DISTINCT execution_id FROM added) a; + RETURN NULL; + END; + $function$ +; + +DROP TRIGGER exec_event_notify_on_insert ON exec_event; +CREATE TRIGGER exec_event_notify_on_insert AFTER INSERT ON exec_event REFERENCING NEW TABLE AS added FOR EACH STATEMENT EXECUTE FUNCTION exec_event_notify(); diff --git a/crates/weft-task-store/migrations/execution_search/origin.sql b/crates/weft-task-store/migrations/execution_search/origin.sql new file mode 100644 index 00000000..f84d6954 --- /dev/null +++ b/crates/weft-task-store/migrations/execution_search/origin.sql @@ -0,0 +1,40 @@ +CREATE TABLE IF NOT EXISTS execution_search ( + execution_id TEXT PRIMARY KEY, + words TSVECTOR NOT NULL + ); +CREATE INDEX IF NOT EXISTS idx_execution_search_words ON execution_search USING GIN (words); +CREATE OR REPLACE FUNCTION weft_run_search_text(p_execution_id TEXT, p_max INTEGER, p_value_max INTEGER) RETURNS TEXT AS $$ + DECLARE + -- Collected, then joined once: growing one string would + -- copy all of it at every value. + v_parts TEXT[] := '{}'; + v_length INTEGER := 0; + v_payload TEXT; + v_json JSONB; + v_value TEXT; + BEGIN + FOR v_payload IN SELECT payload_json FROM exec_event WHERE execution_id = p_execution_id ORDER BY id LOOP + IF strpos(v_payload, '\u0000') > 0 THEN + BEGIN + v_json := replace(v_payload, '\u0000', '')::jsonb; + EXCEPTION WHEN untranslatable_character OR invalid_text_representation THEN + v_json := NULL; + END; + CONTINUE WHEN v_json IS NULL; + ELSE + v_json := v_payload::jsonb; + END IF; + -- One row's values at once; the cap is looked at row by + -- row, and the last row is cut to it. + SELECT string_agg(left(j #>> '{}', p_value_max), ' ') INTO v_value + FROM jsonb_path_query(v_json, 'strict $.** ? (@.type() == "string" || @.type() == "number")') j; + CONTINUE WHEN v_value IS NULL; + v_parts := array_append(v_parts, v_value); + v_length := v_length + length(v_value) + CASE WHEN v_length = 0 THEN 0 ELSE 1 END; + IF v_length >= p_max THEN + RETURN left(array_to_string(v_parts, ' '), p_max); + END IF; + END LOOP; + RETURN array_to_string(v_parts, ' '); + END; + $$ LANGUAGE plpgsql; diff --git a/crates/weft-task-store/migrations/task/20261005T150244_task_line_cancels_run_search_arrival.sql b/crates/weft-task-store/migrations/task/20261005T150244_task_line_cancels_run_search_arrival.sql new file mode 100644 index 00000000..191e4ff7 --- /dev/null +++ b/crates/weft-task-store/migrations/task/20261005T150244_task_line_cancels_run_search_arrival.sql @@ -0,0 +1,16 @@ +CREATE OR REPLACE FUNCTION task_ready_notify() + RETURNS trigger + LANGUAGE plpgsql +AS $function$ + BEGIN + IF NEW.kind = 'cancel_execution' THEN + PERFORM weft_announce('weft_cancel', COALESCE(NEW.project_id::text, '') || ' ' || COALESCE(NEW.execution_id, '')); + ELSE + PERFORM weft_announce('weft_task_ready', + CASE WHEN NEW.target = 'dispatcher' THEN 'dispatcher' + ELSE 'worker:' || COALESCE(NEW.project_id::text, '') END); + END IF; + RETURN NULL; + END; + $function$ +; diff --git a/crates/weft-task-store/migrations/trigger_activation/20261005T183820_trigger_activation_line_cancels_run_search_arrival.sql b/crates/weft-task-store/migrations/trigger_activation/20261005T183820_trigger_activation_line_cancels_run_search_arrival.sql new file mode 100644 index 00000000..5cedd677 --- /dev/null +++ b/crates/weft-task-store/migrations/trigger_activation/20261005T183820_trigger_activation_line_cancels_run_search_arrival.sql @@ -0,0 +1,31 @@ +ALTER TABLE trigger_activation ADD COLUMN went_down_at_unix bigint; + +ALTER TABLE trigger_activation ADD COLUMN went_down_with text; + +CREATE OR REPLACE FUNCTION trigger_activation_old_health_flag() + RETURNS trigger + LANGUAGE plpgsql +AS $function$ + BEGIN + IF NEW.status = 'activating' THEN + NEW.went_down_with := NULL; + NEW.went_down_at_unix := NULL; + ELSIF NEW.deactivated_by_health AND NEW.went_down_with IS NULL THEN + NEW.went_down_with := 'health'; + NEW.went_down_at_unix := EXTRACT(EPOCH FROM NOW())::BIGINT; + ELSIF NOT NEW.deactivated_by_health AND NEW.went_down_with = 'health' THEN + NEW.went_down_with := NULL; + NEW.went_down_at_unix := NULL; + END IF; + RETURN NEW; + END; + $function$ +; + +DROP TRIGGER trigger_activation_held_on_status ON trigger_activation; +CREATE TRIGGER trigger_activation_held_on_status AFTER UPDATE OF status, accepting_fires, fires_deadline_unix ON trigger_activation FOR EACH ROW WHEN (((new.status IS DISTINCT FROM old.status) OR (new.accepting_fires IS DISTINCT FROM old.accepting_fires) OR (new.fires_deadline_unix IS DISTINCT FROM old.fires_deadline_unix))) EXECUTE FUNCTION signal_held_notify(); + +CREATE TRIGGER trigger_activation_old_health_flag BEFORE INSERT OR UPDATE ON trigger_activation FOR EACH ROW EXECUTE FUNCTION trigger_activation_old_health_flag(); + +DROP TRIGGER trigger_activation_routes_on_status ON trigger_activation; +CREATE TRIGGER trigger_activation_routes_on_status AFTER UPDATE OF status, accepting_fires, fires_deadline_unix ON trigger_activation FOR EACH ROW WHEN (((new.status IS DISTINCT FROM old.status) OR (new.accepting_fires IS DISTINCT FROM old.accepting_fires) OR (new.fires_deadline_unix IS DISTINCT FROM old.fires_deadline_unix))) EXECUTE FUNCTION trigger_activation_routes_notify(); diff --git a/crates/weft-task-store/src/announce.rs b/crates/weft-task-store/src/announce.rs new file mode 100644 index 00000000..a0423fb0 --- /dev/null +++ b/crates/weft-task-store/src/announce.rs @@ -0,0 +1,200 @@ +//! Announcing the writes a run makes, without making every commit wait on +//! every other. +//! +//! A transaction that calls `pg_notify` takes one lock shared by the whole +//! database while it commits, and holds it until its commit is on disk, +//! because Postgres hands notifications out in commit order. Every write +//! that announces itself therefore commits one after another, whatever +//! rows they touch. A run writes several (its birth, its journal rows, its +//! task's end), so under load the database spent most of its time with +//! writes queued on that one lock. +//! +//! So the writes a run makes do not notify. Their triggers put what they +//! would have announced in [`TABLE`] (`weft_announce(channel, payload)`), +//! a row like any other that commits with the write or not at all, and +//! takes no shared lock. After the write commits, its writer pokes this +//! process's flusher ([`committed`]), which takes every row waiting there +//! and sends them all in ONE transaction: the shared lock is taken once +//! per batch, not once per write. Under load the flusher runs back to back +//! and each batch carries every write that committed meanwhile; with +//! nothing to do it sends nothing. +//! +//! Listeners are unchanged: they hear the same channels with the same +//! payloads, and read their rows when woken. A writer that forgets to poke +//! only delays its announcement until the next poke of any writer on the +//! database, since a flush takes every row waiting; a process that dies +//! between its commit and its flush leaves its rows for the next one the +//! same way, and a process's flusher sends what it finds waiting as it +//! starts. While a release rolls out, a process still on the one before +//! writes through the same triggers and never pokes: what it announces +//! goes out with the next write of a process on the new one, or, on a +//! quiet database, at the waiter's own next look. +//! +//! The rare writes (a connection changed, an infra copy's status, a domain) +//! still notify from their own trigger: they commit seldom, so the lock +//! costs them nothing, and they keep needing no poke. + +use std::collections::HashMap; +use std::sync::{Arc, Mutex, OnceLock}; +use std::time::Duration; + +use sqlx::postgres::PgPool; + +/// Where the writes a run makes leave what they announce. +pub const TABLE: &str = "weft_announcement"; + +/// How many announcements one flush sends at most; a fuller outbox is +/// flushed again at once. +const BATCH: i32 = 10_000; + +pub static GROUP: crate::SchemaGroup = crate::SchemaGroup { + name: "announcement", + tables: &["weft_announcement"], + ddl: &[ + // UNLOGGED: an announcement only says when to look at rows that + // are durable on their own, so a database crash losing the ones + // not sent yet costs a waiter its next safety look, never data. + r#"CREATE UNLOGGED TABLE IF NOT EXISTS weft_announcement ( + channel TEXT NOT NULL, + payload TEXT NOT NULL + )"#, + // What a trigger on a run's path calls instead of `pg_notify`. + r#"CREATE OR REPLACE FUNCTION weft_announce(p_channel TEXT, p_payload TEXT) RETURNS VOID AS $$ + INSERT INTO weft_announcement (channel, payload) VALUES (p_channel, p_payload) + $$ LANGUAGE sql"#, + // The flush ([`flush`]): take up to `p_limit` waiting rows (those + // another flusher is sending right now are left to it), send each + // distinct one, all in the caller's one transaction, and answer + // how many rows were taken. + r#"CREATE OR REPLACE FUNCTION weft_flush_announcements(p_limit INTEGER) RETURNS INTEGER AS $$ + DECLARE + v_channels TEXT[]; + v_payloads TEXT[]; + BEGIN + WITH taken AS ( + DELETE FROM weft_announcement a + WHERE a.ctid IN (SELECT ctid FROM weft_announcement FOR UPDATE SKIP LOCKED LIMIT p_limit) + RETURNING channel, payload) + SELECT array_agg(channel), array_agg(payload) INTO v_channels, v_payloads FROM taken; + IF v_channels IS NULL THEN + RETURN 0; + END IF; + PERFORM pg_notify(d.channel, d.payload) + FROM (SELECT DISTINCT u.channel, u.payload FROM unnest(v_channels, v_payloads) AS u(channel, payload)) d; + RETURN array_length(v_channels, 1); + END; + $$ LANGUAGE plpgsql"#, + ], + seed: &[], +}; + +/// Tell this process's flusher for `pool`'s database that a write which +/// may have left announcements just committed. +pub fn committed(pool: &PgPool) { + flusher(pool).wanted.notify_one(); +} + +/// One database's flusher in this process, and the pool it sends through: +/// the last one a write poked with that is still open, so a pool closing +/// (a test's, at its end) never leaves the database without one. +struct Flusher { + wanted: tokio::sync::Notify, + pool: Mutex, +} + +/// The flushers of this process, by database ([`database_key`]). An entry +/// leaves when its flusher stops, so the next poke starts another. +type Flushers = Mutex>>; + +fn flushers() -> &'static Flushers { + static FLUSHERS: OnceLock = OnceLock::new(); + FLUSHERS.get_or_init(Mutex::default) +} + +/// Where `pool` connects: one flusher per database, whatever the number of +/// pools on it. +fn database_key(pool: &PgPool) -> String { + let options = pool.connect_options(); + format!("{}:{}/{}", options.get_host(), options.get_port(), options.get_database().unwrap_or("")) +} + +/// The flusher of `pool`'s database, started when there is none. +fn flusher(pool: &PgPool) -> Arc { + let key = database_key(pool); + let mut flushers = flushers().lock().expect("announcement flushers"); + if let Some(flusher) = flushers.get(&key) { + let mut sends_through = flusher.pool.lock().expect("announcement flusher pool"); + if sends_through.is_closed() { + *sends_through = pool.clone(); + } + return flusher.clone(); + } + let flusher = Arc::new(Flusher { wanted: tokio::sync::Notify::new(), pool: Mutex::new(pool.clone()) }); + tokio::spawn(flush_when_wanted(key.clone(), flusher.clone())); + flushers.insert(key, flusher.clone()); + flusher +} + +/// Takes a flusher out of [`flushers`] if its task ends (the runtime it +/// ran on went away), so a later poke on the same database starts a new +/// one instead of waking nobody. +struct Stopped { + key: String, + flusher: Arc, +} + +impl Drop for Stopped { + fn drop(&mut self) { + let mut flushers = flushers().lock().expect("announcement flushers"); + if flushers.get(&self.key).is_some_and(|f| Arc::ptr_eq(f, &self.flusher)) { + flushers.remove(&self.key); + } + } +} + +/// Flush at once (what a process that died between its commit and its +/// flush left waiting goes out with the first write after it), then each +/// time a write asks. +async fn flush_when_wanted(key: String, flusher: Arc) { + let _stopped = Stopped { key, flusher: flusher.clone() }; + loop { + let pool = flusher.pool.lock().expect("announcement flusher pool").clone(); + flush_waiting(&pool).await; + flusher.wanted.notified().await; + } +} + +/// How long a flush that failed waits before it is tried again, at first +/// and at most. +const RETRY_FIRST: Duration = Duration::from_millis(500); +const RETRY_LONGEST: Duration = Duration::from_secs(30); + +/// Send everything waiting, in batches of [`BATCH`]. A flush that fails is +/// tried again until it goes through: the rows stay in the outbox +/// meanwhile, and the writes that left them wait on what they announce. +async fn flush_waiting(pool: &PgPool) { + let mut retry = RETRY_FIRST; + loop { + match flush(pool).await { + Ok(sent) if sent >= BATCH as usize => retry = RETRY_FIRST, + Ok(_) => return, + Err(_) if pool.is_closed() => return, + Err(e) => { + tracing::warn!( + target: "weft_task_store::announce", + error = %e, retry_in_ms = retry.as_millis() as u64, + "could not send the waiting announcements; trying again" + ); + tokio::time::sleep(retry).await; + retry = (retry * 2).min(RETRY_LONGEST); + } + } + } +} + +/// Send what waits in the outbox, in one transaction +/// (`weft_flush_announcements`): how many rows it took. +async fn flush(pool: &PgPool) -> Result { + let taken: i32 = sqlx::query_scalar("SELECT weft_flush_announcements($1)").bind(BATCH).fetch_one(pool).await?; + Ok(taken.max(0) as usize) +} diff --git a/crates/weft-task-store/src/kinds.rs b/crates/weft-task-store/src/kinds.rs index 2ee3d8ad..6b0a0b9e 100644 --- a/crates/weft-task-store/src/kinds.rs +++ b/crates/weft-task-store/src/kinds.rs @@ -33,8 +33,8 @@ pub enum TaskKind { /// Worker: resume a suspended execution after a fire. Resume, /// Worker: cancel a running execution by execution. Never claimed: the - /// worker driving the execution takes it through its cancel wait - /// (`tasks::take_cancels`). + /// worker driving the execution hears it announced on its line and asks + /// for it (`tasks::cancels_asked`). CancelExecution, /// Dispatcher: journal a `CostReported` event for one metered /// call (a provider meter's figure). Routed diff --git a/crates/weft-task-store/src/lib.rs b/crates/weft-task-store/src/lib.rs index cec47553..b6e105a1 100644 --- a/crates/weft-task-store/src/lib.rs +++ b/crates/weft-task-store/src/lib.rs @@ -18,6 +18,8 @@ //! - `pg_signal`: the process's one Postgres `LISTEN` connection, //! which every wait on a row sleeps on (`terminal` is the task //! waiter built on it). +//! - `announce`: how the writes a run makes announce themselves without +//! every commit waiting on every other (an outbox, flushed in batches). //! - `held_copy`: a process's copy of rows read on every request, //! dropped the moment a notification says they changed. //! - `schema_guard`: the schema runner every boot routes its @@ -26,6 +28,7 @@ //! group's migration files. pub mod alarm; +pub mod announce; pub mod db; pub mod drain; pub mod executor; diff --git a/crates/weft-task-store/src/schema_guard.rs b/crates/weft-task-store/src/schema_guard.rs index 61ebc515..f9d5dd3e 100644 --- a/crates/weft-task-store/src/schema_guard.rs +++ b/crates/weft-task-store/src/schema_guard.rs @@ -25,20 +25,30 @@ //! each way and compares them, which is what catches a migration that runs //! but does not land on the same shape. //! -//! Applied in one transaction under an advisory lock, in the same order -//! the agreement test replays: first the DDL of every group new to this -//! database (its migrations are recorded as history it was born with), -//! then every PENDING migration across every group in one global id -//! order (ids are timestamps, so this is the order the changes were -//! actually written, and what a migration referencing another group's -//! table relies on), then every group's DDL re-run (idempotent, and it -//! repairs a hand-dropped table), and every group's seeds re-run. +//! A database that already holds what the code declares (every group +//! stamped with its current fingerprint, every table there, no migration +//! pending, its history intact) is left alone: a boot only reads, takes no +//! lock, and runs no DDL. That is every boot but the first of a new release, and it matters +//! because a boot can land on a database serving traffic: another copy of +//! the role starting under load. Running a group's DDL locks its tables +//! (a trigger is dropped and created again), so a boot that re-ran it +//! stalled every write and, under enough of them, deadlocked with them. +//! +//! When there is work, it is applied in one transaction under an advisory +//! lock, in the same order the agreement test replays: first the DDL of +//! every group new to this database (its migrations are recorded as +//! history it was born with), then every PENDING migration across every +//! group in one global id order (ids are timestamps, so this is the order +//! the changes were actually written, and what a migration referencing +//! another group's table relies on), then every group's seeds. The +//! transaction waits a short while for each table lock it needs, never +//! behind the writes queued on it; one that cannot get its locks, or meets +//! a write in a deadlock, is rolled back whole and tried again, since a +//! release's work on a busy database is exactly that. //! //! **A stamp that moved is refreshed on evidence, never on faith**: the //! canonical DDL is built into a scratch schema in the same transaction -//! and compared, object by object, with what the live schema holds, before -//! the group's own DDL is run on it (a replaced function would otherwise -//! make the live schema the code's whatever migrations ran). A +//! and compared, object by object, with what the live schema holds. A //! match restamps silently (the fingerprint moved with no shape change, //! or a migration provably landed the shape). A difference is the drift //! case: the boot fails naming the group and the exact objects that @@ -50,18 +60,6 @@ use std::collections::HashMap; use sha2::{Digest, Sha256}; use sqlx::PgPool; -/// Run `group`'s canonical DDL on the live schema (every statement is -/// idempotent on a database that already holds its shape). -async fn run_ddl(tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, group: &SchemaGroup) -> anyhow::Result<()> { - for stmt in group.ddl { - sqlx::raw_sql(stmt) - .execute(&mut **tx) - .await - .map_err(|e| anyhow::anyhow!("schema group '{}': DDL failed: {e}", group.name))?; - } - Ok(()) -} - /// One change to one group's tables: the file's SQL, the group it sits under, /// and the id it is filed by. Ids sort chronologically /// (`20260823T1412_add_owner`), which is the order they run in. @@ -99,11 +97,12 @@ pub struct SchemaGroup { /// semicolon-separated statements (executed via `raw_sql`). pub ddl: &'static [&'static str], /// Idempotent DML the group needs present (a cursor row, a seed - /// record), re-run on EVERY boot after the DDL. Seeds are not - /// schema: they are outside the fingerprint and outside the - /// migration history, so editing one is a non-event rather than a - /// "write the migration" dead end (a seed row is invisible to the - /// schema reader, so no migration could ever be generated for it). + /// record), run after the DDL whenever a boot has work to do. Seeds + /// are in the fingerprint, so editing one moves the stamp and the next + /// boot runs it; they are outside the migration history, so editing + /// one is a non-event rather than a "write the migration" dead end (a + /// seed row is invisible to the schema reader, so the shape check + /// finds nothing to differ and restamps). pub seed: &'static [&'static str], } @@ -218,16 +217,27 @@ fn for_group<'a>(migrations: &'a [Migration], group: &str) -> Vec<&'a Migration> migrations.iter().filter(|m| m.group == group).collect() } -/// Hex sha256 over the group's DDL statements, each followed by a newline. -/// Any textual edit to the DDL (a new column, a changed default, a comment) -/// changes the fingerprint; that is the point: the stamp certifies "the -/// database's tables were created by exactly this text". -fn fingerprint(ddl: &[&str]) -> String { +/// Hex sha256 over the group's DDL statements, each followed by a newline, +/// then its seeds the same way. Any textual edit to the DDL (a new column, a +/// changed default, a comment) changes the fingerprint; that is the point: +/// the stamp certifies "the database's tables were created by exactly this +/// text". The seeds are in it so a changed seed is a moved stamp, which is +/// what makes a boot run it: a boot whose stamps all match runs nothing. +fn fingerprint(group: &SchemaGroup) -> String { + fingerprint_of(group.ddl, group.seed) +} + +fn fingerprint_of(ddl: &[&str], seed: &[&str]) -> String { let mut hasher = Sha256::new(); for stmt in ddl { hasher.update(stmt.as_bytes()); hasher.update(b"\n"); } + hasher.update(b"-- seeds\n"); + for stmt in seed { + hasher.update(stmt.as_bytes()); + hasher.update(b"\n"); + } hex(hasher.finalize().as_slice()) } @@ -349,10 +359,137 @@ pub async fn apply_groups_with( migrations: &[Migration], ) -> anyhow::Result<()> { validate_identifiers(groups)?; + if holds_everything(pool, groups, migrations).await? { + return Ok(()); + } + let started = std::time::Instant::now(); + let mut wait = CONTENDED_FIRST; + loop { + match apply_under_lock(pool, groups, migrations).await { + Ok(()) => return Ok(()), + Err(e) if contended(&e) && started.elapsed() < CONTENDED_FOR => { + tracing::warn!( + target: "weft_task_store::schema_guard", + error = %format!("{e:#}"), + retry_in_ms = wait.as_millis() as u64, + "the schema's changes met writes holding their tables; trying again" + ); + tokio::time::sleep(wait).await; + wait = (wait * 2).min(CONTENDED_LONGEST); + } + Err(e) => return Err(e), + } + } +} + +/// How long a release's schema work waits for one table lock before it +/// gives way to the writes queued behind it, and how long, in all, it keeps +/// trying again before the boot fails naming the contention. +const LOCK_WAIT: &str = "5s"; +const CONTENDED_FIRST: std::time::Duration = std::time::Duration::from_millis(500); +const CONTENDED_LONGEST: std::time::Duration = std::time::Duration::from_secs(10); +const CONTENDED_FOR: std::time::Duration = std::time::Duration::from_secs(180); + +/// Whether `e` is the database giving up a lock (its wait ran out, or it +/// broke a deadlock with a write): the whole transaction rolled back and can +/// be tried again as it was. +fn contended(e: &anyhow::Error) -> bool { + e.chain().any(|cause| { + cause + .downcast_ref::() + .and_then(|e| e.as_database_error()) + .and_then(|db| db.code()) + .is_some_and(|code| code == "55P03" || code == "40P01") + }) +} + +/// Whether the database already holds everything `groups` and `migrations` +/// declare, read without a lock: every group stamped with its current +/// fingerprint, every table there, no migration pending, and its history +/// intact (what ran +/// unedited, nothing recorded this build has no file for, the frozen origin +/// recorded as built). Anything else, including a history this build +/// refuses, is the locked path's to do or to explain. +async fn holds_everything(pool: &PgPool, groups: &[&SchemaGroup], migrations: &[Migration]) -> anyhow::Result { + let ledgers: (bool,) = sqlx::query_as( + "SELECT to_regclass('weft_schema_stamp') IS NOT NULL AND to_regclass('weft_migration') IS NOT NULL", + ) + .fetch_one(pool) + .await?; + if !ledgers.0 { + return Ok(false); + } + let stamps: HashMap = + sqlx::query_as::<_, (String, String)>("SELECT group_name, fingerprint FROM weft_schema_stamp") + .fetch_all(pool) + .await? + .into_iter() + .collect(); + let mut applied: HashMap> = HashMap::new(); + for (group, id, sum) in + sqlx::query_as::<_, (String, String, String)>("SELECT group_name, id, checksum FROM weft_migration") + .fetch_all(pool) + .await? + { + applied.entry(group).or_default().insert(id, sum); + } + if !applied.get(HISTORY_GROUP).is_some_and(|ids| ids.contains_key(HISTORY_EPOCH)) { + return Ok(false); + } + if !missing_tables(pool, groups).await?.is_empty() { + return Ok(false); + } + let none = HashMap::new(); + Ok(groups.iter().all(|group| { + let applied = applied.get(group.name).unwrap_or(&none); + let mine = for_group(migrations, group.name); + stamps.get(group.name).is_some_and(|stamped| *stamped == fingerprint(group)) + && pending(&mine, applied).is_empty() + && edited_after_running(&mine, applied).is_empty() + && recorded_without_file(&mine, applied).is_empty() + && embedded_origin(group.name).is_none_or(|origin| applied.get(ORIGIN_ID) == Some(&checksum(origin))) + })) +} + +/// What resets `group` on a database someone is iterating on: its tables +/// dropped and its records gone, so the next boot builds it from the +/// current DDL. +fn reset_sql(group: &SchemaGroup) -> String { + let mut sql = String::new(); + for table in group.tables { + sql.push_str(&format!("DROP TABLE IF EXISTS {table} CASCADE;\n")); + } + sql.push_str(&format!( + "DELETE FROM weft_schema_stamp WHERE group_name = '{}';\n\ + DELETE FROM weft_migration WHERE group_name = '{}';\n", + group.name, group.name + )); + sql +} + +/// The tables `groups` declare that the database does not hold. +async fn missing_tables<'e>( + executor: impl sqlx::Executor<'e, Database = sqlx::Postgres>, + groups: &[&SchemaGroup], +) -> anyhow::Result> { + let tables: Vec<&str> = groups.iter().flat_map(|g| g.tables.iter().copied()).collect(); + Ok(sqlx::query_scalar("SELECT t FROM unnest($1::text[]) AS t WHERE to_regclass(t) IS NULL") + .bind(&tables) + .fetch_all(executor) + .await?) +} + +/// The schema's work, in one transaction under the advisory lock (see the +/// module doc). +async fn apply_under_lock(pool: &PgPool, groups: &[&SchemaGroup], migrations: &[Migration]) -> anyhow::Result<()> { let mut tx = pool.begin().await?; sqlx::query("SELECT pg_advisory_xact_lock(hashtextextended('weft:schema-migrate', 0))") .execute(&mut *tx) .await?; + // Every table lock taken from here waits this long at most, so the + // work never sits at the head of a table's queue holding back the + // writes behind it (`apply_groups_with` tries again). + sqlx::query(&format!("SET LOCAL lock_timeout = '{LOCK_WAIT}'")).execute(&mut *tx).await?; sqlx::raw_sql( "CREATE TABLE IF NOT EXISTS weft_schema_stamp (\ group_name TEXT PRIMARY KEY, fingerprint TEXT NOT NULL);\ @@ -444,7 +581,7 @@ pub async fn apply_groups_with( } states.push(GroupState { group, - fp: fingerprint(group.ddl), + fp: fingerprint(group), stamped: stored.map(|(fp,)| fp), applied, }); @@ -458,9 +595,10 @@ pub async fn apply_groups_with( continue; } for stmt in st.group.ddl { - sqlx::raw_sql(stmt).execute(&mut *tx).await.map_err(|e| { - anyhow::anyhow!("schema group '{}': DDL failed: {e}", st.group.name) - })?; + sqlx::raw_sql(stmt) + .execute(&mut *tx) + .await + .map_err(|e| anyhow::Error::from(e).context(format!("schema group '{}': DDL failed", st.group.name)))?; } sqlx::query("INSERT INTO weft_schema_stamp (group_name, fingerprint) VALUES ($1, $2)") .bind(st.group.name) @@ -475,6 +613,30 @@ pub async fn apply_groups_with( } } + // Pass 2b: a group this database holds at its current shape (stamp + // current, nothing pending) whose table is gone (dropped by hand) is + // built again from its canonical DDL, every statement of which is + // idempotent on what is still there (it runs on the group's other + // tables too, and on any table its triggers sit on). A group with a + // migration pending is not: a table its migration creates is missing + // until pass 3 runs it, and the canonical DDL is the shape after every + // migration, which a pending one would then run on. Its missing tables + // are pass 3's to make, or pass 4's to name. + for st in &states { + if st.stamped.as_deref() != Some(st.fp.as_str()) + || !pending(&for_group(migrations, st.group.name), &st.applied).is_empty() + || missing_tables(&mut *tx, &[st.group]).await?.is_empty() + { + continue; + } + tracing::warn!(target: "weft_task_store::schema_guard", group = st.group.name, "a table of this group is gone; building it again"); + for stmt in st.group.ddl { + sqlx::raw_sql(stmt).execute(&mut *tx).await.map_err(|e| { + anyhow::Error::from(e).context(format!("schema group '{}': rebuilding a table that is gone failed", st.group.name)) + })?; + } + } + // Pass 3: one global run of everything pending across every stamped // group, sorted by id. Ids are timestamps, so this is the order the // changes were written, the same order the agreement test replays, @@ -492,17 +654,13 @@ pub async fn apply_groups_with( todo.sort_by_key(|(_, m)| m.id); for (group, m) in todo { sqlx::raw_sql(m.sql).execute(&mut *tx).await.map_err(|e| { - anyhow::anyhow!("schema group '{}': migration '{}' failed: {e}", group.name, m.id) + anyhow::Error::from(e).context(format!("schema group '{}': migration '{}' failed", group.name, m.id)) })?; record_migration(&mut tx, group.name, m.id, m.sql).await?; } // Pass 4: verify every moved stamp against the database itself, as it - // stands after pass 3's migrations and before any DDL of this boot runs - // on it: a `CREATE OR REPLACE` or a dropped and re-created trigger would - // otherwise rewrite the live schema to the new text first, and the - // check would compare the code with itself, letting an edit no - // migration carries boot. The canonical DDL is built into a scratch + // stands after pass 3's migrations. The canonical DDL is built into a scratch // schema inside this same transaction and compared, object by object, // with what the live schema holds. A group whose live shape matches is // restamped: a fingerprint that moved with no shape change (a comment, @@ -527,10 +685,7 @@ pub async fn apply_groups_with( for group in groups { for stmt in group.ddl { sqlx::raw_sql(stmt).execute(&mut *tx).await.map_err(|e| { - anyhow::anyhow!( - "schema group '{}': DDL failed in the shape check: {e}", - group.name - ) + anyhow::Error::from(e).context(format!("schema group '{}': DDL failed in the shape check", group.name)) })?; } } @@ -563,18 +718,11 @@ pub async fn apply_groups_with( // swallow the one message this mechanism exists to produce. if !mismatched.is_empty() { let names: Vec<&str> = mismatched.iter().map(|(g, _)| g.name).collect(); - let mut reset_sql = String::new(); + let mut reset = String::new(); let mut diffs = String::new(); for (group, diff) in &mismatched { diffs.push_str(&format!("group '{}':\n{diff}", group.name)); - for table in group.tables { - reset_sql.push_str(&format!("DROP TABLE IF EXISTS {table} CASCADE;\n")); - } - reset_sql.push_str(&format!( - "DELETE FROM weft_schema_stamp WHERE group_name = '{}';\n\ - DELETE FROM weft_migration WHERE group_name = '{}';\n", - group.name, group.name - )); + reset.push_str(&reset_sql(group)); } anyhow::bail!( "the canonical schema changed and this database does not hold the shape it \ @@ -586,26 +734,17 @@ pub async fn apply_groups_with( then restart the service so they are rebuilt from the current DDL.", names.join(", "), diffs, - reset_sql + reset ); } - // Pass 5: re-run every stamped group's DDL, in the groups' order - // (idempotent on the shape just verified; repairs a hand-dropped table - // of a group whose stamp did not move). - for st in &states { - if st.stamped.is_some() { - run_ddl(&mut tx, st.group).await?; - } - } - - // Pass 6: seeds, idempotent DML re-run on every boot, after every - // table exists. + // Pass 5: seeds, idempotent DML, after every table exists. for group in groups { for stmt in group.seed { - sqlx::raw_sql(stmt).execute(&mut *tx).await.map_err(|e| { - anyhow::anyhow!("schema group '{}': seed failed: {e}", group.name) - })?; + sqlx::raw_sql(stmt) + .execute(&mut *tx) + .await + .map_err(|e| anyhow::Error::from(e).context(format!("schema group '{}': seed failed", group.name)))?; } } @@ -854,6 +993,12 @@ const THING_QUERIES: &[&str] = &[ FROM pg_type t JOIN pg_enum e ON e.enumtypid = t.oid \ JOIN pg_namespace n ON n.oid = t.typnamespace \ WHERE n.nspname = $1 GROUP BY t.typname", + // A table written without the write-ahead log (`CREATE UNLOGGED + // TABLE`): its rows go with a crash, which is part of what the table + // promises, so a table that lost or gained it differs. + "SELECT 'persistence' AS kind, rel.relname AS \"table\", 'unlogged' AS name, 'UNLOGGED' AS body \ + FROM pg_class rel JOIN pg_namespace n ON n.oid = rel.relnamespace \ + WHERE n.nspname = $1 AND rel.relkind = 'r' AND rel.relpersistence = 'u'", ]; /// Who a planned statement belongs to, so [`file_per_group`] can file it @@ -1315,6 +1460,7 @@ fn added(thing: &Thing) -> String { "constraint" => { format!("ALTER TABLE {} ADD CONSTRAINT {} {};", thing.table, thing.name, thing.body) } + "persistence" => format!("ALTER TABLE {} SET UNLOGGED;", thing.table), _ => format!("{};", thing.body), } } @@ -1430,6 +1576,7 @@ fn removed(thing: &Thing) -> String { "constraint" => { format!("ALTER TABLE {} DROP CONSTRAINT IF EXISTS {};", thing.table, thing.name) } + "persistence" => format!("ALTER TABLE {} SET LOGGED;", thing.table), _ => format!( "-- Throws away what is in {}.{}. Ship this in a later release than the one \n\ -- that stopped reading the column, so replicas still on the old release do not fall over.\n\ @@ -2157,7 +2304,7 @@ mod tests { // a database still owes. use std::collections::HashMap; - use super::{checksum, edited_after_running, fingerprint, pending, recorded_without_file, Migration, ORIGIN_ID}; + use super::{checksum, edited_after_running, fingerprint_of, pending, recorded_without_file, Migration, ORIGIN_ID}; static A: Migration = Migration { group: "g", id: "20260101_a", draft: false, sql: "SELECT 1" }; static B: Migration = Migration { group: "g", id: "20260202_b", draft: false, sql: "SELECT 2" }; @@ -2165,8 +2312,8 @@ mod tests { #[test] fn same_ddl_same_fingerprint() { - let a = fingerprint(&["CREATE TABLE t (id INT)", "CREATE INDEX i ON t(id)"]); - let b = fingerprint(&["CREATE TABLE t (id INT)", "CREATE INDEX i ON t(id)"]); + let a = fingerprint_of(&["CREATE TABLE t (id INT)", "CREATE INDEX i ON t(id)"], &[]); + let b = fingerprint_of(&["CREATE TABLE t (id INT)", "CREATE INDEX i ON t(id)"], &[]); assert_eq!(a, b); // Hex sha256: 64 lowercase hex chars. assert_eq!(a.len(), 64); @@ -2175,8 +2322,8 @@ mod tests { #[test] fn changed_ddl_changes_fingerprint() { - let base = fingerprint(&["CREATE TABLE t (id INT)"]); - let edited = fingerprint(&["CREATE TABLE t (id INT, name TEXT)"]); + let base = fingerprint_of(&["CREATE TABLE t (id INT)"], &[]); + let edited = fingerprint_of(&["CREATE TABLE t (id INT, name TEXT)"], &[]); assert_ne!(base, edited); } @@ -2203,8 +2350,8 @@ mod tests { fn statement_boundaries_matter() { // The per-statement newline separator keeps ["ab"] distinct from // ["a", "b"]: concatenation without a separator would collide. - let joined = fingerprint(&["ab"]); - let split = fingerprint(&["a", "b"]); + let joined = fingerprint_of(&["ab"], &[]); + let split = fingerprint_of(&["a", "b"], &[]); assert_ne!(joined, split); } diff --git a/crates/weft-task-store/src/tasks.rs b/crates/weft-task-store/src/tasks.rs index db812f64..d6c005a4 100644 --- a/crates/weft-task-store/src/tasks.rs +++ b/crates/weft-task-store/src/tasks.rs @@ -141,12 +141,33 @@ pub struct ClaimedExecution { } /// The channel a task that has just become claimable (inserted pending, -/// or put back to pending by a requeue or a reclaim) notifies on, from -/// the `task_ready_notify` trigger in [`GROUP`]. A lease that merely -/// expires announces nothing: a waiter's own deadline is what rescues -/// it. +/// or put back to pending by a requeue or a reclaim) is announced on, from +/// the `task_ready_notify` trigger in [`GROUP`] (through +/// `crate::announce`). A lease that merely expires announces nothing: a +/// waiter's own deadline is what rescues it. A cancel is announced on +/// [`CANCEL_CHANNEL`] instead: only the worker driving its run cares. +// SYNC: TASK_READY_CHANNEL <-> 'weft_task_ready' in the `task_ready_notify` function in GROUP pub const TASK_READY_CHANNEL: &str = "weft_task_ready"; +/// The channel a cancel for a running execution is announced on, with +/// ` ` ([`cancel_payload`]): the broker pushes +/// it down the lines of the project's workers, and the one driving the +/// execution asks for it ([`cancels_asked`]). +// SYNC: CANCEL_CHANNEL <-> weft_broker_client::line::LINE_CHANNELS, crates/weft-broker/src/line.rs (audience), 'weft_cancel' in the `task_ready_notify` function in GROUP +pub const CANCEL_CHANNEL: &str = "weft_cancel"; + +/// A [`CANCEL_CHANNEL`] payload. +// SYNC: cancel_payload <-> the `task_ready_notify` function in `GROUP`, parse_cancel_payload +pub fn cancel_payload(project_id: Uuid, execution_id: &str) -> String { + format!("{project_id} {execution_id}") +} + +/// The project and execution a [`CANCEL_CHANNEL`] payload names. +pub fn parse_cancel_payload(payload: &str) -> Option<(Uuid, &str)> { + let (project, execution) = payload.split_once(' ')?; + Some((project.parse().ok()?, execution)) +} + /// A [`TASK_READY_CHANNEL`] payload: `dispatcher` for a dispatcher task, /// `worker:` for a worker one, since only that project's /// workers can claim it (and the delivery sweep wakes on it). @@ -282,16 +303,22 @@ pub static GROUP: crate::SchemaGroup = crate::SchemaGroup { $$ LANGUAGE plpgsql"#, // Announce every task that has just become claimable, so the // pickers sleep until there is work instead of asking on a - // timer. From a trigger rather than from each writer, so no - // write path (an enqueue, a live admission, a requeue, the - // orphan reclaim) can forget it, and the notification goes out - // when the write commits. - // SYNC: task_ready_notify's payload <-> ready_payload above. + // timer, and a cancel to the workers of its project, so the one + // driving its run hears it. From a trigger rather than from each + // writer, so no write path (an enqueue, a live admission, a + // requeue, the orphan reclaim) can forget it; through the + // announcement outbox (`crate::announce`), since a task is + // written on every run. + // SYNC: task_ready_notify's channels and payloads <-> TASK_READY_CHANNEL, CANCEL_CHANNEL, ready_payload, cancel_payload above. r#"CREATE OR REPLACE FUNCTION task_ready_notify() RETURNS trigger AS $$ BEGIN - PERFORM pg_notify('weft_task_ready', - CASE WHEN NEW.target = 'dispatcher' THEN 'dispatcher' - ELSE 'worker:' || COALESCE(NEW.project_id::text, '') END); + IF NEW.kind = 'cancel_execution' THEN + PERFORM weft_announce('weft_cancel', COALESCE(NEW.project_id::text, '') || ' ' || COALESCE(NEW.execution_id, '')); + ELSE + PERFORM weft_announce('weft_task_ready', + CASE WHEN NEW.target = 'dispatcher' THEN 'dispatcher' + ELSE 'worker:' || COALESCE(NEW.project_id::text, '') END); + END IF; RETURN NULL; END; $$ LANGUAGE plpgsql"#, @@ -330,8 +357,9 @@ pub static GROUP: crate::SchemaGroup = crate::SchemaGroup { // SYNC: execution.owner_replica has exactly two writers, // this trigger and `bind_execution_id_owner` below (the appointed-driver // path for processes that never claim a task); the readers are - // crates/weft-broker/src/handlers.rs journal_record and - // crates/weft-broker/src/auth.rs resolve_storage_caller. + // crates/weft-broker/src/handlers.rs (require_owner, task_cancels_asked), + // crates/weft-broker/src/auth.rs resolve_storage_caller, and + // weft_journal_append in crates/weft-dispatcher/src/journal/postgres.rs. r#"CREATE OR REPLACE FUNCTION weft_bind_execution_id_owner() RETURNS trigger AS $$ BEGIN IF NEW.execution_id IS NOT NULL AND NEW.claimed_by IS NOT NULL THEN @@ -349,14 +377,13 @@ pub static GROUP: crate::SchemaGroup = crate::SchemaGroup { // task UPDATE (complete, heartbeat-renew, requeue), so the // hot path stamps ownership exactly once per claim; AND // - tasks that actually DRIVE the execution: 'execute' and - // 'resume'. A side-channel task that merely carries an execution - // (a 'cancel_execution' addressed to the owner) must NOT - // restamp ownership: ownership follows the driver. (cancel is - // pinned to the owner anyway, so even if it did fire the - // stamp would be owner->owner, but scoping by kind makes a - // future execution-bearing task kind unable to steal ownership by + // 'resume'. A task that merely carries an execution must NOT + // restamp ownership: ownership follows the driver. (A + // 'cancel_execution' is never claimed, only read by the worker + // driving its run, but scoping by kind makes any + // execution-bearing task kind unable to steal ownership by // accident, which is the property we want to hold by - // construction, not by every caller remembering to pin.) + // construction.) r#"CREATE TRIGGER task_claim_binds_execution_id_owner AFTER UPDATE OF claimed_by ON task FOR EACH ROW @@ -392,6 +419,7 @@ pub async fn enqueue(pool: &PgPool, spec: NewTask) -> Result { .bind(now) .execute(pool) .await?; + crate::announce::committed(pool); Ok(id) } @@ -408,12 +436,15 @@ pub async fn enqueue(pool: &PgPool, spec: NewTask) -> Result { /// AlreadyLive. The whole of it is one statement (`weft_enqueue_dedup`), /// so on its own it is its own transaction: one round trip. pub async fn enqueue_dedup(pool: &PgPool, spec: NewTask) -> Result { - enqueue_dedup_in(&mut *pool.acquire().await?, spec).await + let outcome = enqueue_dedup_in(&mut *pool.acquire().await?, spec).await?; + crate::announce::committed(pool); + Ok(outcome) } /// [`enqueue_dedup`] on a caller-owned connection: inside the caller's /// transaction, the insert commits with whatever else it writes, and the -/// advisory lock lasts until it ends. +/// advisory lock lasts until it ends. The caller pokes the announcement +/// flusher once it commits (`crate::announce::committed`). pub async fn enqueue_dedup_in( conn: &mut sqlx::PgConnection, spec: NewTask, @@ -519,6 +550,7 @@ pub async fn enqueue_or_rearm(pool: &PgPool, spec: NewTask) -> Result enqueue_dedup_in(&mut tx, spec).await?, }; tx.commit().await?; + crate::announce::committed(pool); Ok(outcome) } @@ -807,6 +839,7 @@ pub async fn fail_undeliverable(pool: &PgPool, task_id: Uuid, error: &str) -> Re .bind(task_id) .fetch_all(pool) .await?; + crate::announce::committed(pool); Ok(()) } @@ -827,26 +860,21 @@ pub struct CancelAsked { pub cause: weft_core::exec::CancelCause, } -/// Take the pending `cancel_execution` tasks of `project_id` for any of -/// `execution_ids` (the executions the asking worker drives), completing each as -/// it is taken: the worker fires the execution's flag the moment it hears. -pub async fn take_cancels(pool: &PgPool, project_id: Uuid, execution_ids: &[String]) -> Result> { +/// The pending `cancel_execution` tasks of `project_id` for any of +/// `execution_ids` (the executions the asking worker drives): the worker +/// fires each execution's flag the moment it hears. Asking only reads, so +/// an answer lost on its way back costs nothing (the worker asks again +/// and finds the same cancels); a cancel goes once nothing drives its +/// execution any more ([`drop_stale_cancels`]). +pub async fn cancels_asked(pool: &PgPool, project_id: Uuid, execution_ids: &[String]) -> Result> { if execution_ids.is_empty() { return Ok(Vec::new()); } - let now = unix_now(); - // Taken and announced finished in one statement: a canceller waiting - // on the task hears it the moment the worker has it. - let rows = sqlx::query(&format!( - "WITH taken AS ( \ - UPDATE task SET status = 'complete', completed_at_unix = $1, result = 'null'::jsonb \ - WHERE target = 'worker' AND kind = 'cancel_execution' AND status = 'pending' \ - AND project_id = $2 AND execution_id = ANY($3) \ - RETURNING id, execution_id, payload) \ - SELECT execution_id, payload, pg_notify('{}', id::text) FROM taken", - crate::terminal::TERMINAL_CHANNEL - )) - .bind(now) + let rows = sqlx::query( + "SELECT execution_id, payload FROM task \ + WHERE target = 'worker' AND kind = 'cancel_execution' AND status = 'pending' \ + AND project_id = $1 AND execution_id = ANY($2)", + ) .bind(project_id) .bind(execution_ids) .fetch_all(pool) @@ -899,6 +927,7 @@ pub async fn requeue(pool: &PgPool, task_id: Uuid, replica: &str) -> Result 0) } @@ -948,16 +977,17 @@ pub async fn stored_result(pool: &PgPool, task_id: Uuid) -> Result } /// `update` (an `UPDATE task ... ` that makes rows terminal) wrapped so -/// each row it changes notifies [`crate::terminal::TERMINAL_CHANNEL`] -/// with its id, in the same statement: the notification goes out when -/// the write commits. Returns one row per task it changed. A row the -/// update put back to pending (`rerun_requested`) is changed but not -/// terminal, so it is returned without the notification. +/// each row it changes is announced on [`crate::terminal::TERMINAL_CHANNEL`] +/// with its id, in the same statement (`crate::announce`: the +/// announcement commits with the write, and its caller pokes the flusher). +/// Returns one row per task it changed. A row the update put back to +/// pending (`rerun_requested`) is changed but not terminal, so it is +/// returned without the announcement. fn notify_terminal(update: &str) -> String { format!( "WITH done AS ({update} RETURNING id, status) \ SELECT CASE WHEN status = 'pending' THEN NULL \ - ELSE pg_notify('{}', id::text) END FROM done", + ELSE weft_announce('{}', id::text) END FROM done", crate::terminal::TERMINAL_CHANNEL ) } @@ -997,6 +1027,7 @@ pub async fn complete( .bind(replica) .fetch_all(pool) .await?; + crate::announce::committed(pool); if updated.is_empty() { anyhow::bail!("complete: task {task_id} no longer claimed by {replica}"); } @@ -1021,6 +1052,7 @@ pub async fn fail_pending(pool: &PgPool, task_id: Uuid, error: &str) -> Result Result { Ok(rows.rows_affected()) } -/// Delete the cancels nobody will take: pending for longer than a claim's -/// duration while nothing drives their execution any more (the drive ended, -/// or its worker went away, before the worker's cancel wait took them). -/// The execution's own terminal row is the cancel's record; the task was -/// only its delivery. Returns how many. +/// Delete the cancels nothing will ask for any more: pending for longer +/// than a claim's duration while nothing drives their execution (the drive +/// ended, cancelled or not, or its worker went away). The execution's own +/// terminal row is the cancel's record; the task was only its delivery. +/// Returns how many. pub async fn drop_stale_cancels(pool: &PgPool) -> Result { let now = unix_now(); Ok(sqlx::query( r#"DELETE FROM task c - WHERE c.kind = 'cancel_execution' AND c.status = 'pending' + WHERE c.status = 'pending' AND c.target = 'worker' AND c.project_id IS NOT NULL + AND c.kind = 'cancel_execution' AND c.created_at_unix < $1 - $2 AND NOT EXISTS (SELECT 1 FROM task d WHERE d.execution_id = c.execution_id AND d.kind IN ('execute', 'resume') diff --git a/crates/weft-task-store/src/terminal.rs b/crates/weft-task-store/src/terminal.rs index e74ca5ad..2ca7f75d 100644 --- a/crates/weft-task-store/src/terminal.rs +++ b/crates/weft-task-store/src/terminal.rs @@ -1,11 +1,12 @@ //! Waking whoever waits on a task the moment it finishes. //! //! Every write that makes a task terminal (`complete`, `fail`, -//! `fail_pending`) sends `pg_notify(TERMINAL_CHANNEL, )` in the -//! same statement, so the notification goes out exactly when the row -//! commits. The process's [`PgSignalWatch`] hears it and the waiter reads -//! the row, so a wait ends when the task does, instead of on the next -//! tick of a poll. +//! `fail_pending`) announces it on [`TERMINAL_CHANNEL`] with the task's id +//! in the same statement, through the announcement outbox +//! (`crate::announce`), sent once the row commits and its writer pokes +//! the flusher. The process's [`PgSignalWatch`] hears it and the waiter +//! reads the row, so a wait ends when the task does, instead of on the +//! next tick of a poll. use std::time::Duration; @@ -15,8 +16,9 @@ use uuid::Uuid; use crate::pg_signal::PgSignalWatch; -/// The channel a terminal task write notifies on, with the task's id as -/// the payload. +/// The channel a terminal task write is announced on, with the task's id +/// as the payload, through the announcement outbox (`crate::announce`): +/// sent once the write commits and its writer pokes the flusher. pub const TERMINAL_CHANNEL: &str = "weft_task_terminal"; /// Wait for `task_id` to finish, or for `timeout` to pass, and hand back diff --git a/crates/weft-task-store/src/traits.rs b/crates/weft-task-store/src/traits.rs index d00e5139..9b1a67e1 100644 --- a/crates/weft-task-store/src/traits.rs +++ b/crates/weft-task-store/src/traits.rs @@ -56,10 +56,10 @@ pub trait TaskStoreClient: Send + Sync { async fn fail(&self, task_id: Uuid, replica: &str, error: String) -> Result<()>; /// The cancels asked for any of `execution_ids` of `project_id` (the - /// executions the asking worker drives), each taken as it is - /// answered; when there are none, hold for up to `wait` for one. An - /// empty answer once `wait` passed. - async fn wait_cancels(&self, project_id: Uuid, execution_ids: Vec, wait: Duration) -> Result>; + /// executions the asking worker drives); asking only reads, so asking + /// again finds them again. A worker asks when it hears one announced + /// ([`crate::tasks::CANCEL_CHANNEL`]) and when it may have missed some. + async fn cancels_asked(&self, project_id: Uuid, execution_ids: Vec) -> Result>; } // ---------- Postgres impls ---------- @@ -121,23 +121,8 @@ impl TaskStoreClient for PostgresTaskStoreClient { crate::tasks::fail(&self.pool, task_id, replica, error).await } - async fn wait_cancels(&self, project_id: Uuid, execution_ids: Vec, wait: Duration) -> Result> { - let deadline = tokio::time::Instant::now() + wait; - // Subscribed before the first look, so a cancel that lands - // between an empty look and the wait still wakes it: a cancel - // task is a worker task of the project, announced on its ready - // payload like any other. - let mut signals = self.signals.subscribe(); - let ready = crate::tasks::ready_payload(crate::tasks::TaskTarget::Worker, Some(project_id)); - loop { - let taken = crate::tasks::take_cancels(&self.pool, project_id, &execution_ids).await?; - if !taken.is_empty() { - return Ok(taken); - } - if !signals.woken_before(deadline, |c, p| c == crate::tasks::TASK_READY_CHANNEL && p == ready).await? { - return Ok(Vec::new()); - } - } + async fn cancels_asked(&self, project_id: Uuid, execution_ids: Vec) -> Result> { + crate::tasks::cancels_asked(&self.pool, project_id, &execution_ids).await } } diff --git a/crates/weft-task-store/tests/schema_guard.rs b/crates/weft-task-store/tests/schema_guard.rs index b7afce26..f1ec2908 100644 --- a/crates/weft-task-store/tests/schema_guard.rs +++ b/crates/weft-task-store/tests/schema_guard.rs @@ -56,8 +56,8 @@ async fn fresh_apply_stamps_and_reruns_are_idempotent(pool: PgPool) { .unwrap(); assert_eq!(fp, fp2); - // A matching stamp still re-runs the DDL, so a manually dropped table - // is repaired on the next boot. + // A matching stamp with a table gone builds the group again, so a + // manually dropped table is repaired on the next boot. sqlx::raw_sql("DROP TABLE guard_probe").execute(&pool).await.unwrap(); apply(&pool, &[&GUARDED], NONE).await.expect("repair apply"); sqlx::query("SELECT 1 FROM guard_probe") diff --git a/crates/weft-task-store/tests/signals.rs b/crates/weft-task-store/tests/signals.rs index a017b8ec..aada9627 100644 --- a/crates/weft-task-store/tests/signals.rs +++ b/crates/weft-task-store/tests/signals.rs @@ -112,10 +112,10 @@ async fn a_task_is_announced_exactly_when_it_becomes_claimable(pool: PgPool) { let dispatcher = tasks::enqueue(&pool, task(TaskTarget::Dispatcher, "d")).await.unwrap(); tasks::enqueue_dedup(&pool, task(TaskTarget::Worker, "w")).await.unwrap(); - assert_eq!( - on(&drain(&mut heard).await, TASK_READY_CHANNEL), - vec!["dispatcher".to_string(), format!("worker:{PROJECT}")], - ); + // Sent in one batch or two (`announce`), in no promised order. + let mut ready = on(&drain(&mut heard).await, TASK_READY_CHANNEL); + ready.sort(); + assert_eq!(ready, vec!["dispatcher".to_string(), format!("worker:{PROJECT}")]); claim_one(&pool, "disp-1").await.unwrap().expect("claimed"); tasks::heartbeat(&pool, dispatcher, "disp-1").await.unwrap(); @@ -143,56 +143,45 @@ fn cancel(execution_id: &str) -> tasks::NewTask { } } -/// A worker's cancel wait held open ends the moment a cancel for one of -/// its executions is enqueued, and comes back empty at its deadline when none -/// is. +/// A cancel is announced to the workers of its project with the +/// execution it stops, never as work to claim. #[sqlx::test] -async fn a_held_cancel_wait_ends_when_a_cancel_arrives_or_at_its_deadline(pool: PgPool) { +async fn a_cancel_is_announced_with_its_execution(pool: PgPool) { setup(&pool).await; - let client = PostgresTaskStoreClient::new(pool.clone(), signals(&pool).await).expect("client"); - let execution_ids = vec!["c1".to_string()]; - - let started = tokio::time::Instant::now(); - assert!(client.wait_cancels(PROJECT, execution_ids.clone(), Duration::from_millis(500)).await.unwrap().is_empty()); - assert!(started.elapsed() >= Duration::from_millis(500)); - - let enqueuer = { - let pool = pool.clone(); - tokio::spawn(async move { - tokio::time::sleep(Duration::from_millis(300)).await; - tasks::enqueue_dedup(&pool, cancel("c1")).await.unwrap(); - }) - }; - let started = tokio::time::Instant::now(); - let taken = client.wait_cancels(PROJECT, execution_ids, Duration::from_secs(20)).await.unwrap(); - enqueuer.await.unwrap(); - assert_eq!(taken.len(), 1); - assert!(started.elapsed() < Duration::from_secs(10), "woken, not timed out: {:?}", started.elapsed()); + let watch = signals(&pool).await; + let mut heard = watch.subscribe(); + let client = PostgresTaskStoreClient::new(pool.clone(), watch.clone()).expect("client"); + + tasks::enqueue_dedup(&pool, cancel("c1")).await.unwrap(); + let heard = drain(&mut heard).await; + assert_eq!(on(&heard, tasks::CANCEL_CHANNEL), vec![tasks::cancel_payload(PROJECT, "c1")]); + assert!(on(&heard, TASK_READY_CHANNEL).is_empty(), "a cancel is no work for a picker"); + assert_eq!(tasks::parse_cancel_payload(&tasks::cancel_payload(PROJECT, "c1")), Some((PROJECT, "c1"))); + + assert!(client.cancels_asked(PROJECT, vec!["other".into()]).await.unwrap().is_empty()); + assert_eq!(client.cancels_asked(PROJECT, vec!["c1".into()]).await.unwrap().len(), 1); } -/// The subscription is taken before the first look, so a cancel that -/// lands anywhere around it (before it, during it, just after the empty -/// answer) still ends the hold. Raced many times with the enqueue at -/// shifting offsets, since a subscribe-after-look window is only a few -/// microseconds wide. +/// What a write leaves in the outbox goes out once it commits and its +/// writer pokes the flusher, each announcement once however many times it +/// was left; what a rolled-back write left never does. #[sqlx::test] -async fn a_cancel_landing_around_the_wait_is_never_missed(pool: PgPool) { +async fn an_announcement_goes_out_once_its_write_commits(pool: PgPool) { setup(&pool).await; - let client = std::sync::Arc::new(PostgresTaskStoreClient::new(pool.clone(), signals(&pool).await).expect("client")); - for round in 0..40u64 { - let execution_id = format!("race-{round}"); - let waiter = { - let client = client.clone(); - let execution_ids = vec![execution_id.clone()]; - tokio::spawn(async move { client.wait_cancels(PROJECT, execution_ids, Duration::from_secs(20)).await }) - }; - tokio::time::sleep(Duration::from_micros(round * 150)).await; - tasks::enqueue_dedup(&pool, cancel(&execution_id)).await.unwrap(); - let taken = tokio::time::timeout(Duration::from_secs(10), waiter) - .await - .unwrap_or_else(|_| panic!("round {round}: the wait missed its cancel")) - .unwrap() - .unwrap(); - assert_eq!(taken.len(), 1, "round {round}"); + let watch = signals(&pool).await; + let mut heard = watch.subscribe(); + + let mut tx = pool.begin().await.unwrap(); + sqlx::query("SELECT weft_announce($1, 'rolled back')").bind(TASK_READY_CHANNEL).execute(&mut *tx).await.unwrap(); + tx.rollback().await.unwrap(); + let mut tx = pool.begin().await.unwrap(); + for _ in 0..3 { + sqlx::query("SELECT weft_announce($1, 'committed')").bind(TASK_READY_CHANNEL).execute(&mut *tx).await.unwrap(); } + tx.commit().await.unwrap(); + assert!(on(&drain(&mut heard).await, TASK_READY_CHANNEL).is_empty(), "nothing goes out before the flusher is poked"); + weft_task_store::announce::committed(&pool); + assert_eq!(on(&drain(&mut heard).await, TASK_READY_CHANNEL), vec!["committed".to_string()]); + let left: i64 = sqlx::query_scalar("SELECT count(*) FROM weft_announcement").fetch_one(&pool).await.unwrap(); + assert_eq!(left, 0, "the outbox is empty once sent"); } diff --git a/crates/weft-task-store/tests/support/mod.rs b/crates/weft-task-store/tests/support/mod.rs index b056978c..4a579715 100644 --- a/crates/weft-task-store/tests/support/mod.rs +++ b/crates/weft-task-store/tests/support/mod.rs @@ -11,7 +11,7 @@ use weft_task_store::tasks; /// Every channel this crate's writes notify on. (Each test file /// compiles this module; the ones that never wait leave these unused.) #[allow(dead_code)] -pub const CHANNELS: &[&str] = &[tasks::TASK_READY_CHANNEL, weft_task_store::terminal::TERMINAL_CHANNEL]; +pub const CHANNELS: &[&str] = &[tasks::TASK_READY_CHANNEL, tasks::CANCEL_CHANNEL, weft_task_store::terminal::TERMINAL_CHANNEL]; /// A signal watch on the test database, listening on [`CHANNELS`]. #[allow(dead_code)] @@ -21,7 +21,7 @@ pub async fn signals(pool: &PgPool) -> Arc { /// Apply the task and worker-process schema to a fresh test database. pub async fn setup(pool: &PgPool) { - weft_task_store::apply_groups(pool, &[&tasks::GROUP]) + weft_task_store::apply_groups(pool, &[&weft_task_store::announce::GROUP, &tasks::GROUP]) .await .expect("tasks schema"); // `tasks::GROUP` creates a trigger ON `task` that stamps diff --git a/crates/weft-task-store/tests/worker_claims.rs b/crates/weft-task-store/tests/worker_claims.rs index 0166ca6a..d59fdc36 100644 --- a/crates/weft-task-store/tests/worker_claims.rs +++ b/crates/weft-task-store/tests/worker_claims.rs @@ -321,26 +321,27 @@ fn cancel(execution_id: &str) -> tasks::NewTask { } } -/// A cancel is taken only by a worker driving its execution, once, and never -/// delivered as work. +/// A cancel answers an ask for its execution, every time it is asked (an +/// answer lost on its way loses nothing), and is never delivered as work. #[sqlx::test] -async fn a_cancel_reaches_the_worker_driving_its_execution_id_once(pool: PgPool) { +async fn a_cancel_answers_every_ask_for_its_execution(pool: PgPool) { setup(&pool).await; let execution_id = Uuid::new_v4().to_string(); tasks::enqueue_dedup(&pool, cancel(&execution_id)).await.unwrap(); assert!(tasks::take_deliveries(&pool, 10).await.unwrap().is_empty(), "a cancel is never delivered"); - assert!(tasks::take_cancels(&pool, PROJECT, &["someone-else".into()]).await.unwrap().is_empty()); - let taken = tasks::take_cancels(&pool, PROJECT, std::slice::from_ref(&execution_id)).await.unwrap(); - assert_eq!(taken.len(), 1); - assert_eq!(taken[0].execution_id, execution_id); - assert!(tasks::take_cancels(&pool, PROJECT, &[execution_id]).await.unwrap().is_empty(), "taken once"); + assert!(tasks::cancels_asked(&pool, PROJECT, &["someone-else".into()]).await.unwrap().is_empty()); + for _ in 0..2 { + let asked = tasks::cancels_asked(&pool, PROJECT, std::slice::from_ref(&execution_id)).await.unwrap(); + assert_eq!(asked.len(), 1); + assert_eq!(asked[0].execution_id, execution_id); + } } -/// A cancel nobody will take (its execution is not driven any more) is -/// dropped once it is older than a claim; one waiting on a live drive is -/// kept. +/// A cancel nothing will ask for (its execution is not driven any more) +/// is dropped once it is older than a claim; one waiting on a live drive +/// is kept. #[sqlx::test] -async fn a_cancel_nobody_will_take_is_dropped(pool: PgPool) { +async fn a_cancel_nothing_will_ask_for_is_dropped(pool: PgPool) { setup(&pool).await; let stale = Uuid::new_v4().to_string(); let driven = Uuid::new_v4().to_string(); @@ -351,7 +352,7 @@ async fn a_cancel_nobody_will_take_is_dropped(pool: PgPool) { tasks::enqueue_dedup(&pool, cancel(&driven)).await.unwrap(); sqlx::query("UPDATE task SET created_at_unix = 0 WHERE kind = 'cancel_execution'").execute(&pool).await.unwrap(); assert_eq!(tasks::drop_stale_cancels(&pool).await.unwrap(), 1); - assert_eq!(tasks::take_cancels(&pool, PROJECT, &[driven]).await.unwrap().len(), 1, "the driven execution's cancel stays"); + assert_eq!(tasks::cancels_asked(&pool, PROJECT, &[driven]).await.unwrap().len(), 1, "the driven execution's cancel stays"); } // ----- orphans ------------------------------------------------------------- diff --git a/deploy/terraform/gcp/machines.tf b/deploy/terraform/gcp/machines.tf new file mode 100644 index 00000000..ceab73e1 --- /dev/null +++ b/deploy/terraform/gcp/machines.tf @@ -0,0 +1,59 @@ +# A machine running a unit of a project's infra says itself when how its +# unit stands changes (its agent asks the broker for a look). One thing +# it cannot say is that the whole machine went away: Compute Engine +# stopped it, a host failed, a preemptible one was taken back. Compute +# Engine writes each of those to the project's audit logs, and this +# carries them to the supervisor's tick, which looks at the health of +# what it owns. Nothing else wakes the supervisor while infra runs fine. + +resource "google_pubsub_topic" "machine_events" { + name = "${var.name}-machine-events" + depends_on = [google_project_service.apis] +} + +# Every infra machine is named `wi-...` (crates/weft-core/src/infra/resolve.rs, +# resource_base). Another install in the same project shares the prefix, +# which costs its supervisor one look that finds nothing. +# SYNC: the `wi-` prefix <-> crates/weft-core/src/infra/resolve.rs (NodeRef::resource_base) +resource "google_logging_project_sink" "machine_events" { + name = "${var.name}-machine-events" + destination = "pubsub.googleapis.com/${google_pubsub_topic.machine_events.id}" + filter = <<-EOT + resource.type="gce_instance" + protoPayload.resourceName:"/instances/wi-" + protoPayload.methodName=("compute.instances.preempted" OR "compute.instances.hostError" OR "compute.instances.guestTerminate" OR "compute.instances.automaticRestart" OR "v1.compute.instances.stop" OR "v1.compute.instances.suspend" OR "v1.compute.instances.delete" OR "v1.compute.instances.reset") + EOT + unique_writer_identity = true +} + +resource "google_pubsub_topic_iam_member" "machine_events_published_by_the_sink" { + topic = google_pubsub_topic.machine_events.id + role = "roles/pubsub.publisher" + member = google_logging_project_sink.machine_events.writer_identity +} + +# The push presents the core account's identity, as the build +# notifications do (builds.tf grants Pub/Sub that). +# SYNC: the tick path <-> crates/weft-platform-traits/src/roles.rs (TICK_PATH) +resource "google_pubsub_subscription" "machine_events_wake_the_supervisor" { + name = "${var.name}-machine-events-wake-the-supervisor" + topic = google_pubsub_topic.machine_events.id + + push_config { + push_endpoint = "${local.role_urls["supervisor"]}/_weft/tick" + oidc_token { + service_account_email = google_service_account.core.email + audience = local.role_urls["supervisor"] + } + } + + # A look that cannot be delivered soon is worth nothing later: the next + # event, or the next command, brings a fresh one. + message_retention_duration = "600s" + ack_deadline_seconds = 600 + # Weeks without a machine event are normal. + expiration_policy { + ttl = "" + } + depends_on = [google_service_account_iam_member.pubsub_signs_as_core] +} diff --git a/deploy/terraform/gcp/versions.tf b/deploy/terraform/gcp/versions.tf index 2763182e..53b1ea9b 100644 --- a/deploy/terraform/gcp/versions.tf +++ b/deploy/terraform/gcp/versions.tf @@ -42,6 +42,8 @@ resource "google_project_service" "apis" { "compute.googleapis.com", "iam.googleapis.com", "iamcredentials.googleapis.com", + # carrying infra machine events to the supervisor (machines.tf) + "logging.googleapis.com", "pubsub.googleapis.com", "run.googleapis.com", "secretmanager.googleapis.com", diff --git a/docs/src/running/architecture.md b/docs/src/running/architecture.md index 382da1c7..1fe0a9bf 100644 --- a/docs/src/running/architecture.md +++ b/docs/src/running/architecture.md @@ -80,9 +80,14 @@ runs from an image named after a hash of its contents, so two projects that comp thing share an image. On your machine a worker is a Docker container, one per program and image, -started when a run needs it and stopped after five minutes with nothing to do. -On a cloud install it is a Cloud Run service, which scales to zero between -calls and out when calls pile up. +started when a run needs it and stopped after thirty seconds with nothing to +do (`workerIdleStopSeconds` in the install's `config.json` changes that). On a +cloud install it is a Cloud Run service, which scales to zero between calls +and out when calls pile up; Cloud Run decides when an idle copy goes, and an +idle copy costs nothing there unless the project keeps its CPU on between +calls (`weft workers set --cpu-always-allocated true`). A worker whose memory is nearly full turns new +work away (`503`, with a `Retry-After`) until its runs end, rather than taking +on a run that would bring every run on it down. A worker keeps, between runs, what its runs read of the program's infrastructure and connections: where each piece answers, the connections it @@ -130,6 +135,16 @@ that and starts at once, on your machine and on a cloud install alike. On a cloud install, where the dispatcher can be at zero, whichever part of weft wrote the work also calls it to start one. +The writes every run makes (its birth, its journal rows, its task's end) are +announced in batches. Postgres hands announcements out in the order their +writes committed, so a write that announces itself makes every other one +wait its turn to commit; with every run announcing several, a busy install +spent most of its time with writes queued on that one turn. Instead each of +those writes leaves its announcement in a table as part of the write, and +right after it commits the process that wrote sends every announcement +waiting there in one go: the turn is taken once per batch rather than once +per write. + ## The listener One listener serves every project on the install. Each kind of trigger says @@ -179,6 +194,16 @@ A supervisor can still die between changing something and recording that it did. The next one works out what to do from what is actually running, rather than trusting the record. +On your machine the supervisor looks at the health of what it runs every +thirty seconds. On a cloud install nothing looks while everything is fine, so +infrastructure that runs fine costs no look at all and lets the rest of the +install sleep: the agent on each machine watches its unit right there, for +free, and asks for a look when how the unit stands changes (a container +stopped, a check started failing or passing again), and Compute Engine's own +record of a machine stopping or failing asks too. Once something is not fine, +the supervisor keeps looking until it is settled, which is what its flaky and +recovery windows need. + ## The broker, and who may talk to what Only the dispatcher and the broker hold a database connection; the listener, @@ -186,18 +211,23 @@ the supervisor and every worker go through the broker. Each of those processes keeps one WebSocket open to the broker, and every call it makes rides that one connection, numbered and answered in whatever order -they finish. A call the broker holds until something happens (a cancel, new -journal rows) costs nothing while it waits, so a worker takes one seat at the -broker however many calls its runs hold open. The broker also pushes the -changes a worker keeps a copy of down the same connection (see [the -worker](#the-worker)). When the connection breaks, it comes back by itself: a -call not yet sent goes out on the new one if it is back within the call's wait. +they finish. A call the broker holds until something happens (new journal +rows) costs nothing while it waits, so a worker takes one seat at the broker +however many calls its runs hold open, and a call whose caller stops waiting +is dropped at the broker too. The broker also pushes down the same connection +the changes a worker keeps a copy of (see [the worker](#the-worker)), and the +cancel of a run the worker drives. A connection with nothing to do for a +minute closes, and opens again on the next call, so an idle process holds +nothing open at the broker; a worker driving runs keeps it open, to hear +their cancels. When the connection breaks, it comes back by itself: a call +not yet sent goes out on the new one if it is back within the call's wait. Past that, or when a sent call got no answer, two kinds of call are made again because doing so is safe: a journal write that never reached the broker, and -a read of a run's history. Any other call fails the way a request whose -connection reset does, since it may have landed. Uploads and -downloads of files stay ordinary requests, so a big file never holds up the -calls behind it. +a read of a run's history. A call the broker says it could not even start +(every one of its database connections busy) is sent again too, since nothing +of it ran. Any other call fails the way a request whose connection reset +does, since it may have landed. Uploads and downloads of files stay ordinary +requests, so a big file never holds up the calls behind it. The broker checks every request. The caller proves who it is with a token its platform gave it: on your machine, one the runtime signed when it started the diff --git a/docs/src/running/cli.md b/docs/src/running/cli.md index 24a56d74..5499f366 100644 --- a/docs/src/running/cli.md +++ b/docs/src/running/cli.md @@ -87,7 +87,7 @@ reads the file again. | Command | What it does | |---|---| -| `weft executions` | Past runs, newest first. `--limit` (50), `--offset`, `--project`, `--phase`, `--node`, `--since 2h`, `--status`, `--instance`, `--tag`. A run of a `Route` with `recorded: false` shows only if it failed | +| `weft executions` | Past runs, newest first. `--limit` (50), `--offset`, `--project`, `--phase`, `--node` (the node that started the run), `--since 2h`, `--status`, `--instance`, `--tag`. `--through ` keeps the runs in which that node fired, wherever it sits. `--search ` keeps the finished runs that carried every one of those words somewhere in what they recorded (the trigger's input, what a node sent on, an error, a log line): an email, an order id, a phrase in quotes. A run of a `Route` with `recorded: false` shows only if it failed | | `weft events ` | One run's events in order. `--node`, `--kind`, `--full` for whole values. If you want one time round a loop, `--iteration 3` keeps the fourth (they count from 0), and `3.0` the first time round a loop inside it | | `weft logs []` | What the nodes wrote, plus every failure. A run that wrote nothing lists what it skipped and why (under `skipped` with `--json`). `--limit` | | `weft status` | The cwd project: registration, listener, every infra node the program declares (one never started says `not started`, a `@per_instance` one how many instances have a copy), each trigger and the version of the source it fires (with the events an instance's trigger holds until a field is filled), recent runs, what drifted, and what you can do next. While the install builds, it names each image building and where its log is; while infra starts, how long it has been going and what it waits on | @@ -124,8 +124,8 @@ answers "has my trigger fired since I changed it". | Flag | What it does | |---|---| -| `--mode wipe\|hibernate\|park` | What happens to work in flight. `wipe` drops it all, the other two keep it | -| `--grace ` | How long hibernate accepts late answers. 15 by default | +| `--mode wipe\|hibernate\|park` | What happens to work that arrives while the triggers are down. `park` keeps all of it and runs it once they are back, `hibernate` does that for a grace window, `wipe` drops it ([the full rules](lifecycles.md#choosing-what-happens-to-work-in-flight)) | +| `--grace ` | How long hibernate keeps taking work. 15 by default | | `--running-policy cancel\|wait` | Cancel the running executions, or wait for them. `cancel` unless you say otherwise; `wipe` refuses `wait`. On `weft infra stop` and `terminate` it works on an inactive project too: no triggers to take down, but a run may still be using the infra | | `--drain-timeout ` | Cap on a wait, so only beside `--running-policy wait`; passed with cancel it is refused. 60 by default. What is still running at the cap is cancelled | @@ -142,10 +142,10 @@ Every trigger verb (`activate`, `deactivate`, `resync`, `bake`, | Command | What it does | |---|---| -| `weft infra start` | Brings up whatever is down | +| `weft infra start` | Brings up whatever is down, and turns back on the triggers that came down with it (never one you switched off yourself) | | `weft infra stop` | Scales to nothing, keeping the disk. The project's running executions are cancelled first unless you pass `--running-policy wait`, because they may be using this infra | | `weft infra terminate` | Deletes it, disk included, unless the node asked for the disk to be kept. Asks first, and off a terminal or with `--json` it needs `--yes`. The same running-policy rule as stop | -| `weft infra upgrade` | Rebuilds against your current source. Leaves the project deactivated. When a build changes an infra node's image, every command that builds says so: a copy started from then on gets the new image, and one already running keeps its own until this | +| `weft infra upgrade` | Rebuilds against your current source; the triggers it took down come back once the new copies are up. When a build changes an infra node's image, every command that builds says so: a copy started from then on gets the new image, and one already running keeps its own until this | | `weft infra status` | Where each copy that exists stands, with its address. A start or stop on its way shows as `provisioning` or `stopping` at once | | `weft infra list-doors` | Which pieces you can reach from this machine, and at what address | | `weft infra logs []` | What the containers printed. `--tail` (200), `-f` | diff --git a/docs/src/running/cloud.md b/docs/src/running/cloud.md index ce81448a..c129a26f 100644 --- a/docs/src/running/cloud.md +++ b/docs/src/running/cloud.md @@ -55,9 +55,16 @@ Google, so that pair is the one place both free tiers sit side by side. PROJECT=my-project REGION=us-west1 FORK=me/weft NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services just -# enabled, with the account just made), so each IAM step below is retried -# until it holds. -retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } +# enabled, with the account just made), so each IAM step below is tried +# again, quietly, for up to three minutes; a last try shows its own error. +retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" +} gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ @@ -188,7 +195,7 @@ few hours that wakes each part for a moment. | The holders | a Cloud Run worker pool for the triggers that keep a connection open between events (a stream, a socket, an event subscription that dials out). weft runs one holder per 200 such triggers and none when there are none | | Your database | everything weft keeps, at the address you gave it | | Cloud Build | builds your programs' images when you deploy | -| Pub/Sub | the `cloud-builds` topic, on which Cloud Build announces each build's end, so the dispatcher hears it at once | +| Pub/Sub | the `cloud-builds` topic, on which Cloud Build announces each build's end, so the dispatcher hears it at once, and one carrying Compute Engine's record of an infrastructure machine stopping or failing to the supervisor | | Cloud Tasks | every timer, schedule and poll your programs set, and the wakes weft schedules for itself to check on its own pending work | | Compute Engine | one machine per infrastructure unit your programs start (a database, a GPU model) | | A storage bucket | weft's files, reached as weft's own service account: no key is made for it, so an organization that forbids service-account keys runs weft as is | @@ -318,9 +325,10 @@ weft resync --on prod --mode park ``` On prod, pick `park` or `hibernate`, because the calls arriving while the -triggers are down can be real people's: `park` holds them and runs them once -the triggers are back, `hibernate` keeps them for a grace window (`--grace`, -in minutes). `wipe` drops them and cancels the work waiting on the triggers, +triggers are down can be real people's: `park` holds them (a caller on a +route stays on the line) and runs them on the new version once the triggers +are back, `hibernate` does the same for a grace window (`--grace`, in +minutes). `wipe` drops them and cancels the work waiting on the triggers, which is fine on a dev install. Only the change is uploaded, and only the image it touches is rebuilt. If diff --git a/docs/src/running/lifecycles.md b/docs/src/running/lifecycles.md index ca84adeb..a02cffd9 100644 --- a/docs/src/running/lifecycles.md +++ b/docs/src/running/lifecycles.md @@ -20,7 +20,7 @@ Nothing starts it for you. Not `weft run`, not `weft build`, not activation. | `weft infra start` | Brings up whatever is down and leaves whatever is already up alone | | `weft infra stop` | Scales to nothing and **keeps the disk**. Start it again and your rows are there | | `weft infra terminate` | Deletes it, **disk included**, unless the step asked for the disk to be kept | -| `weft infra upgrade` | Rebuilds against your current source. Leaves the project deactivated afterwards, so you choose when the new version takes traffic | +| `weft infra upgrade` | Rebuilds against your current source. The triggers it took down come back once the new infrastructure is up | | `weft infra status` | Where each piece stands, with its address | | `weft infra doors` | The addresses this project's infra answers on | | `weft infra logs ` | What the containers actually printed | @@ -91,10 +91,12 @@ listens and none is part way through turning on or off, and `weft status` lists each trigger's own state under `triggers:`. If you stopped infrastructure that some triggers read, those triggers came -down with it and the rest stayed on. After `weft infra start`, a plain `weft -activate` turns back on the ones that are off and leaves the others as they -are. A route whose trigger is off answers its callers 503 at once, saying it -is switched off: a caller is never held while it waits. +down with it and the rest stayed on. When `weft infra start` (or the +program's own `ctx.infra(..).start()`) has the infrastructure up again, the +triggers that came down with it come back on by themselves; one you switched +off yourself since stays off. A plain `weft activate` turns back on whatever +is still off. While a route's trigger is parked, its callers are kept on the +line and served once it is back. A trigger that reads an instance's copy or an instance's value (`@per_instance`, `@instance_filled`) exists once per instance, and its copies @@ -114,13 +116,17 @@ Prepare it again: `weft bake` does it without listening, `weft activate` does it ### Choosing what happens to work in flight -`weft deactivate` and `weft resync` both ask: +`weft deactivate` and `weft resync` both ask. If you want nothing lost while +the triggers are down, park them: -| Mode | What it does | -|---|---| -| `wipe` | Drops every waiting signal and cancels suspended runs. A clean slate, and what you want while building | -| `hibernate` | Keeps the signals, hides pending tasks from the browser extension, parks late answers | -| `park` | Keeps the signals visible and queues new answers until you reactivate | +| Mode | Work that arrives meanwhile | What a person sees | +|---|---|---| +| `park` | Nothing is dropped. A call from outside, an answer to a waiting run, and every fire a trigger makes by itself (a schedule's tick, a message on a connection it holds) wait, and run once the triggers are back, on the version they come back with. Someone calling a route is kept on the line until then | Questions stay listed for the people who answer them, and an answer sent now waits too | +| `hibernate` | The same as `park` for the grace window (`--grace`, in minutes). After it, nothing new is taken and the triggers stop listening | Questions are hidden for the whole time; someone who already has one open can still send their answer within the window | +| `wipe` | Refused. Everything waiting is dropped and the runs waiting on the triggers are cancelled: a clean slate, and what you want while building | Questions are deleted | + +A schedule parked for two hours has two hours of ticks waiting when it comes +back; the reactivate choice below decides whether they run. With no terminal attached, or with `--json`, the default is `wipe`. Running executions are cancelled unless you pass `--running-policy wait`, which lets diff --git a/extension-vscode/package.json b/extension-vscode/package.json index 0c6dc3ba..1b0f8f6b 100644 --- a/extension-vscode/package.json +++ b/extension-vscode/package.json @@ -2,7 +2,7 @@ "name": "weft-vscode", "displayName": "Weft", "description": "Write Weft programs and watch them run: a live, editable graph of every .weft file, with diagnostics, runs and executions in the editor.", - "version": "0.2.259", + "version": "0.2.260", "publisher": "weavemind", "license": "SEE LICENSE IN LICENSE", "repository": { @@ -180,6 +180,24 @@ "category": "Weft", "icon": "$(list-tree)" }, + { + "command": "weft.searchRuns", + "title": "Search runs", + "category": "Weft", + "icon": "$(search)" + }, + { + "command": "weft.clearRunSearch", + "title": "Show every run", + "category": "Weft", + "icon": "$(close)" + }, + { + "command": "weft.showRunsOn", + "title": "Show the runs on…", + "category": "Weft", + "icon": "$(cloud)" + }, { "command": "weft.branchHere", "title": "Branch here", @@ -282,6 +300,16 @@ "when": "view == weft.executions", "group": "navigation@3" }, + { + "command": "weft.searchRuns", + "when": "view == weft.executions", + "group": "navigation@3" + }, + { + "command": "weft.showRunsOn", + "when": "view == weft.executions", + "group": "navigation@3" + }, { "command": "weft.checkpointLabel", "when": "view == weft.executions", @@ -343,6 +371,11 @@ "command": "weft.deleteExecution", "when": "view == weft.executions && viewItem =~ /^weftRun-/", "group": "inline@2" + }, + { + "command": "weft.clearRunSearch", + "when": "view == weft.executions && viewItem == weftRunSearch", + "group": "inline" } ] }, diff --git a/extension-vscode/src/extension.ts b/extension-vscode/src/extension.ts index 05bc048b..33da33c6 100644 --- a/extension-vscode/src/extension.ts +++ b/extension-vscode/src/extension.ts @@ -31,6 +31,7 @@ import { fetchRunningSource, installAccess, listTargets, onArgs, type InstallTar import { ProjectsProvider, ProjectNode, type WeftProject } from './sidebar/projects'; import { ExecutionsProvider, ExecutionNode, RunNode, VersionNode, type ExecutionSummary } from './sidebar/executions'; +import { parseRunSearch } from './sidebar/runSearch'; import { runWeftJson, WeftCliError } from './cli'; import { ExecutionFollower } from './execFollower'; import { AutoFollowController } from './autoFollow'; @@ -438,6 +439,57 @@ export function activate(context: vscode.ExtensionContext) { } } + /// Ask what to search the runs for, and show the runs it finds (see + /// `sidebar/runSearch` for what it reads). An empty answer shows every + /// run again; one that does not read is said, and asked again. + async function searchRuns(): Promise { + let value = executionsProvider.currentSearch() ?? ''; + for (;;) { + const text = await vscode.window.showInputBox({ + title: `Search the runs on ${pinnedProject ? installOf(pinnedProject.id) : LOCAL_INSTALL}`, + value, + prompt: 'Words a run carried (an email, an id, "a phrase"), and filters: status:failed route: node: instance: tag: since:2h', + }); + if (text === undefined) return; + if (!text.trim()) { + executionsProvider.setSearch(undefined); + return; + } + const read = parseRunSearch(text, Math.floor(Date.now() / 1000)); + if ('error' in read) { + void vscode.window.showWarningMessage(read.error); + value = text; + continue; + } + executionsProvider.setSearch({ text: text.trim(), params: read.params }); + return; + } + } + + /// Pick the install whose runs the list shows (and the graph follows + /// them on): the local one, or a target the project's `weft.toml` names. + async function showRunsOn(): Promise { + const project = pinnedProject; + if (!project) { + void vscode.window.showInformationMessage('Open a project to see its runs.'); + return; + } + let targets: InstallTarget[]; + try { + targets = await listTargets(project.rootPath); + } catch (err) { + void vscode.window.showErrorMessage(`could not list the project's installs: ${err instanceof Error ? err.message : String(err)}`); + return; + } + const current = installOf(project.id); + const names = [LOCAL_INSTALL, ...targets.map((t) => t.name).filter((n) => n !== LOCAL_INSTALL)]; + const picked = await vscode.window.showQuickPick( + names.map((name) => ({ label: name, description: name === current ? 'shown now' : undefined })), + { title: 'Show the runs on' }, + ); + if (picked && picked.label !== current) await switchInstall(picked.label); + } + /// Which version the files on disk are, beside the one the install /// holds, so the graph can say when they differ. async function refreshDiskVersion(project: WeftProject): Promise { @@ -1484,6 +1536,9 @@ export function activate(context: vscode.ExtensionContext) { n instanceof RunNode ? viewExecution(n.summary, n.versionId) : viewExecution('summary' in n ? n.summary : n), ), vscode.commands.registerCommand('weft.toggleExecutionsMode', () => executionsProvider.toggleMode()), + vscode.commands.registerCommand('weft.searchRuns', () => searchRuns()), + vscode.commands.registerCommand('weft.clearRunSearch', () => executionsProvider.setSearch(undefined)), + vscode.commands.registerCommand('weft.showRunsOn', () => showRunsOn()), vscode.commands.registerCommand('weft.branchHere', (n: VersionNode | RunNode) => branchHere(n)), vscode.commands.registerCommand('weft.checkpointLabel', () => checkpointLabel()), vscode.commands.registerCommand('weft.pruneVersion', (n: VersionNode) => pruneVersion(n)), diff --git a/extension-vscode/src/sidebar/executions.ts b/extension-vscode/src/sidebar/executions.ts index 0ddef346..4ba4f716 100644 --- a/extension-vscode/src/sidebar/executions.ts +++ b/extension-vscode/src/sidebar/executions.ts @@ -115,6 +115,9 @@ export class ExecutionsProvider implements vscode.TreeDataProvider } | undefined; /// `installArgs` answers the `--on` the tree's CLI call carries: the /// install the graph shows the pinned project on. @@ -145,6 +148,23 @@ export class ExecutionsProvider implements vscode.TreeDataProvider } | undefined): void { + this.search = search; + this.mode = 'flat'; + this.cache = []; + this.total = 0; + this.loaded = ExecutionsProvider.PAGE_SIZE; + this._onDidChange.fire(); + void this.startRefresh(this.pinnedProject?.id); + } + setFocusedGroup(group: string | null): void { if (this.focusedGroup === group) return; this.focusedGroup = group; @@ -211,16 +231,18 @@ export class ExecutionsProvider implements vscode.TreeDataProvider { - const params = new URLSearchParams({ - limit: String(ExecutionsProvider.PAGE_SIZE), - offset: String(offset), - }); + /** Fetch ONE page (`PAGE_SIZE` rows, newest first, filtered to the + * pinned project and the search), the one after `after` (the last run + * of the page before): each page picks up exactly where the last left + * off, whatever runs start or end meanwhile. Kept to PAGE_SIZE so a + * single request never exceeds the dispatcher's per-request page cap. */ + private async fetchPage(after: ExecutionSummary | undefined, projectId: string | undefined): Promise { + const params = new URLSearchParams({ limit: String(ExecutionsProvider.PAGE_SIZE), ...this.search?.params }); if (projectId) params.set('project_id', projectId); + if (after) { + params.set('before_started', String(after.started_at)); + params.set('before_execution', after.execution_id); + } return this.client.get(`/executions?${params}`); } @@ -310,9 +332,11 @@ export class ExecutionsProvider implements vscode.TreeDataProvider `${e.execution_id}:${e.status}`).join(','), versions, runs, @@ -386,7 +411,7 @@ export class ExecutionsProvider implements vscode.TreeDataProvider { + it('reads every filter and leaves the rest as words', () => { + expect(parseRunSearch('status:failed route:notes.list.door node:hash instance:wa tag:vip since:2h ada@example.com', 10_000)).toEqual({ + params: { + status: 'failed', + entry_node: 'notes.list.door', + node: 'hash', + instance: 'wa', + tag: 'vip', + started_after: String(10_000 - 7200), + search: 'ada@example.com', + }, + }); + }); + + it('keeps a quoted phrase whole, a colon inside it included', () => { + expect(parseRunSearch('"payment declined: card" order 42', 0)).toEqual({ params: { search: '"payment declined: card" order 42' } }); + }); + + it('reads an address with a colon as a word', () => { + expect(parseRunSearch('https://example.com/x', 0)).toEqual({ params: { search: 'https://example.com/x' } }); + }); + + it('says what does not read', () => { + expect(parseRunSearch('status:broken', 0)).toEqual({ + error: 'status:broken is not a status; one of running, waiting_for_input, completed, failed, cancelled', + }); + expect(parseRunSearch('since:yesterday', 0)).toEqual({ error: 'since:yesterday is not a time; write it like 30s, 10m, 2h or 3d' }); + }); + + it('is nothing at all when empty', () => { + expect(parseRunSearch(' ', 0)).toEqual({ params: {} }); + }); +}); diff --git a/extension-vscode/src/sidebar/runSearch.ts b/extension-vscode/src/sidebar/runSearch.ts new file mode 100644 index 00000000..d30258ad --- /dev/null +++ b/extension-vscode/src/sidebar/runSearch.ts @@ -0,0 +1,71 @@ +// What a person types in the runs view's search, read as the listing's +// filters (`GET /executions`). Each `name:value` narrows by one thing a +// run has, and every other word is searched for in what the finished run +// recorded (the trigger's input, what each node sent on, an error, a +// log line), the way `weft executions --search` does: +// +// status:failed route:notes.list.door since:2h ada@example.com +// +// - `status:` running, waiting_for_input, completed, failed, cancelled +// - `route:` (or `trigger:`) the node that started the run +// - `node:` a node that fired in the run, wherever it sits +// - `instance:` the instance the run was for +// - `tag:` a tag the run carries +// - `since:` how long ago at most: 30s, 10m, 2h, 3d +// +// A phrase in quotes is searched as written. + +/** The listing's query parameters a search stands for, or why it does not read. */ +export type RunSearch = { params: Record } | { error: string }; + +// SYNC: STATUSES <-> crates/weft-core/src/program.rs RunStatus, crates/weft-dispatcher/src/journal/postgres.rs list_executions (status clause) +const STATUSES = ['running', 'waiting_for_input', 'completed', 'failed', 'cancelled']; + +const UNITS: Record = { s: 1, m: 60, h: 3600, d: 86400 }; + +/** Read `text` at `nowSecs` (unix seconds). */ +export function parseRunSearch(text: string, nowSecs: number): RunSearch { + const params: Record = {}; + const words: string[] = []; + // A quoted phrase is one token, quotes kept for the words search. + for (const token of text.match(/"[^"]*"|\S+/g) ?? []) { + const named = /^([a-z_]+):(.+)$/.exec(token); + if (!named || token.startsWith('"')) { + words.push(token); + continue; + } + const [, name, value] = named; + switch (name) { + case 'status': + if (!STATUSES.includes(value)) { + return { error: `status:${value} is not a status; one of ${STATUSES.join(', ')}` }; + } + params.status = value; + break; + case 'route': + case 'trigger': + params.entry_node = value; + break; + case 'node': + params.node = value; + break; + case 'instance': + params.instance = value; + break; + case 'tag': + params.tag = value; + break; + case 'since': { + const ago = /^(\d+)([smhd])$/.exec(value); + if (!ago) return { error: `since:${value} is not a time; write it like 30s, 10m, 2h or 3d` }; + params.started_after = String(nowSecs - Number(ago[1]) * UNITS[ago[2]]); + break; + } + default: + // An address with a colon in it (`http://...`) is a word, not a filter. + words.push(token); + } + } + if (words.length) params.search = words.join(' '); + return { params }; +} diff --git a/packages/weft-graph/src/webview/lib/components/project/DeactivationPicker.svelte b/packages/weft-graph/src/webview/lib/components/project/DeactivationPicker.svelte index 59cd9f2a..6bb45b65 100644 --- a/packages/weft-graph/src/webview/lib/components/project/DeactivationPicker.svelte +++ b/packages/weft-graph/src/webview/lib/components/project/DeactivationPicker.svelte @@ -42,18 +42,18 @@ value: 'park', label: 'Park', detail: - 'Park submissions indefinitely. Project visible to consumers; submissions drained on reactivate.', + 'Nothing is dropped: calls, answers and the triggers\' own fires wait, and run on the version they come back with. A caller on a route is kept on the line. Questions stay listed.', }, { value: 'hibernate', label: 'Hibernate', detail: - 'Park submissions for a grace window, then refuse them. Project hidden from consumer enumeration the entire time.', + 'Like park for a grace window, with its questions hidden; after it, nothing new is taken and the triggers stop listening.', }, { value: 'wipe', label: 'Wipe', - detail: 'Drop every signal + cancel suspended runs. Reactivate is a fresh boot.', + detail: 'Drop everything waiting and cancel the runs waiting on the triggers. Reactivating starts fresh.', }, ]; @@ -71,7 +71,7 @@ value: 'wait', label: 'Wait for running executions', detail: - 'New fires park immediately; in-flight runs drain naturally. You can cancel running anytime.', + 'New work waits as the mode says; the runs going now finish first. You can cancel them anytime.', }, ]; diff --git a/scripts/load-route.sh b/scripts/load-route.sh new file mode 100755 index 00000000..5296aa8f --- /dev/null +++ b/scripts/load-route.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# Call one route many times at once and say how it held up: how many calls +# a second it answered, how long they took, and what they answered. +# +# scripts/load-route.sh [calls] [at-once] [method] [body] +# +# scripts/load-route.sh http://127.0.0.1:14111/connect/local/load/ping 1000 50 +# scripts/load-route.sh http://127.0.0.1:14111/connect/local/load/burn 200 20 POST '{"rounds":200000}' +# +# Every call has a 30 second cap, and the whole run is capped at ten +# minutes. A call that hit its cap shows as status 000. The calls are made +# with curl, one process each, so past a few hundred calls a second it is +# this script, not the route, that is the limit. +set -euo pipefail + +url=${1:?"usage: scripts/load-route.sh [calls] [at-once] [method] [body]"} +calls=${2:-1000} +at_once=${3:-50} +method=${4:-GET} +body=${5:-} + +out=$(mktemp) +trap 'rm -f "$out"' EXIT + +args=(-s -o /dev/null -m 30 -X "$method" -w '%{http_code} %{time_total}\n') +if [[ -n "$body" ]]; then + args+=(-H 'content-type: application/json' --data "$body") +fi + +now() { perl -MTime::HiRes=time -e 'printf "%.3f", time'; } +started=$(now) +seq "$calls" | timeout 600 xargs -P "$at_once" -I{} curl "${args[@]}" "$url" >"$out" || true +ended=$(now) + +elapsed=$(echo "$ended - $started" | bc) +n=$(wc -l <"$out" | tr -d ' ') +if [[ "$n" -eq 0 ]]; then + echo "no call answered" + exit 1 +fi +# The times in order, for the median and the 95th of them. +times=$(awk '{ print $2 }' "$out" | sort -n) +nth() { echo "$times" | sed -n "$1p"; } +echo "$n calls in $(printf '%.1f' "$elapsed")s: $(echo "scale=1; $n / $elapsed" | bc) a second" +echo "took: average $(awk '{ s += $2 } END { printf "%.3f", s / NR }' "$out")s, median $(nth $((n / 2 + 1)))s, 95th $(nth $((n * 95 / 100 + 1)))s, slowest $(nth "$n")s" +echo "answered: $(awk '{ print $1 }' "$out" | sort | uniq -c | awk '{ printf "%s x%s ", $2, $1 }')" diff --git a/tangle/claude-code/.claude/agents/deployer.md b/tangle/claude-code/.claude/agents/deployer.md index 9dfd52fe..c7708eb1 100644 --- a/tangle/claude-code/.claude/agents/deployer.md +++ b/tangle/claude-code/.claude/agents/deployer.md @@ -9,7 +9,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -25,7 +25,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/claude-code/.claude/skills/weft-cloud-install/SKILL.md b/tangle/claude-code/.claude/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/claude-code/.claude/skills/weft-cloud-install/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/claude-code/.claude/skills/weft-deploying/SKILL.md b/tangle/claude-code/.claude/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/claude-code/.claude/skills/weft-deploying/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/claude-code/.claude/skills/weft-editor/SKILL.md b/tangle/claude-code/.claude/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/claude-code/.claude/skills/weft-editor/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/claude-code/.claude/skills/weft-running/SKILL.md b/tangle/claude-code/.claude/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/claude-code/.claude/skills/weft-running/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it. diff --git a/tangle/cline/.cline/skills/deployer/SKILL.md b/tangle/cline/.cline/skills/deployer/SKILL.md index 75dc0109..637c599d 100644 --- a/tangle/cline/.cline/skills/deployer/SKILL.md +++ b/tangle/cline/.cline/skills/deployer/SKILL.md @@ -21,7 +21,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -37,7 +37,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/cline/.cline/skills/weft-cloud-install/SKILL.md b/tangle/cline/.cline/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/cline/.cline/skills/weft-cloud-install/SKILL.md +++ b/tangle/cline/.cline/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/cline/.cline/skills/weft-deploying/SKILL.md b/tangle/cline/.cline/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/cline/.cline/skills/weft-deploying/SKILL.md +++ b/tangle/cline/.cline/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/cline/.cline/skills/weft-editor/SKILL.md b/tangle/cline/.cline/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/cline/.cline/skills/weft-editor/SKILL.md +++ b/tangle/cline/.cline/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/cline/.cline/skills/weft-running/SKILL.md b/tangle/cline/.cline/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/cline/.cline/skills/weft-running/SKILL.md +++ b/tangle/cline/.cline/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it. diff --git a/tangle/codex/.agents/skills/weft-cloud-install/SKILL.md b/tangle/codex/.agents/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/codex/.agents/skills/weft-cloud-install/SKILL.md +++ b/tangle/codex/.agents/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/codex/.agents/skills/weft-deploying/SKILL.md b/tangle/codex/.agents/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/codex/.agents/skills/weft-deploying/SKILL.md +++ b/tangle/codex/.agents/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/codex/.agents/skills/weft-editor/SKILL.md b/tangle/codex/.agents/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/codex/.agents/skills/weft-editor/SKILL.md +++ b/tangle/codex/.agents/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/codex/.agents/skills/weft-running/SKILL.md b/tangle/codex/.agents/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/codex/.agents/skills/weft-running/SKILL.md +++ b/tangle/codex/.agents/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it. diff --git a/tangle/codex/.codex/agents/deployer.toml b/tangle/codex/.codex/agents/deployer.toml index cada5b29..7bf8deb6 100644 --- a/tangle/codex/.codex/agents/deployer.toml +++ b/tangle/codex/.codex/agents/deployer.toml @@ -8,7 +8,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -24,7 +24,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/cursor/.cursor/agents/deployer.md b/tangle/cursor/.cursor/agents/deployer.md index 643d9abb..94b3e3b5 100644 --- a/tangle/cursor/.cursor/agents/deployer.md +++ b/tangle/cursor/.cursor/agents/deployer.md @@ -8,7 +8,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -24,7 +24,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/cursor/.cursor/skills/weft-cloud-install/SKILL.md b/tangle/cursor/.cursor/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/cursor/.cursor/skills/weft-cloud-install/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/cursor/.cursor/skills/weft-deploying/SKILL.md b/tangle/cursor/.cursor/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/cursor/.cursor/skills/weft-deploying/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/cursor/.cursor/skills/weft-editor/SKILL.md b/tangle/cursor/.cursor/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/cursor/.cursor/skills/weft-editor/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/cursor/.cursor/skills/weft-running/SKILL.md b/tangle/cursor/.cursor/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/cursor/.cursor/skills/weft-running/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it. diff --git a/tangle/devin-desktop/.windsurf/skills/deployer/SKILL.md b/tangle/devin-desktop/.windsurf/skills/deployer/SKILL.md index e2c7181c..28f92f91 100644 --- a/tangle/devin-desktop/.windsurf/skills/deployer/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/deployer/SKILL.md @@ -21,7 +21,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -37,7 +37,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/devin-desktop/.windsurf/skills/weft-cloud-install/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-cloud-install/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/devin-desktop/.windsurf/skills/weft-deploying/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-deploying/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/devin-desktop/.windsurf/skills/weft-editor/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-editor/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/devin-desktop/.windsurf/skills/weft-running/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-running/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it. diff --git a/tangle/fallback/.agents/skills/deployer/SKILL.md b/tangle/fallback/.agents/skills/deployer/SKILL.md index 55537f21..2010ee03 100644 --- a/tangle/fallback/.agents/skills/deployer/SKILL.md +++ b/tangle/fallback/.agents/skills/deployer/SKILL.md @@ -7,7 +7,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -23,7 +23,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/fallback/.agents/skills/weft-cloud-install/SKILL.md b/tangle/fallback/.agents/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/fallback/.agents/skills/weft-cloud-install/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/fallback/.agents/skills/weft-deploying/SKILL.md b/tangle/fallback/.agents/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/fallback/.agents/skills/weft-deploying/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/fallback/.agents/skills/weft-editor/SKILL.md b/tangle/fallback/.agents/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/fallback/.agents/skills/weft-editor/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/fallback/.agents/skills/weft-running/SKILL.md b/tangle/fallback/.agents/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/fallback/.agents/skills/weft-running/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it. diff --git a/tangle/gemini-cli/.gemini/agents/deployer.md b/tangle/gemini-cli/.gemini/agents/deployer.md index eafcca57..61610a9c 100644 --- a/tangle/gemini-cli/.gemini/agents/deployer.md +++ b/tangle/gemini-cli/.gemini/agents/deployer.md @@ -11,7 +11,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -27,7 +27,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/gemini-cli/.gemini/skills/weft-cloud-install/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-cloud-install/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/gemini-cli/.gemini/skills/weft-deploying/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-deploying/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/gemini-cli/.gemini/skills/weft-editor/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-editor/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/gemini-cli/.gemini/skills/weft-running/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-running/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it. diff --git a/tangle/github-copilot/.github/agents/deployer.agent.md b/tangle/github-copilot/.github/agents/deployer.agent.md index f1084764..c07384b1 100644 --- a/tangle/github-copilot/.github/agents/deployer.agent.md +++ b/tangle/github-copilot/.github/agents/deployer.agent.md @@ -8,7 +8,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -24,7 +24,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/github-copilot/.github/skills/weft-cloud-install/SKILL.md b/tangle/github-copilot/.github/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/github-copilot/.github/skills/weft-cloud-install/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/github-copilot/.github/skills/weft-deploying/SKILL.md b/tangle/github-copilot/.github/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/github-copilot/.github/skills/weft-deploying/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/github-copilot/.github/skills/weft-editor/SKILL.md b/tangle/github-copilot/.github/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/github-copilot/.github/skills/weft-editor/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/github-copilot/.github/skills/weft-running/SKILL.md b/tangle/github-copilot/.github/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/github-copilot/.github/skills/weft-running/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it. diff --git a/tangle/junie/.junie/agents/deployer.md b/tangle/junie/.junie/agents/deployer.md index e873538d..f87d47e8 100644 --- a/tangle/junie/.junie/agents/deployer.md +++ b/tangle/junie/.junie/agents/deployer.md @@ -9,7 +9,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -25,7 +25,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/junie/.junie/skills/weft-cloud-install/SKILL.md b/tangle/junie/.junie/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/junie/.junie/skills/weft-cloud-install/SKILL.md +++ b/tangle/junie/.junie/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/junie/.junie/skills/weft-deploying/SKILL.md b/tangle/junie/.junie/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/junie/.junie/skills/weft-deploying/SKILL.md +++ b/tangle/junie/.junie/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/junie/.junie/skills/weft-editor/SKILL.md b/tangle/junie/.junie/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/junie/.junie/skills/weft-editor/SKILL.md +++ b/tangle/junie/.junie/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/junie/.junie/skills/weft-running/SKILL.md b/tangle/junie/.junie/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/junie/.junie/skills/weft-running/SKILL.md +++ b/tangle/junie/.junie/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it. diff --git a/tangle/kilo-code/.kilo/agent/deployer.md b/tangle/kilo-code/.kilo/agent/deployer.md index 893aacc0..a36b6dab 100644 --- a/tangle/kilo-code/.kilo/agent/deployer.md +++ b/tangle/kilo-code/.kilo/agent/deployer.md @@ -11,7 +11,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -27,7 +27,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/kilo-code/.kilo/skills/weft-cloud-install/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/kilo-code/.kilo/skills/weft-cloud-install/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/kilo-code/.kilo/skills/weft-deploying/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/kilo-code/.kilo/skills/weft-deploying/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/kilo-code/.kilo/skills/weft-editor/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/kilo-code/.kilo/skills/weft-editor/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/kilo-code/.kilo/skills/weft-running/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/kilo-code/.kilo/skills/weft-running/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it. diff --git a/tangle/opencode/.opencode/agents/deployer.md b/tangle/opencode/.opencode/agents/deployer.md index 893aacc0..a36b6dab 100644 --- a/tangle/opencode/.opencode/agents/deployer.md +++ b/tangle/opencode/.opencode/agents/deployer.md @@ -11,7 +11,7 @@ You are the deployer for this weft project. Tangle dispatched you with [the targ ## Running commands -You never sit on a quiet command. Anything that can take more than a few seconds starts in the background, and every wait on it has a cap equal to the time that command normally takes. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. +Every command you run has a cap, written into the command itself: `timeout `, or your tool's own timeout, set to how long that command normally takes. That goes for a loop you write as much as for one command: a wait for a machine to be gone, a disk to be deleted or a status to change is `timeout bash -c 'until ; do echo ", still waiting"; sleep 10; done'`, never a bare `until`, and it prints a line each time it looks so whoever watches sees it move. Before writing a loop, check whether a weft command already waits for that thing and says how it goes (`weft infra stop`, `terminate` and `start` wait for the copy and print each step); use it, under its cap. Anything that can take more than a few seconds starts in the background. At the cap you look (its output, `weft status --json`, `weft daemon logs`): if it is still moving it gets one more period at most, and if it went quiet you stop it and find out why. You never just wait longer, and nothing in weft normally runs for thirty minutes. For you: a build or `weft activate` that compiles the project's own nodes takes 1 to 3 minutes (cap 3 minutes, looking every 30 seconds); `weft activate`, `resync` or `deactivate` with nothing to build takes under 30 seconds; `weft infra start` takes under a minute with its image already there and 2 to 5 minutes when it builds or pulls one (cap 5 minutes); on a cloud install the first one also builds the program on the install's builder (about 2 minutes) before each piece boots a machine (2 to 3 minutes), so cap it at 6 minutes and watch `weft status --on `. The full table, command by command, is in the `weft-running` skill. ## Rules @@ -27,7 +27,7 @@ You never sit on a quiet command. Anything that can take more than a few seconds 1. Read the `weft-deploying` skill. Putting weft itself on the cloud, upgrading it or resizing it is never yours: every step of it is the user's, and Tangle walks them through it in conversation, which you cannot hold. If that is what you were sent to do, do nothing and say so in your report. 2. Check the target exists and you can reach it: `weft target list` shows it and whether a key is stored; `weft status --on ` answers. A missing key ends here, with `weft login ` named for the user. -3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background: the first deploy of a project builds its images from scratch and can take a long time. While it runs, read its output now and then: it names each image it builds and where that build's log is. If it is still printing, keep waiting and put no deadline on it. If it has printed nothing for a long stretch, check `weft status --on `. If the build is still moving there, keep waiting. If it is not, report the silence, the status output, and that the command is still running. +3. Do [the ask], as the skill lays it out. A deploy is `weft activate --on ` while the program is off, and `weft resync --on --mode park` once its triggers are on (`weft status --on ` says which; `hibernate` if the user prefers it, `wipe` only when the user says so), started in the background under its cap: 6 minutes when it builds images (the first deploy of a project, or a change to its nodes), 1 minute when nothing builds. While it runs, read its output every 30 seconds: it names each image it builds and where that build's log is. At the cap, check `weft status --on `: if a build is still moving there, give it one more period; if not, stop the command and report the silence, the status output, and the last lines it printed. 4. Prove it: `weft status --on ` shows the project active on the version you deployed. For each route, call `/connect/local/` once with no credentials and quote the status code: a 401 or 403 means the route is live and gated. Never send a call that would get past a gate on a cloud install: it starts a real run. ## Report diff --git a/tangle/opencode/.opencode/skills/weft-cloud-install/SKILL.md b/tangle/opencode/.opencode/skills/weft-cloud-install/SKILL.md index 555784b9..8badd94a 100644 --- a/tangle/opencode/.opencode/skills/weft-cloud-install/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-cloud-install/SKILL.md @@ -80,8 +80,16 @@ to do before anything that creates or costs something: NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is - # retried until it holds. - retry() { until "$@"; do echo "waiting for IAM to catch up, trying again in 10 seconds"; sleep 10; done; } + # tried again, quietly, for up to three minutes; a last try shows its + # own error. + retry() { + for _ in $(seq 18); do + "$@" >/dev/null 2>&1 && return 0 + echo "waiting for IAM to catch up, trying again in 10 seconds" + sleep 10 + done + "$@" + } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ diff --git a/tangle/opencode/.opencode/skills/weft-deploying/SKILL.md b/tangle/opencode/.opencode/skills/weft-deploying/SKILL.md index f03af324..101b2a19 100644 --- a/tangle/opencode/.opencode/skills/weft-deploying/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-deploying/SKILL.md @@ -222,9 +222,10 @@ running, activate refuses with "these triggers' infra is not running: `weft activate` is for a program that is off. Once its triggers are on, it refuses, and a change goes live with `weft resync --on prod --mode `, which takes the triggers down and brings them back on the new version. On -prod, pick `park` (calls that arrive meanwhile wait and run once the -triggers are back) or `hibernate` (they keep being answered for a grace -window), because those calls can be real people's. `wipe` drops them and +prod, pick `park` (calls that arrive meanwhile wait, a caller on a route +kept on the line, and run on the new version once the triggers are back) or +`hibernate` (the same for a grace window), because those calls can be real +people's. `wipe` drops them and cancels the work waiting on the triggers: fine on a dev install, and on prod only when the user says so. diff --git a/tangle/opencode/.opencode/skills/weft-editor/SKILL.md b/tangle/opencode/.opencode/skills/weft-editor/SKILL.md index 579ea293..01eaa635 100644 --- a/tangle/opencode/.opencode/skills/weft-editor/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-editor/SKILL.md @@ -111,8 +111,9 @@ line) and the verb is not sent, so a build that cannot run never starts. "Deactivate" / "Resync" (amber "Out of sync" when the project changed since activation). An eye toggle "Show trigger subgraph". Deactivating opens the picker "Deactivate: how should triggers come down?": **Park** - (submissions wait indefinitely), **Hibernate** (grace window, then - refuse), **Wipe** (drop everything, cancels suspended runs), plus what to + (nothing dropped: everything that arrives waits and runs once the + triggers are back), **Hibernate** (the same for a grace window, questions + hidden, then refuse), **Wipe** (drop everything, cancels suspended runs), plus what to do with running executions. Reactivating offers "Execute parked + keep suspensions", "Keep suspensions only", "Wipe all". diff --git a/tangle/opencode/.opencode/skills/weft-running/SKILL.md b/tangle/opencode/.opencode/skills/weft-running/SKILL.md index 1c8b1013..a22e504f 100644 --- a/tangle/opencode/.opencode/skills/weft-running/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-running/SKILL.md @@ -170,8 +170,8 @@ status` and the graph print back. | `weft events [--node ] [--kind ] [--iteration ] [--full] [--json]` | a run's events in order, one compact line each (see Reading a run, in the `weft-debugging` skill) | | `weft logs [execution-id]` | a run's log (no argument: the latest execution of the project in the current directory; see Reading a run, in the `weft-debugging` skill) | | `weft follow ` | live events for a project | -| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project needs `--mode ` (a [mode], defined under The three modes). Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | -| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it needs the same `--mode` answer as `deactivate`; without it, it stops and asks | +| `weft activate` / `weft deactivate` | turn triggers on / off. `deactivate` on an active project takes `--mode ` (a [mode], defined under The three modes), `wipe` when you leave it out. Both take `--running-policy `, default `cancel`: on `activate` it says what happens to a worker still up from an older build (cancel what it runs and replace it now, or `wait` for its executions to land, up to `--drain-timeout` seconds); on `deactivate` the same for the project's running executions | +| `weft resync` | deactivate + activate against a fresh build, after editing a trigger subgraph. Only while some trigger is on, the program's or an instance's (with none on it refuses: `weft activate` first), and it takes the same `--mode` as `deactivate`, `wipe` when you leave it out | | `weft infra start` / `status` / `stop` / `upgrade` / `terminate` / `cancel` / `logs` / `show` / `press` / `env` | the project's [infra] (see The infra verbs) | | `weft connect-lib` | copy weft's connect library into the frontend (`front/src/lib/weft-connect`, `--into ` for another folder) so its pages show the editor's connection pickers; the `weft-frontend` skill has when and how. Replaces that folder each run | | `weft token mint` / `ls` / `revoke` | signal tokens: scoped access for an outside listener such as the browser extension. `mint` prints the connect URL, then the bare token on its own line for a script | @@ -260,20 +260,27 @@ not a bug or a gap in weft, so it does not go in a report as friction. ### The three modes -A [mode] is what happens to the runs parked on a person or a timer when the -triggers go down (`deactivate`, `resync` on an active project, the [infra] -verbs that deactivate on the way): - -- `wipe`: their forms and timers are dropped and the runs end cancelled. You - pass it only when nothing is in flight (`weft executions` shows no - suspended run of the project) or the user said to drop the waiting work. -- `hibernate`: the runs stay alive for a grace window (`--grace `, - 15 unless set); a fire arriving inside it is held and delivered when the - project comes back. Past the window new fires are refused; the waiting runs - and the project survive (`wipe` is the [mode] that drops them). -- `park`: the runs stay alive with no time limit; every fire is held until - the project is reactivated. Your pick when the user is editing and people - are mid-conversation. +A [mode] is what happens to the work that arrives while the triggers are +down, and to the runs waiting on a person or a timer (`deactivate`, `resync` +on an active project, the [infra] verbs that deactivate on the way): + +- `wipe`: everything waiting is dropped, the questions deleted, and the + runs waiting on the triggers end cancelled; new work is refused. You pass + it only when nothing is in flight (`weft executions` shows no suspended + run of the project) or the user said to drop the waiting work. +- `park`: nothing is dropped. A call from outside, an answer to a waiting + run, and every fire a trigger makes by itself (a schedule's tick, a + message on a connection it holds) wait, and run once the triggers are + back, on the version they come back with; someone calling a route is kept + on the line until then. The questions stay listed for the people who + answer them. Your pick when the user is editing and people are + mid-conversation. A schedule parked for two hours has two hours of ticks + waiting when it comes back. +- `hibernate`: the same as `park` for a grace window (`--grace `, + 15 unless set), with the questions hidden (someone who already had one + open can still send their answer). Past the window, new work is refused + and the triggers stop listening; the waiting runs and the project survive + (`wipe` is the [mode] that drops them). **`wipe` is what you pass while you are building.** Nothing waiting on the program is anyone's conversation yet, so dropping it costs nothing and the @@ -288,10 +295,13 @@ under `hibernate` and `park` alike (the mode says what happens to the parked work, the policy what happens to the running work: two separate answers), and it ends by cancelling whatever is still running at its cap (`--drain-timeout`, 60 seconds unless set); under `wipe` waiting is refused. -With no `--mode` and no terminal (which is every command you run) the mode -is `wipe`. That is the right answer while you are building, so you rarely -type it; you type `--mode hibernate` or `--mode park` when the program is one -people are using and the work in flight has to survive. +With no `--mode` and no terminal (which is every command you run), +`deactivate` and `resync` use `wipe`. That is the right answer while you are +building, so you rarely type it; you type `--mode hibernate` or `--mode park` +when the program is one people are using and the work in flight has to +survive. The [infra] verbs never pick for you: when a trigger reads the +[infra], they stop and name `--mode` and `--grace`, and you run them again +with one. ## The infra verbs @@ -301,11 +311,13 @@ restarts. You pick the verb by what you want to keep: - **If the [infra] is not running yet**, or was stopped: `weft infra start`. It brings every unit up to its spec and waits until ready. Running it again - does nothing for units already up. It leaves activation alone. + does nothing for units already up. The triggers that came down with the + [infra] (a stop, a terminate) come back on by themselves once it is up; one + switched off with `weft deactivate` stays off. - **If you changed an infra node's spec** (image, env, volumes) and want it live: `weft infra upgrade`. Each unit whose spec changed goes down and comes back on the new spec, the others are left alone, every disk is kept. - Once the [infra] reports ready, `weft activate`. + The triggers it took down come back on once the [infra] reports ready. - **If you want it off for a while and the data kept** (a paired WhatsApp session, a database's rows): `weft infra stop`. The containers go, the disks stay, and `weft infra start` brings it back with everything in it.