|
| 1 | +# Effect recovery coordination |
| 2 | + |
| 3 | +`emit` returns a JSON-serializable handle containing the public `effect_id`. |
| 4 | +Registering `on_recovery` opts the effect into retirement when its owner has |
| 5 | +stopped heartbeating. `on_status` is optional and receives responses only to |
| 6 | +explicit `request_effect_recovery(handle)` intents. Normal success and failure |
| 7 | +retain their existing callbacks. |
| 8 | + |
| 9 | +## Watchdog using supported APIs |
| 10 | + |
| 11 | +```ruby |
| 12 | +class ReportExport < SolidObjects::Actor |
| 13 | + attribute :revision, default: 0 |
| 14 | + attribute :export_effect, default: nil |
| 15 | + attribute :artifact_key, default: "" |
| 16 | + attribute :applied_effect_id, default: nil |
| 17 | + |
| 18 | + def start |
| 19 | + self.revision += 1 |
| 20 | + self.export_effect = emit(:build_report, |
| 21 | + revision: revision, |
| 22 | + on_success: :export_finished, |
| 23 | + on_failure: :export_failed, |
| 24 | + on_recovery: :recover_export, |
| 25 | + on_status: :inspect_export, |
| 26 | + recovery_timeout: 120) |
| 27 | + schedule(at: Time.now + 30, key: "export-watchdog").watchdog |
| 28 | + nil |
| 29 | + end |
| 30 | + |
| 31 | + def watchdog |
| 32 | + request_effect_recovery(export_effect) if export_effect |
| 33 | + end |
| 34 | + |
| 35 | + def recover_export(effect_id:, arguments:, outcome:) |
| 36 | + return unless effect_id == export_effect&.fetch("effect_id") |
| 37 | + return unless arguments.fetch("revision") == revision |
| 38 | + |
| 39 | + start |
| 40 | + end |
| 41 | + |
| 42 | + def export_finished(effect_id:, arguments:, result:) |
| 43 | + apply_export_result(effect_id:, arguments:, result:) |
| 44 | + end |
| 45 | + |
| 46 | + def export_failed(effect_id:, arguments:, error:) |
| 47 | + end |
| 48 | + |
| 49 | + def inspect_export(effect_id:, outcome:, arguments: nil, result: nil) |
| 50 | + return unless effect_id == export_effect&.fetch("effect_id") |
| 51 | + |
| 52 | + case outcome |
| 53 | + when SolidObjects::EffectRecoveryOutcome::COMPLETED |
| 54 | + apply_export_result(effect_id:, arguments:, result:) |
| 55 | + when SolidObjects::EffectRecoveryOutcome::DEFERRED, SolidObjects::EffectRecoveryOutcome::PENDING |
| 56 | + schedule(at: Time.now + 30, key: "export-watchdog").watchdog |
| 57 | + end |
| 58 | + end |
| 59 | + |
| 60 | + private |
| 61 | + |
| 62 | + def apply_export_result(effect_id:, arguments:, result:) |
| 63 | + return unless effect_id == export_effect&.fetch("effect_id") |
| 64 | + return unless arguments.fetch("revision") == revision |
| 65 | + return if applied_effect_id == effect_id |
| 66 | + |
| 67 | + self.artifact_key = result.fetch("artifact_key") |
| 68 | + self.applied_effect_id = effect_id |
| 69 | + end |
| 70 | +end |
| 71 | +``` |
| 72 | + |
| 73 | +Register `build_report` through the ordinary effect registry. Its successful |
| 74 | +result in this example is `{ "artifact_key" => "reports/example.pdf" }`. |
| 75 | +The library builds the retirement payload, including `"outcome" => "retired"`; |
| 76 | +the effect handler does not return that outcome itself. Ruby actor operations |
| 77 | +receive keywords. `recover_export` needs handle/revision guards but no outcome |
| 78 | +guard because only a new retirement invokes it. The same guarded result helper |
| 79 | +handles success and completed-status repair, preventing duplicate application. |
| 80 | +Only recovery emits a replacement; status observations never do. |
| 81 | + |
| 82 | +The watchdog is optional: `on_recovery` alone enables automatic retirement. |
| 83 | +`on_status` alone does not enable retirement, polling, or subscriptions. Explicit |
| 84 | +checks require both bindings persisted by `emit` and cannot replace either. |
| 85 | + |
| 86 | +## Public envelopes and timeout |
| 87 | + |
| 88 | +`SolidObjects::effect_handle` describes `{ "effect_id" => String }`. |
| 89 | +`SolidObjects::effect_retired_payload[Arguments]` requires the effect ID, original |
| 90 | +arguments, and `"outcome" => "retired"`. Status uses |
| 91 | +`SolidObjects::effect_recovery_payload[Arguments, Result]`, a record union. |
| 92 | +Every variant has an effect ID; retired and completed require original arguments; |
| 93 | +only completed has a recorded result (including `nil`). Other Ruby observations |
| 94 | +contain only the effect ID and outcome. |
| 95 | + |
| 96 | +Strict packaged-consumer tests verify the constant literals and individual |
| 97 | +records. Steep 2.0 does not narrow this string-keyed record union after comparing |
| 98 | +`payload["outcome"]` with `COMPLETED`; accessing `result` through the union still |
| 99 | +fails its return-type check. Use the concrete completed/retired record in typed |
| 100 | +helpers after validating the discriminator, with an explicit type assertion if |
| 101 | +needed. The library retains precise records rather than weakening them to an |
| 102 | +untyped hash. Ordinary Ruby keyword dispatch needs no payload hydration. |
| 103 | + |
| 104 | +| Frozen `SolidObjects::EffectRecoveryOutcome` constant | Wire value | Meaning | |
| 105 | +| --- | --- | --- | |
| 106 | +| `RETIRED` | `"retired"` | This check retired the effect; separate recovery owns replacement. | |
| 107 | +| `DEFERRED` | `"deferred"` | Fresh owner; preserve its claim and attempts. | |
| 108 | +| `PENDING` | `"pending"` | Initial execution or retry remains with the scheduler. | |
| 109 | +| `COMPLETED` | `"completed"` | Original arguments and recorded result are available. | |
| 110 | +| `DEAD` | `"dead"` | Preserve terminal failure and its existing callback. | |
| 111 | +| `ALREADY_RETIRED` | `"already_retired"` | Earlier retirement; no additional recovery notification. | |
| 112 | +| `MISSING` | `"missing"` | Owned binding exists but effect data was pruned. | |
| 113 | + |
| 114 | +`recovery_timeout` must be positive finite seconds and requires `on_recovery`. |
| 115 | +Fractional durations are allowed. Omission uses the current runtime |
| 116 | +`process_alive_threshold`, normally 60 seconds. Smaller positive values are |
| 117 | +floored at that runtime threshold; changing configuration changes the effective |
| 118 | +floor even for existing effects. Database lookup errors surface as errors, |
| 119 | +never as missing/stale observations. |
| 120 | + |
| 121 | +Effect workers maintain their process heartbeat while the handler waits on |
| 122 | +external I/O and while committing success or failure. A long-running healthy |
| 123 | +handler therefore remains protected beyond the recovery timeout. This requires |
| 124 | +an available database connection for the heartbeat, as well as runtime threads |
| 125 | +that can continue running. |
| 126 | + |
| 127 | +Failed updates emit `solid_objects.process.heartbeat_failed` and retry at the |
| 128 | +configured heartbeat interval without consuming effect attempts. If an outage |
| 129 | +lasts beyond the freshness window, recovery can still be permitted; retries do |
| 130 | +not cancel external work or extend the configured window. |
| 131 | + |
| 132 | +## Compatibility and installation |
| 133 | + |
| 134 | +Upgrade all effect workers and process cleanup roles before emitting effects |
| 135 | +with recovery enabled. Older runtimes do not honor the persisted bindings or |
| 136 | +the new lock protocol. |
| 137 | + |
| 138 | +Run `solid_objects:install:migrations` and your application's normal migration |
| 139 | +process before starting upgraded workers. The additive migration creates the |
| 140 | +durable binding table; it does not change existing effect status constraints. |
| 141 | + |
| 142 | +`emit` now returns its handle, including without recovery options. Callers may |
| 143 | +ignore it. Wrappers must return `super`; operations whose last expression used |
| 144 | +to be `emit` may now return the handle to callers. End those operations with |
| 145 | +`nil` if their previous result must remain unchanged. The return-value change is |
| 146 | +intentional and is not strictly backward compatible. |
| 147 | + |
| 148 | +## Transaction and lock protocol |
| 149 | + |
| 150 | +Emission, actor state, the effect, and its recovery binding share the actor's |
| 151 | +fenced commit. An explicit check executes on that same connection. Automatic |
| 152 | +recovery performs one independent library transaction per candidate, with no |
| 153 | +application transaction waiting on a second connection. |
| 154 | + |
| 155 | +The lock order is originating instance, effect rows ordered by public effect |
| 156 | +ID, recovery binding rows in the same order, then owner processes ordered by ID. |
| 157 | +Completion and failure must acquire the instance before the effect. Pending |
| 158 | +claims lock only their effect and do not subsequently acquire an instance lock. |
| 159 | +Mailbox insertion reuses the instance lock already held by the decision. |
| 160 | +Multiple checks in one actor commit lock all their effects and bindings before |
| 161 | +locking any processes. Unlocked candidate reads are hints, never decisions. |
| 162 | + |
| 163 | +Automatic passes prefilter owner freshness and the effective per-effect timeout |
| 164 | +using database time, and visit at most `claim_scan_limit` stale candidates. Fresh |
| 165 | +actors and owners are not locked, including owners protected by extended grace. |
| 166 | +Remaining stale effects are revisited on later polls. Every candidate still |
| 167 | +undergoes the authoritative locked recheck, and successful retirement announces |
| 168 | +the committed mailbox work through the existing wake-up mechanism. |
| 169 | + |
| 170 | +The decision samples database wall time after obtaining the owner lock. The |
| 171 | +effective timeout is the larger of the runtime's `process_alive_threshold` and |
| 172 | +the effect's persisted `recovery_timeout`, in seconds. A heartbeat newer than |
| 173 | +the cutoff is fresh; equality is stale. A stopped or draining process with |
| 174 | +fresh heartbeat evidence still protects an opted-in effect until that timeout. |
| 175 | +Cleanup preserves opted-in claims, and process pruning excludes processes that |
| 176 | +still own effects. Later effect polling and process cleanup revisit deferred |
| 177 | +effects without resetting their last heartbeat. |
| 178 | + |
| 179 | +Retirement stores a durable `retired_at` in `effect_recoveries` and moves the |
| 180 | +effect into the existing terminal `completed` storage state, clearing its |
| 181 | +claim. The recovery record distinguishes retirement from successful completion; |
| 182 | +no success callback is generated. All recovery observations consult that record |
| 183 | +before interpreting the effect row. A late completion or failure is rejected by |
| 184 | +the existing processing/claim fence. This representation avoids rewriting the |
| 185 | +existing effect-status constraint across adapters. |
| 186 | + |
| 187 | +The retirement record and the recovery mailbox message commit atomically. A |
| 188 | +winning explicit check additionally enqueues its status response after the |
| 189 | +retirement notification. Failure to insert either message rolls back the whole |
| 190 | +decision. Retirement is deduplicated per effect; check responses use separate |
| 191 | +per-request idempotency keys. Wake-up signals are delivery hints after commit. |
| 192 | + |
| 193 | +Recovery bindings survive effect/message pruning and remain until the originating |
| 194 | +instance is destroyed or pruned. They do not prevent normal message or instance |
| 195 | +retention. Within that lifetime, a removed non-retired effect reports `missing` |
| 196 | +and a retirement record reports `already_retired`. A handle without an owned |
| 197 | +binding raises an error, without disclosing another actor's state or recreating |
| 198 | +a destroyed actor. Status-response message idempotency follows normal mailbox |
| 199 | +retention; callers cannot supply or reuse internal check request IDs. |
| 200 | + |
| 201 | +An owner heartbeat measures process liveness, not effect progress. Retirement |
| 202 | +does not cancel the old handler or prove a remote request stopped. External |
| 203 | +actions still require idempotency across retries and replacement generations. |
0 commit comments