TC-1176: Write Probe signed-in reads, Better Stack heartbeat /fail, dense promote watch

[samgbafa]

Refs TC-1176 (incident 2026-10-14: signed-in KV reads and SQL 504 after ~62 s while /healthz stayed green) and TC-959 (Better Stack as code).

Holden's direction is one probe, not two, so this extends the TC-313 Write Probe instead of adding a host-timer probe. It already did a signed-in read-back, and it failed at 23:36Z and 23:50Z during TC-1176. But it ran only every 15 minutes with no per-stage deadline, and a red scheduled run only e-mails whoever last edited the cron line, so nobody was paged.

What changes

Reads, the paging signal. signin, space, then a kv read of the fixed key read-probe/fixed and a sql select of the fixed row read_probe.id=1. Each stage has a hard 20 s client deadline. The probe writes the fixed key and row itself, once, the first time they're missing.

• A good read pings the Better Stack heartbeat Node authorized reads (Write Probe) (ID 509295; managed as code in tinycloud-dev ops/betterstack/uptime.json, in the companion PR).
• A failed read is counted in a state file carried in the Actions cache. From the 2nd failure in a row the probe POSTs <heartbeat>/fail with one line, e.g.
  TC-1176 node read probe FAIL (2 in a row, normal): stage=kv outcome="timeout >=20s" duration_ms=20004 build=1.20.6 host=node.tinycloud.xyz at=2026-10-15T01:05:07Z
  The outcome is timeout >=20s, HTTP <code>, connection refused, DNS, and so on, so Amos can tell a read hang (restart) from the node being down (rollback). One failure alone never pages, so a ~60 s deploy restart is absorbed. Per Amos, there is no pause for deploys.

Writes, TC-313's round-trip (put -> get -> delete). These keep their thresholds and now also have the 20 s deadline. A breach still fails the job but doesn't touch the heartbeat. The PROBE_THRESHOLD_SIGNIN_MS override is removed: sign-in is now a read stage under the 20 s deadline.

Cadence */5 (the repo is public, so Actions minutes are free). Heartbeat period 5 min plus 55 min grace, so it pages after 60 min with no ping. I measured this workflow's actual schedule over 600 runs (2026-10-06..14, cron */15): the gap between runs was p50 19 min, p99 48 min, max 54 min. A shorter window would page falsely whenever GitHub drops runs. Failed reads don't wait for the grace: they alert through /fail.

Dense promote watch. workflow_dispatch mode=dense reads every 30 s for 5 min (alternating KV and SQL, so each endpoint gets at most 1 request per 30 s), then every 5 min for 60 min. A read over 5 s counts as a failure; 2 in a row POST /fail. The job summary lists every sample's latency. The new docker.yml job promote-watch dispatches it after every Phala deploy attempt. It's a separate job, so the job holding production secrets never gets actions: write. While a dense watch runs, scheduled runs skip themselves, so only one probe owns the heartbeat.

force_failure dispatch input: points the run at https://read-probe-forced-failure.invalid (fails at DNS), to exercise the alert path.

Secrets: the probe never prints PROBE_PRIVATE_KEY or the new BETTERSTACK_HEARTBEAT_URL. Errors are reduced to a classified outcome, raw details are redacted, the heartbeat URL is never logged, and it must be a Better Stack heartbeat URL.

Tests

• test/probe/probe-core.test.ts (12, stub client, mocked timers; new Write Probe tests workflow): hung read → timeout >=20s at the deadline, and the next stages are skipped; 504, connection refused and DNS outcomes from SDK-shaped errors; the slow threshold; the 2-in-a-row rule and its rate limit; corrupt state; the /fail body format; ping GET vs /fail POST; result lines and redacted logs that never contain the key or URL; dense alternation and schedule; dense 2-slow → /fail → ping.
• Local smoke with the real SDK, using a fresh local throwaway key against a forced-failure host and a local fake 504 server: outcomes DNS and node /info not OK (the SDK's sign-in pre-check gives no status); the 2nd run decides heartbeat_action=fail with the expected body.
• actionlint is clean on write-probe.yml, write-probe-tests.yml and docker.yml.

QA / Amos: gh workflow run write-probe.yml -R TinyCloudLabs/tinycloud-node --ref main -f mode=dense. The run is titled Write Probe (dense promote watch); conclusion=success means every sample was under 5 s.

[samgbafa]

Astra security review (gpt-6-astra): APPROVE. That's after three rounds. Fixed: separate KV/SQL failure counts (a healthy SQL read no longer hides a broken KV read); only a 2xx counts as delivered, and a rejected /fail is retried; the failure state is shared between normal and dense mode; heartbeat ownership checks fail closed. No credential leakage found.

Live proof on this branch (UTC 2026-10-15):

• green run 38661022682: kv 828 ms, sql 1735 ms, ping HTTP 200, heartbeat 509295 went up;
• forced failures 38661279164 then 38661416422: the 2nd POSTed /fail (stage=signin outcome="DNS"), the heartbeat went down, and incident 1032191869 opened at 00:41:43Z;
• green run 38661590418 resolved it at 00:43:21Z.

Heartbeat 509295 is paused until this merges, because main's old probe doesn't ping. Resume it in the Better Stack UI after the merge. The first scheduled run on main will then ping.