feat(instance): add an Operations console for sending any operations-API request

DESIGN.md

@@ -28,6 +28,16 @@ The `/apis` route is a **custom, in-house API explorer** ([`src/features/instanc
 - **Try-it-out is credentialed cross-origin `fetch`** (`credentials: 'include'`), the same model Swagger used — which is why the **CORS warning + one-click enable** flow in [`APIDocs.tsx`](src/features/instance/apis/APIDocs.tsx) matters and must be preserved. Auth offers Cookie (session, default) / Basic / Bearer; Basic/Bearer add an `Authorization` header.
 - **Persistence**: the server + auth selections persist to `localStorage` under `LocalStorageKeys.ApiExplorerSettings`, **keyed by entity id**, so a credential set for one instance never applies to another.
 
+## Operations console (the instance/cluster "Operations" tab)
+
+[`src/features/instance/operations/`](src/features/instance/operations/) sends one freeform operations-API body through the entity's own `getInstanceClient` client (direct, basic, or proxied — whatever every other page uses). The rules, each pinned by a test there:
+
+- **The typed text is the request body.** It is parsed only to validate it and read `operation`; re-serializing the parse would round integers past 2^53 (a primary key) before Harper saw them. Axios sends a string body that parses as JSON as-is, only trimmed.
+- **At most one send.** It sets `skipGatewayRetry`, because the shared client replays a 502/503/504 for any body whose `operation` reads by name, and a component's `get_…` operation is no proof that it only reads. Only a request Studio refused before sending counts as unsent. Anything that reached Harper marks data stale, an error answer included, because an arbitrary operation can change something before it fails; and a request abandoned mid-flight (the page unmounted) is reported as possibly applied.
+- **Only Harper's own reads skip the confirmation.** [`isBuiltInRead`](src/features/instance/operations/operationKinds.ts) is the restart gate's naming rule ([`isReadOperation`](src/integrations/api/isReadOperation.ts)) _and_ membership in the role editor's catalog of Harper's operations, because a component can register a mutating `get_…`. Everything else — unknown names, every `sql` — confirms, showing the exact body. It guards against a mistyped paste; Harper's `verifyPerms` is what authorizes each operation.
+- **Operations Studio has to know about are not sent blind.** A `restart` is tracked with `markRestarting` exactly like the Restart button's (released at once on a 4xx; a 5xx can be a gateway losing the answer to a restart that ran), so the page's reads are held (an instance) or their connectivity errors muted (a cluster, which restarts one member) until `waitUntilAnsweringAfterRestart` sees the target again. `logout` is refused: it would end the session the rest of Studio is still using. Any other operation with such a side effect needs the same treatment rather than a raw send.
+- **Nothing leaves the page.** Failures render inline and never pass through `console.error` or the global React Query handlers (RUM reports the former, and bodies carry passwords and secrets). The editor buffer is not persisted, and the console remounts per entity so a confirmation written for one instance can never be sent to another. After a possible write, entity queries are marked stale with `refetchType: 'none'`, never refetched into whatever the operation just did. The download is abandoned once progress passes 5 MB (XHR reports progress about every 50 ms, so this bounds, not caps, what arrives), and display truncates like the API explorer.
+
 ## Cloud authentication presentation
 
 All cloud authentication routes use `AuthLayout`; keep form submission, validation, OAuth targets, and error handling in the existing route components/hooks. `AuthInput` keeps `FormControl` directly around the input so label/error associations and React Hook Form refs reach the DOM. React 19 passes `ref` through props; browser tests verify first-invalid-field focus. Password selectors must use an exact label match because the reveal button also has “password” in its accessible name.
@@ -39,7 +49,7 @@ The wide hero artwork spans both columns beneath the form card in three independ
 [`restartTracker.ts`](src/lib/restart/restartTracker.ts) is the one record of which clusters and instances are restarting, so a planned outage doesn't read as a failure. Read it before adding anything that takes an instance down.
 
 - **Two sources.** Studio's own Harper `restart` runs mark and release their targets (`markRestarting`, released in `finally`) — central manager never hears about these. Container ops are central manager's: every `getClusterInfo` fetch mirrors its `RESTARTING`/`STARTING` statuses into the tracker — including a one-node cluster whose only member is restarting under an instance-only op, which leaves the cluster's own status `RUNNING`/`PARTIAL`. A new flow that takes an instance down must use one of the two, or its failures toast like before.
-- **Reads are held; everything else is refused.** Every `getInstanceClient` client holds a read (`get_`/`search_`/`describe_`/`list_`/`read_`…) while its entity is `down` and sends it on recovery, but refuses a write at once with `ERR_TARGET_RESTARTING` — a write queued for minutes could land after the user gave up on it, or retried it ([`restartGate.ts`](src/lib/restart/restartGate.ts)). Token minting counts as a write on purpose, so a route guard's Fabric Connect attempt falls through to sign-in instead of stalling. Requests that drive or watch a restart must pass `skipRestartGate: true`, or they wait on themselves. A held read is cancelled if the entity's connection generation changes before release: credentials are read at send time, so it would otherwise go out as whoever connected meanwhile.
+- **Reads are held; everything else is refused.** Every `getInstanceClient` client holds a read (`get_`/`search_`/`describe_`/`list_`/`read_`…) while its entity is `down` and sends it on recovery, but refuses a write at once with `ERR_TARGET_RESTARTING` — a write queued for minutes could land after the user gave up on it, or retried it ([`restartGate.ts`](src/lib/restart/restartGate.ts); the read rule is [`isReadOperation`](src/integrations/api/isReadOperation.ts), shared with the operations console). Token minting counts as a write on purpose, so a route guard's Fabric Connect attempt falls through to sign-in instead of stalling. Requests that drive or watch a restart must pass `skipRestartGate: true`, or they wait on themselves. A held read is cancelled if the entity's connection generation changes before release: credentials are read at send time, so it would otherwise go out as whoever connected meanwhile.
 - **Central manager's operations proxy refuses a transitional target outright** — `400 "Cluster is not active"` / `"Instance is not active"` for any `INACTIVE_STATES` status, which includes a _rolling_ restart's `RESTARTING` cluster — and relays the host manager's connection errors as a `500` carrying the raw error text. That is why a proxied client holds for marks with `proxyRefuses`, and why `isConnectivityFailure` counts those two shapes. The set also covers ERROR, FAILED and the terminal states, which no restart releases: a poll through the proxy must be gated on [`isOperationsProxyRefused`](src/components/ui/utils/badgeStatus.tsx), not `isStoppedOrTransitioning`, or it 400s every tick for as long as the page is open (HarperFast/studio#1569).
 - **Only query errors are muted, and only connectivity-class ones.** Mutations still toast; a 403 or a Harper 500 during a restart still toasts.
 - **Detection is per tab.** A page learns of a container op started elsewhere only from a cluster-record fetch, and the in-cluster studio pages don't poll the cluster, so they don't notice one started in another tab.
@@ -52,6 +62,29 @@ The wide hero artwork spans both columns beneath the form card in three independ
 - **On 5.2+ the first instance is still written and restarted alone.** Only once it is back up on the new config is the write fanned out with `replicated: true` from that instance — even when it was the last instance to restart, because the fan-out also reaches live members Studio does not restart (not RUNNING, or failed earlier), which the per-instance path leaves on the old config. Fanning out first would leave an unproven config pending on every node, which a Cancel could no longer contain.
 - **A later instance skips its write only when the fan-out acknowledged that exact node** — its CM `replicationFqdn` (lower-cased) appears among the successful `replicated[]` entries. Fabric names each Harper node after that FQDN (host-manager sets `NODE_HOSTNAME` and `REPLICATION_HOSTNAME` to it). A count of acknowledgements is not proof: the origin's replication peers are not CM's restart targets, so a stale peer can stand in for a missing one. A target with no `replicationFqdn`, a rejected fan-out (the instance client rejects any response with a failed peer), or a response with no usable `replicated[]` all keep the per-instance write.
 
+## Instance clients replay only reads
+
+Every `getInstanceClient` client replays a 502/503/504 up to three times (5s/10s/20s), but only for a request [`isReadRequest`](src/integrations/api/isReadOperation.ts) accepts: a GET, or a POST whose body's `operation` is a read by Harper's naming — the restart gate's rule, one predicate for both ([`retryGatewayErrors.ts`](src/integrations/api/retryGatewayErrors.ts)). A write is sent once and its first 5xx is final, because no gateway status proves the write was not applied: Harper itself answers 503 after handing an operation to a worker that exits or overruns. Do not add an exemption for a "usually safe" status or a write that looks idempotent; a caller whose write is safe to repeat retries it itself, where it can say why.
+
+## A write with no verdict never reads as failed
+
+The same rule decides what a failure says. [`isUnansweredWrite`](src/integrations/api/unansweredWrite.ts) is a request `isReadRequest` rejects, answered 502/503/504 or not at all (`ERR_NETWORK`, a timeout). [`describeError`](src/react-query/queryClient.ts) checks it before reading any body and returns "Outcome unknown … check whether it took effect before you try again", so the global mutation toast and every `describeError` caller say so with no copy of their own. Gate on status, never body (a gateway's body describes the gateway), and keep it to display: rewriting `error.message` would defeat `shouldKeepEvent`, which recognises timeouts and `Network Error` by message.
+
+- **A caller that adds its own failure toast to a mutation** asks [`shouldToastOwnFailure`](src/react-query/queryClient.ts) first, so "Failed to …" never sits beside the global warning. A caller with no mutation behind it (rename, the chat tools), or one whose mutation opts out of the global toast (the deploy mutation behind import, redeploy and staged-release activation), shows `describeError`'s text itself, under a title from [`deployFailureTitle`](src/features/instance/applications/modals/reportDeployHealth.ts) for a deploy ("Couldn't confirm … was redeployed", never "Failed to …"), and a loop that tallies one write per target (cluster restart, rolling config, licenses) counts these apart from failures.
+- **Do not re-wrap an AxiosError into a plain `Error`** for a write: the predicate needs its `config`. `secrets.ts` passes an unanswered one through as is.
+- **A restart with no verdict stays marked until its target answers again or the watch gives up**, as an accepted one does, because a gateway can lose the answer to a restart that ran. The Restart button holds the mark while it watches, and the cluster loop waits on that member before restarting the next.
+- **These toasts are never collapsed** onto a shared id: each warns about a different write, and a read timing out alongside one must not replace it through `request-timeout`.
+- **A read that is costly to repeat** stays a read here and opts out of the replay instead (`package_component`, `skipGatewayRetry`).
+
+## A restart Harper accepted never reads as failed
+
+Harper's answer to the restart request is its only verdict on the restart: `restart` answers before it tears down (harper `bin/restart.ts` schedules the teardown 50ms out so the reply can go first) and `restart_service` once its job has started (`executeJob`). So [`restartInstance`](src/integrations/api/instance/status/restartInstance.ts) rejects with the request's own error only when that request failed; after a 2xx, a target Studio never sees answer the `user_info` watch successfully rejects with [`RestartUnconfirmedError`](src/integrations/api/instance/status/restartUnconfirmed.ts), which reads "Not back yet", never "Failed", and is never a reason to send the restart again.
+
+- **The watch outlasts the usual recovery time** (one settle, then two probe schedules), and `onSlowToReturn` fires once between them so a caller can say it is still waiting. Every restart mark uses `RESTART_MARK_TTL_MS`, derived from the same constants, so no mark lapses while its watch still runs; the probes skip the gateway replay for the same reason.
+- **Only a successful probe counts as back, never a 401/403.** Through central manager's operations proxy a 403 is CM refusing the caller before it contacts the instance (`notAllowed()` in its `HDBInstance.js`), so it says nothing about the target; reading it as "back" would report a restart Studio never saw finish. That is also why the copy says Studio "couldn't confirm it came back", not that the target never answered.
+- **A one-at-a-time loop stops at a member that is not back** (cluster restart, rolling config update): restarting the next one could leave no member serving. Both loops also wait on a member whose restart got no verdict, and stop if that follow-up watch gives up too: it may have restarted and still be down, or, in a config update, restarted onto a configuration that broke it. Either report names the members it left alone, and it, like the no-verdict warning, outranks Cancel, which would otherwise hide the warning not to restart again; the report then says which members Cancel left. A Cancel clicked during a member's configuration write stops before its restart, and the report says the configuration may be pending there. On a 5.2+ cluster the config update never fans out from a canary that is not back, since it never proved it can run the configuration; a stop after the fan-out keeps the note that the members it left may already have the new configuration saved.
+- **The chat tools return `success: true`** for it: their action was carried out, and `false` invites the agent to repeat it.
+
 ## Cluster upgrade picker — which versions an existing cluster is offered
 
 [`buildUpgradeVersionOptions`](src/features/clusters/upsert/lib/buildUpgradeVersionOptions.ts) is the one place that decides, and its test file is the spec. Two rules, and they differ on purpose:

src/features/auth/describeAuthFailure.ts

@@ -1,3 +1,4 @@
+import { NO_RESPONSE_CODES } from '@/integrations/api/unansweredWrite';
 import { describeError } from '@/react-query/queryClient';
 import { AxiosError } from 'axios';
 
@@ -15,24 +16,15 @@ export const OUTCOME_UNKNOWN_MESSAGE =
 	"We didn't get an answer from the server, so we can't tell whether that went through.";
 
 // 503 alone: the server is declining requests, so it very likely never processed this one and a
-// plain retry is honest. (`curryRetryGatewayErrors` also classes 502/504 as retryable, but it is
-// installed on instance clients only, never on `apiClient` — nothing retries these calls
-// automatically, and the retry this copy invites is the user's own.)
+// plain retry is honest. (`curryRetryGatewayErrors` replays 502/503/504, but only for reads and
+// only on instance clients, never on `apiClient` — nothing retries these calls automatically, and
+// the retry this copy invites is the user's own.)
 const TRANSIENT_STATUSES: ReadonlySet<number> = new Set([503]);
 // Every form using this submits a non-idempotent POST, and each of these means the request had
 // already been handed upstream, so the write may have applied (RFC 9110 §9.2.2 — a retry is only
 // safe once you know it did not). Repeating blindly turns a completed sign-up into a 409 with the
 // verification mail already sent (#1668).
 const OUTCOME_UNKNOWN_STATUSES: ReadonlySet<number> = new Set([502, 504]);
-// Axios reports a request that got no response as ERR_NETWORK — including a CORS rejection, and
-// including a connection that dropped *after* the POST was applied — so it cannot be described as
-// "we couldn't reach the server". It is the same unknown outcome. ECONNABORTED/ETIMEDOUT are the
-// timeout pair; which one axios uses depends on `transitional.clarifyTimeoutError`.
-const OUTCOME_UNKNOWN_CODES: ReadonlySet<string> = new Set([
-	AxiosError.ERR_NETWORK,
-	AxiosError.ETIMEDOUT,
-	AxiosError.ECONNABORTED,
-]);
 
 /**
  * Our own copy for a failure the server's own words shouldn't answer, or `undefined` to defer.
@@ -64,7 +56,7 @@ export function describeRetryableAuthFailure(rawErr: unknown, outcomeUnknownReco
 	// ERR_CANCELED is a local abort that reached nothing, and a config fault (ERR_INVALID_URL,
 	// ERR_BAD_OPTION) never left the browser — none of those is ours to describe.
 	const code = axiosErr?.code;
-	return typeof code === 'string' && OUTCOME_UNKNOWN_CODES.has(code) ? outcomeUnknown : undefined;
+	return typeof code === 'string' && NO_RESPONSE_CODES.has(code) ? outcomeUnknown : undefined;
 }
 
 /** Display text for an auth-form rejection, preferring whatever the server said. */