Skip to content

workspace: a memory block whose mirror fails is never retried — local and workspace memory diverge silently #1161

Description

@sahrizvi

Found while running the workspace bind + memory mirror flow end to end against a live tenant.

What happens

When mirrorBlock can't confirm the workspace's memory setting — e.g. the /datamates/ lookup times out — it logs a warning and returns without mirroring:

[WARN] service=altimate-workspace-memory-sync could not confirm workspace memory setting
       err=WorkspaceApiError: Request to <api>/datamates/ timed out after 15s

The block is written locally, nothing is lost, and the block is deliberately left unindexed so that — per the code's own comment — "a later save retries it".

The gap is that nothing constitutes "a later save" for a block the user doesn't touch again. There is no sweep, no reconciliation on session start, and no retry queue. An untouched block that missed its mirror stays local indefinitely, while the workspace silently lacks it.

Why re-linking doesn't rescue it

The obvious recovery — re-run altimate-code link — doesn't help, for a non-obvious reason. backfillOnBind is gated on alreadySeeded, and re-selecting the workspace a directory is already linked to isn't a re-bind, so the bind path never fires and no sweep runs.

Switching to a different workspace does re-seed (sameBinding compares datamateId, so the marker stops matching) — but that's not a recovery path anyone would think to use, and it changes the binding.

Observed

During a period of degraded network, 3 of 5 memory writes failed this way. All 5 blocks existed locally; only 2 reached the workspace. Re-linking to the same workspace changed nothing. Editing one of the missed blocks caused it to mirror immediately, confirming that a re-save is the only recovery. The other two remained local-only indefinitely.

Why it matters

The failure is silent from the user's point of view — the memory tool reports success, because the local write did succeed. Local and workspace memory then diverge permanently, and nothing surfaces the divergence: the SaaS shows fewer records than the user has, and another project bound to the same workspace never sees the missing facts.

On a flaky connection this accumulates.

Possible directions

  • Session-start reconciliation — compare local block ids/hashes against the workspace's record set and push what's missing. Cheapest in the sense that hydrate already fetches that record set every session, so the comparison is nearly free.
  • Un-gate the bind sweep on an explicit re-link — makes re-running link the intuitive recovery it currently isn't.
  • Retry queue — most thorough, most machinery.
  • At minimum, surface it — the tool result currently says "Created" with no indication the mirror didn't happen.

Environment

Reproduced against a live workspace with ALTIMATE_WORKSPACE=1. Not GA-affecting while the pilot flag stays off by default.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions