Skip to content

Deletion support across the board: deletes never propagate from a platform to the eVault or to other platforms #1130

Description

@Bekiboo

Description

Deletion is not supported anywhere in the W3DS sync path. A user can delete a post, a message, a chat or a binding, and that deletion reaches no other platform — and in most cases never leaves the platform's own database. A delete has to be passed down five hops: platform database → web3-adapter → eVault → AaaS → every other platform holding a copy. Four of the five hops are still missing or actively do the wrong thing, so fixing any one of them changes nothing user-visible — hop 3 has since been fixed and nothing user-visible changed.

Scope of this issue is all five hops, at record level. Decommissioning an eVault or removing a registry entry (identity-level erasure) is out of scope and should be tracked separately.

1. A platform's local delete never reaches the adapter. Platform services delete through TypeORM's repository.delete(id) and repository.softDelete(id) — query-builder forms that do not fire entity subscribers. The web3adapter watchers only hook afterRemove, and no subscriber implements afterSoftRemove. Only 7 call sites in the whole monorepo use remove()/softRemove(); every other delete is invisible to sync from the moment it happens.

2. The adapter cannot express a delete. EVaultClient (infrastructure/web3-adapter/src/evault/evault.ts) exposes storeMetaEnvelope, updateMetaEnvelopeById, uploadFile, fetchMetaEnvelope and storeReference — there is no delete method. infrastructure/web3-adapter/src/index.ts has no delete branch. So on the rare path where afterRemove does fire, handleChange (:300) maps the removed entity and upserts a snapshot of it into the eVault — the opposite of the intent, and worse than doing nothing. mappingDb.deleteMapping() exists but is never called, so the local↔global id mapping leaks for every deleted record.

3. eVault deletes emit an awareness packet. Done — not by this issue. removeMetaEnvelope (graphql-server.ts:689) and legacy deleteMetaEnvelope (:1611) now both pass an awareness context into db.deleteMetaEnvelope, which writes an AwarenessOutbox row with operation: "delete" and the record's real schemaId. awareness-outbox-dispatcher.ts:382/:410 forwards operation to AaaS. This matches how every other mutation fans out (create :387, update :525, bulk create :846, binding documents :987/:1088, uploadFile :1474, legacy store/update :1282/:1571; the two delete paths are :729 and :1627).

Landed in #1137 and #1175, neither of which references this issue.

Hop 3 landing without hop 4 made things worse, not neutral. A delete packet carries data: null (db.service.ts:975) and a real schemaId, so receivers no longer skip it as an unknown schema — they run it down the upsert path. Traced end to end:

  • AaaS accepts and stores it. /ingest requires only id and schemaId, and Packet.data is nullable. Packet is a latest-state projection that upserts on the MetaEnvelope id, so the stored latest state for that record has its data overwritten with null. It is then delivered as data: null, operation: "delete" (services/DeliveryEngine.ts:304).
  • fromGlobal({ data: null }) does not throw. getValueByPath returns undefined for a null root (mapper/mapper.ts:62), dereferenceFileValue passes non-file-URI values straight through, and getLocalId(undefined) returns null (mapping.db.ts:120). Every mapped key comes back undefined, so nothing fails loudly here.
  • Then it throws, in the wrong place. In pictique the users branch reaches user.name = req.body.data.displayName (WebhookController.ts:76, :92) and gets a TypeError on null. The catch (:464) only logs — it never responds — so AaaS times out at 5s, retries on the 30s→24h backoff, and dead-letters after the 24h window. 9 of the 10 controllers dereference req.body.data.<field> directly; blabsy is the only one that does not.
  • Where it does not throw, it corrupts. posts with a known mapping assigns local.data.text = "" and post.likedBy = [], so a delete blanks the post's text and clears its likes. With no mapping it calls createPost, so a delete creates a blank post. chats assigns participants = [] and admins = [].

The undefined-valued keys are harmless only because TypeORM's save() skips undefined properties; the "" and [] assignments above do not depend on that.

4. Receivers ignore the operation type. The AaaS packet type already declares operation?: "create" | "update" | "delete" (services/awareness-service/api/src/types.ts:12), stores it on Packet, and passes it to subscribers (services/DeliveryEngine.ts:304). All 10 platform WebhookControllers contain zero references to operation; every inbound packet is treated as an upsert.

5. There is no vocabulary for "deleted" on the receiving side. The eVault side is now settled: since #1175, deleteMetaEnvelope (db.service.ts:956) prunes instead of destroying — the record and its envelopes are relabelled PrunedMetaEnvelope / PrunedEnvelope and a delete version is appended to history, which stays readable. There is no DETACH DELETE left. Platforms still have nothing to write: only 8 of 64 ontology schemas carry an isArchived flag, so a receiver has no way to distinguish "deleted" from "never seen" or "not authorised".

Adjacent gaps that fall out of the same missing path:

  • Orphaned blobs. uploadFile writes an object plus a File meta-envelope. Deleting the meta-envelope never calls storage.deleteObject — that is only used on the upload-rollback path (graphql-server.ts:1517), so the blob stays in object storage forever.
  • No cascade semantics. Deleting a chat, group or post leaves undefined what happens to the messages, comments, votes and signatures that reference it.

Reference

  • Related: eVault: expose creation/update/deletion metadata on MetaEnvelope in GraphQL responses #1129 (creation/update/deletion metadata on MetaEnvelope) — shared the tombstone-vs-hard-delete decision, which feat(evault): versioned MetaEnvelopes, pruning deletes and a history API #1175 has since settled in main. eVault: expose creation/update/deletion metadata on MetaEnvelope in GraphQL responses #1129 likely needs re-checking against the new history API.
  • infrastructure/evault-core/src/core/protocol/graphql-server.ts:689, :1611 — delete resolvers, now fanning out
  • infrastructure/evault-core/src/core/awareness/awareness-outbox-dispatcher.ts:382, :410 — operation forwarded to AaaS
  • infrastructure/evault-core/src/core/db/db.service.ts:956 — pruning delete (was a hard DETACH DELETE at :549 when this was filed)
  • infrastructure/web3-adapter/src/evault/evault.ts — client with no delete method
  • infrastructure/web3-adapter/src/index.ts:300 — handleChange, no delete branch
  • infrastructure/web3-adapter/src/db/mapping.db.ts:140 — deleteMapping, zero callers
  • platforms/*/api/src/web3adapter/watchers/subscriber.ts — afterRemove only
  • services/awareness-service/api/src/types.ts:12, services/DeliveryEngine.ts:304 — operation already carried end to end
  • infrastructure/web3-adapter/src/mapper/mapper.ts:62, src/db/mapping.db.ts:120 — why fromGlobal(null) returns quietly instead of failing
  • platforms/pictique/api/src/controllers/WebhookController.ts:76, :92, :464 — the TypeError on a null data, and the catch that never responds

Acceptance Criteria

  • The eVault emits an awareness packet with operation: "delete" for removeMetaEnvelope and legacy deleteMetaEnvelope, matching how create and update already fan out. (Done in Make AaaS delivery durable and restart-safe #1137 / feat(evault): versioned MetaEnvelopes, pruning deletes and a history API #1175.)
  • EVaultClient gains a delete method, and the adapter has a delete path that calls it instead of upserting the removed entity.
  • A local delete on a platform reliably reaches the adapter — either by moving services onto remove()/softRemove(), adding afterSoftRemove hooks, or making deletes explicit in the service layer rather than relying on subscribers.
  • deleteMapping is called when a record is deleted, so the id mapping does not leak.
  • Platform webhook controllers branch on operation; a delete packet removes or archives the local row instead of upserting it.
  • Receiving a delete for an unknown record is a no-op, not an error — deletes may arrive before or without the corresponding create.
  • Deletes do not ping-pong: honouring an inbound delete must not emit an outbound delete back to the origin (the same guarantee the requestingPlatform check gives create/update).
  • A delete of a File meta-envelope also removes the backing object from storage, or an agreed GC path does.
  • Cascade behaviour is specified per ontology: deleting a parent (chat, group, post) states what happens to its children.
  • ACL is enforced on the propagated delete — a platform cannot cause a deletion it lacks the DELETE bit for.
  • Documented: how a platform should represent a deleted record, and what a consumer may assume from the absence of a record.

Design decisions to settle

  1. Tombstone or hard delete. A tombstone makes propagation, "deleted vs never existed", and late-joining consumers straightforward, but retains a record of the deleted thing — which cuts against the erasure story. A hard delete plus a log-backed delete packet keeps nothing behind, but a consumer that was offline during the delete has no way to catch up. Settled in main by feat(evault): versioned MetaEnvelopes, pruning deletes and a history API #1175 as pruning with readable history, without reference to this issue or eVault: expose creation/update/deletion metadata on MetaEnvelope in GraphQL responses #1129.
  2. Soft or hard on the receiving side. Should an inbound delete hard-delete the local row, or set isArchived? Only 8 of 64 ontologies have such a flag today, so either it is generalised or receivers hard-delete.

Desired Output (may vary)

A user who deletes something in one platform sees it disappear from the eVault and from every other platform that holds a copy — and a platform receiving a delete has a defined way to represent it. Deletion becomes a first-class operation in the sync path, not a gap that silently resurrects data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions