Skip to content

feat: lossless, idempotent export/import round-trip with JSON-LD @id/@type#13

Open
marcdhansen wants to merge 1 commit into
agentralabs:mainfrom
marcdhansen:lossless-export-import
Open

feat: lossless, idempotent export/import round-trip with JSON-LD @id/@type#13
marcdhansen wants to merge 1 commit into
agentralabs:mainfrom
marcdhansen:lossless-export-import

Conversation

@marcdhansen

Copy link
Copy Markdown

Motivation

Export→import round-trips should preserve all node and edge state to enable:

  • Lossless knowledge transfer between systems (e.g., agent checkpoints)
  • Idempotent rebuilds from JSONL — running rebuild twice should skip duplicates, not corrupt data
  • Merge-friendly JSONL format — multiple agents can append to memory.jsonl and a single rebuild ingests all changes atomically
  • JSON-LD compliance — use standard @id / @type fields for semantic metadata

What Was Broken

  1. Loss of node identity on export→import: Exported nodes lost their original IDs, causing ID collisions when re-importing into an existing graph
  2. No explicit node-ID preservation: Import always auto-assigned new IDs via add_node(), making round-trips non-deterministic
  3. Duplicate edge insertion: Idempotent rebuilds re-added identical edges, corrupting edge counts and metrics
  4. Lost access metadata: created_at, access_count, last_accessed, and decay_score were discarded on import
  5. No JSON-LD export: Export lacked semantic @id / @type fields; import couldn't read them

What This Fixes

1. New add_node_with_id() API (memory_graph.rs)

  • Accepts an explicit node ID and preserves it (or skips if duplicate)
  • Returns Ok(false) if node already exists (idempotent skip)
  • Auto-bumps next_id past the inserted ID to prevent future collisions

2. JSON-LD Export (cli/commands.rs cmd_export)

  • Every node now includes @id: "urn:amem:node:<id>" and @type: "<event_type>"
  • Maintains backward compatibility: legacy id and event_type fields still present
  • Standard JSON-LD URN namespace for semantic tooling

3. Dual-Format Import (cli/commands.rs cmd_import)

  • Reads event_type from either event_type (legacy) or @type (JSON-LD)
  • Reads node ID from either id or by parsing @id (urn:amem:node:)
  • Preserves authoritative fields: created_at, access_count, last_accessed, decay_score
  • Calls add_node_with_id() when explicit ID is present

4. Idempotent Edge Dedup (cli/commands.rs cmd_import)

  • Before inserting an edge, checks if (source_id, target_id, edge_type) tuple already exists
  • Skips duplicates instead of re-inserting
  • Preserves edge created_at timestamp on first insert

Test Evidence

All 30 repository tests pass (cargo test):

  • 30 passed; 0 failed

Round-trip proof harness (openloop test-twolayer.sh):

  • T1 dump: .amemmemory.jsonl with @id on every line ✓
  • T2 lossless: memory.jsonl → fresh .amem → re-export is byte-identical ✓
  • T3 idempotent: Second rebuild into existing .amem adds 0 nodes, 0 edges ✓
  • T4 merge: Hand-appended JSONL line (simulating agent merge) lands on rebuild ✓
  • T5 projection: JSONL → JSON-LD wrapper is valid ✓

Files Changed

  • crates/agentic-memory/src/graph/memory_graph.rs (+32 lines)

    • add_node_with_id() public method with full index updates
  • crates/agentic-memory/src/cli/commands.rs (+68 lines, -8 lines)

    • cmd_export(): Add @id and @type fields
    • cmd_import(): Dual-format read, ID preservation, edge dedup, metadata carry-through
    • Status output: Report skipped duplicates for observability

Breaking Changes

None. Export now produces @id and @type in addition to existing fields. Import accepts both old and new formats. Existing consumers are unaffected.

- add MemoryGraph::add_node_with_id (explicit id, idempotent skip, next_id bump)
- import honors id/@id and created_at on nodes and edges, carries
  access_count/last_accessed/decay_score through, dedupes edges on
  (source, target, type)
- export emits JSON-LD @id/@type alongside legacy keys

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant