Skip to content

Proposal: shared tool spec + conformance tests so 7-tool parity can't drift #1731

Closed

Description

@mihhhir08

Problem

The 7-tool surface is defined by hand in each SDK: packages/tools/src/tools-shared.ts (TOOL_DESCRIPTIONS), supermemory_openai/tools.py (TOOL_DESCRIPTIONS), and now supermemory_agent_framework/tools.py (#1730). Nothing checks that they agree, and they already don't.

Comparing TOOL_DESCRIPTIONS in TS and supermemory_openai on current main (normalizing camelCase ↔ snake_case tool names):

Tool Identical?
add_memory yes
document_add yes
search_memories no
get_profile no
document_list no
document_delete no
memory_forget no

Some differences are probably intentional (TS has the containerTag override, Python doesn't). But some change model behavior:

  • search_memories gives the model opposite guidance.

    • TS: "Use when explicitly asked to search or recall, or when past context could materially improve the response; do not invoke reflexively on every turn."
    • Python: "Use proactively whenever prior context could help."

    The same agent searches on every turn or only when needed, depending on the SDK.

  • A description promises a guard the code doesn't have. supermemory_openai's document_delete description says deletion is refused for documents "still processing", but SupermemoryTools.document_delete never checks status. The TS version does (assertDocumentCanBeDeleted, TERMINAL_DOCUMENT_STATUSES).

Each parity pass (#1430, #1432, #1433) has had to fix this by hand across packages, and #1730 exists because one SDK was missed.

Proposal

  1. One spec file, e.g. packages/tools/tool-spec.json, holding each tool's name, parameters (name, type, required) and canonical description. Descriptions refer to other tools with a placeholder like {memory_forget}, which each SDK renders in its own naming (memoryForget / memory_forget).

  2. A conformance test in each SDK that loads the spec and fails if the SDK's tools don't match it:

    • packages/tools: vitest, against TOOL_DESCRIPTIONS and the tool schemas
    • supermemory_openai: pytest, against MEMORY_TOOL_SCHEMAS
    • supermemory_agent_framework: pytest, against get_tools() schemas

    These run in the CI jobs that already exist (ci.yml, ci-python.yml), with no new workflow.

  3. Intentional differences are declared in the spec (e.g. "sdks": ["ts"] on the containerTag parameter), so they're documented instead of accidental.

The Python packages would still ship their own copy of the strings, since the wheels can't read a repo file at runtime. The test just keeps those copies honest.

Questions before I build it

  1. Is TS the source of truth for descriptions, or should the Python wording win anywhere, e.g. the search_memories guidance?
  2. Is packages/tools/tool-spec.json a good location, or would you rather have it in its own package?
  3. Should the Python SDKs vendor the spec at build time instead of keeping a hand copy? That's more work but removes the duplication entirely.

Happy to open the PR once the direction looks right. I'd also fix the missing status check in supermemory_openai.document_delete separately, since that's a bug either way.

Activity

  1. linear-code commented on Sep 30, 2026

    @linear-code
  2. MaheshtheDev commented on Sep 30, 2026

    @MaheshtheDev
    Member

    thanks for the issue but, we are working on making it even simpler and should be out in next few days. thanks for the issue though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions