Skip to content

Python SDK test suites never run in CI #1417

Closed as not planned
Closed as not planned

Description

@inahus99

Problem

The Python packages ship real test suites, and nothing runs them.

ci.yml is TypeScript-only — Bun, turbo run check-types on @supermemory/ai-sdk and @supermemory/memory-graph, then biome ci. The four publish-*-python.yml workflows build and publish to PyPI, but none of them invoke pytest. So these packages are released on the strength of whatever the author ran locally.

Package Tests collected Runs in CI
agent-framework-python 54 no
openai-sdk-python 35 no
cartesia-sdk-python 1 no
pipecat-sdk-python 1 no

Why it matters

The gap is not theoretical. openai-sdk-python spent a month broken on main — its suite could not even be collected against the supermemory release of the day:

ImportError: cannot import name 'MemoryAddResponse' from 'supermemory.types'
  src/supermemory_openai/tools.py:12: in <module>

That is #1235: reported by an outside user on Jul 11, fixed on Aug 12 by #1236. Any CI run of the existing tests would have caught it the day supermemory 3.5 went out, instead of leaving it to a user hitting the import at runtime a month later.

It also costs reviewer time: a Python PR today can only be verified by creating a virtualenv and running pytest by hand, which is friction on every review.

Proposal

A path-filtered workflow that runs each package's suite on PRs touching it. All four suites pass on current main, so all four can be turned on:

  • agent-framework-python — installs the package, runs pytest
  • openai-sdk-python — installs the package plus python-dotenv, runs pytest
  • cartesia-sdk-python and pipecat-sdk-python — these stub their heavy runtime deps (cartesia-line, pipecat-ai, loguru) at import time, so they run under unittest against the sources with nothing installed, and cost seconds

#1418 implements this.


Update, Aug 21 — this issue was written while #1235 was still open, and originally proposed leaving openai-sdk-python out so the workflow would not land red. Now that #1236 has fixed it, that suite runs green on current main (24 passed, 11 skipped — the skips need SUPERMEMORY_API_KEY), so it can be included from the start. The table above has also been corrected to collected test counts on main; the earlier agent-framework-python figure counted tests added in #1415 and #1416, which are not merged.

Activity

  1. linear-code commented on Aug 5, 2026

    @linear-code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions