[vector store 12/14] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) by edwinyyyu · Pull Request #1618 · MemMachine/MemMachine

edwinyyyu · 2026-09-14T19:38:13Z

Purpose of the change

Lets a deployment tune the HNSW index, the optimizers and the quantization of a Qdrant collection. Split out of #1606 on review, since it is a tangential change to the store-as-collection work; it comes over from default (a0753d3).

Stack

Slice 12 of 14, every PR targeting speedkick; merge bottom-up.

#	PR	change
1	#1606 (merged)	Remove per-project filterable properties
2	#1654 (merged)	Remove custom sharding from the Qdrant store
3	#1630	Bound every request to a remote vector store by a configured timeout
4	#1622	Create a session's storage with the session, never on a request
5	#1623	Make the examples that write to a project create it first
6	#1624	Make no memory request create a project
7	#1625	Remove open-or-create and close from both stores
8	#1626	Rename logical collection to partition, and open to get, on both stores
9	#1627	Make a vector store one collection, with string-keyed partitions
10	#1631	Mint an incarnation per partition life in a SQL-arbitrated registry; delete logically, reclaim by purge
11	#1657	Check liveness after every remote operation; keep a deleted partition's tombstone until a clean sweep and a retention
12	#1618 (this PR)	Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization
	#1597	EventMemory session, source and expansion: not a slice of this chain; 13 and 14 need it
13	#1628	Make a vector store filter only on the properties it declares
14	#1616	Close the filter union, and make negation the complement on every backend

This PR's own change is its last commit, 70aff660 (6 files changed, 271 insertions(+), 9 deletions(-)); the rest of its diff is the slices under it, and drops out as they merge. Stacked on #1657; #1628 is stacked on it.

What changed

QdrantConf gains hnsw_config, optimizers_config and quantization_config: plain mappings mirroring qdrant-client's HnswConfigDiff, OptimizersConfigDiff and QuantizationConfig (for example {"turbo": {"always_ram": true, "bits": "bits2"}}), so qdrant-client stays optional for configuration parsing. DatabaseManager passes them to QdrantVectorStoreParams, which validates them against qdrant's own models and reports a failure as a configuration error.
They apply to the store's data collection, never to its registry collection.
m must be 0 or unset: the collection is multi-tenant and disables the global graph in favor of per-partition payload indexing, so a deployment tunes payload_m. It defaults to 16, as before.

Tests

Plain mappings coerce to qdrant's models; m of 0 or unset is accepted and any other value is refused naming payload_m; the native HNSW config merges overrides and pins m at 0; the overrides reach the data collection's creation and not the registry's. Configuration parsing and the manager's wiring assert the three fields round-trip.

uv run pytest packages/server/server_tests packages/client/client_tests: 2219 passed, on the slice-12 tree (2026-09-16, on speedkick 779362c); this tip passes its own run. ruff check, ruff format --check and ty check --project packages/server clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The Qdrant and Milvus clients were built without a timeout, so a remote write could hang a request indefinitely. `request_timeout` on QdrantConf and MilvusConf, in seconds, is passed to the client; it is required, with no default, so a deployment states how long it is willing to wait, and the configuration wizard supplies 30 seconds as the starting point. The sample configurations and the configuration docs show the option. A breaking configuration change on `speedkick`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

A required field bounds only the deployments that added it and fails the rest at load; a default bounds every deployment, including one whose cfg.yml predates the option, and matches every other field on QdrantConf and MilvusConf. The wizard no longer carries the value: it constructs the confs and the field supplies it. Zero and negative values are rejected at load rather than handed to httpx as the request timeout and to pymilvus as the gRPC deadline, where zero expires every request on arrival. Without the option, qdrant-client already bounded a request at 5 seconds (httpx's default for REST, DEFAULT_GRPC_TIMEOUT for gRPC); pymilvus passed no deadline, so a Milvus request could wait forever. The default applies 30 seconds to both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Neither client's keyword carries the unit, and the field mirrors neither (both take `timeout`), so it follows max_retry_interval_seconds on the embedder and language model configurations instead. The sample configurations lose their "seconds" comments, which the name now carries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

MilvusClient's constructor timeout is the time it waits for the channel to become ready, at construction and on reconnect; a request is bounded only by the timeout passed to that request, and pymilvus keeps no default for it, so every request the store made had no deadline. The store now takes request_timeout_seconds and passes it on every request; the client keeps it as its connection bound. A test wraps every request method and checks the timeout reaches each call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

qdrant-client takes its timeout as an int and rounds a fraction up, so a fractional value was honored by pymilvus and silently changed for Qdrant. An int is honored exactly by both, and matches max_retry_interval_seconds on the embedder and language model configurations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…quest The event backend created a session's vector store collection and segment store partition on the first request that opened the session, so a search or a write for an unknown session created storage as a side effect, and the service locator was the only place that knew both stores' create paths. The owner is the session. Every path that creates a session row runs through EpisodicMemoryManager._create_session, which inserts the row and, when the row is new, creates the session's partitions in its segment store and its vector store (create_episodic_memory_storage); an equivalent re-create accepts the row and leaves the storage as it is. The request path binds handles with the stores' lookups and raises SessionPartitionMissingError when a partition is absent: a session without its storage is broken, not new. Deleting a session with no open instance deletes its partitions by key, so a session whose storage was never fully created can still be deleted. MemMachine.create_session goes through the manager for the same reason. The semantic manager owns its one collection and creates it, once, at the storage's first use. With that, nothing calls the stores' open-or-create. The API is unchanged: the manager's open-or-create still creates a session a memory request names, now through the same path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The simple chatbot example, the TypeScript REST demo and the Dify plugin's add-memory tool wrote to a project without creating it, relying on the write to create it. Each now creates its project before its first memory request and accepts 409 as the project already existing. No behavior changes for them; they stop depending on a write creating a project. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Adding memories to, or searching, a project that did not exist created it, with the server's default configuration, without the caller's knowledge. Now only the create-project request creates a project: a write or a search opens the session and answers 404 for an unknown project, as the search endpoint already promised; the manager's open-or-create goes. Two callers depended on the implicit creation. `org_id` and `project_id` default to `universal`, so the API promises the project `universal/universal`; the server creates it, once, at startup, and leaves one that already exists as it is. The MCP add tool names its own project and has no create-project counterpart, so it creates the project it writes to, once, and says so. The API doc strings and the OpenAPI document say which requests create a project. A breaking API change on `speedkick`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Nothing calls them since a session's storage is created with the session: `open_or_create_collection` and `close_collection` leave the vector store interface and its four backends, `open_or_create_partition` and `close_partition` leave the segment store interface and its implementation, and the two config-mismatch errors that only open-or-create raised go with them. A store creates on `create_*`, strictly, and looks up on `open_*`, answering None; create-if-absent is the owner's, where the key's provenance is known. Source changes are deletions only. The tests that exercised open-or-create as a fixture use a test-side create-if-absent instead, and the tests of its own semantics go. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…res' lookup to get_partition A vector store's logical collection becomes a partition, the segment store's word for the same thing, and both stores' lookup is get_partition, answering None like a Python get. Identifiers only, produced by the script below; the (namespace, name) identity, the per-partition config and every docstring are as they were, and the next change gives them their meaning. The native clients' create_collection and delete_collection keep their names. ```sh set -e cd "$(git rev-parse --show-toplevel)" git mv packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_collection.py \ packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_partition.py git ls-files -z 'packages/server/*.py' | xargs -0 perl -0pi -e ' s/VectorStoreCollection(?!Config)/VectorStorePartition/g; s/in_memory_vector_store_collection/in_memory_vector_store_partition/g; s/vector_store_collection(?!_schema|_namespace)/vector_store_partition/g; s/open_collection/get_partition/g; s/def create_collection\(/def create_partition(/g; s/def delete_collection\(/def delete_partition(/g; s/\.create_collection\((\s*namespace=)/.create_partition($1/g; s/\.delete_collection\((\s*namespace=)/.delete_partition($1/g; s/\.create_collection(?=\s*=\s*AsyncMock|\.assert_)/.create_partition/g; s/\.delete_collection(?=\s*=\s*AsyncMock|\.assert_)/.delete_partition/g; s/"create_collection"/"create_partition"/g; s/"delete_collection"/"delete_partition"/g; s/only delete_collection is invoked/only delete_partition is invoked/g; s/test_delete_collection_/test_delete_partition_/g; s/open_partition/get_partition/g; ' uv run ruff check --fix --quiet packages/server uv run ruff format --quiet packages/server ``` Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…uilt by the composition root A vector store was a factory of logical collections, each identified by a (namespace, name) pair and created with its own dimensions and schema; a backend that limits native collections shared one among logical collections of equal configuration, under a name derived from a hash of that configuration, and a registry per namespace mapped names to it. A store is now one collection: `VectorStore(collection, vector_dimensions, indexed_properties)` names its one native collection (or its tables and index files) at construction, every partition of it shares the collection's dimensions and schema, and `provision()` creates the collection's durable resources idempotently, before `startup`. `create_partition(key)`, `get_partition(key)` and `delete_partition(key)` take a string key; a partition is a payload value (Qdrant), a partition-key value (Milvus) or a pair of tables (the SQLite stores) inside the collection, and the registry beside it records what each partition was created under, so a store built with other dimensions or another schema raises VectorStorePartitionSchemaMismatchError instead of reading columns and vectors that are not there. Collection names may be 64 bytes; the hash-derived native names go, and with them `VectorStoreCollectionConfig` and the per-partition config. `DatabaseManager.get_vector_store(backend, collection=, vector_dimensions=, indexed_properties=)` builds and caches one store per (backend, collection), keyed by the service's system keys; asking for a collection again with other dimensions or keys is a configuration error. The event backend's collection is `long_term_memory__<embedder>` and the semantic memory's `semantic_memory__<embedder>`, one cell of the purpose-by-embedder matrix each; the two SQLite stores of one backend share its engine, and MemMachine warms the event backend's store through the locator, since building it needs the embedder's dimensions. The data path is as it was: a partition stores every property of a record and filters on any key, with the declared keys indexed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…bitrates; delete logically, reclaim by purge A partition is identified to callers by its key and inside every store by an incarnation minted when the partition is created. Records, points and index files are keyed by the incarnation, never by the key, so a partition deleted and re-created under the same key starts empty and its predecessor's storage is never adopted by, or reclaimed out from under, the successor. Handles are bound to one incarnation: once it is deleted, every operation of the handle raises VectorStorePartitionHandleStaleError. The registry that maps a key to its incarnation lives in SQL on every backend, so any server process may create, delete and purge, with nothing coordinated outside the database: creation is an insert the primary key arbitrates (a racing creator on any process gets AlreadyExists; a minted incarnation colliding with a live or queued one is re-minted, up to _MAX_MINT_ATTEMPTS, then VectorStoreAttemptsExhaustedError), deletion is one transaction that queues the incarnation for purge and deletes the row, and a purge claim is a row lock the database hands to one purger at a time. delete_partition therefore makes the partition unreachable at once, and the new purge_deleted_partitions reclaims the storage afterward, oldest deletion first, a bounded amount per call, safe to repeat and to run from several processes. Both SQLite stores keep the registry beside their data in the one file: every partition of a collection in shared tables (records, vec0 with the incarnation as its partition key, pending-operation log) with a purge queue beside the registry, writes fenced with a self-checking UPDATE or a registry SELECT under BEGIN IMMEDIATE. Qdrant and Milvus have no transactions, unique constraints or conditional writes, so their registry is `SqlPartitionRegistry` in the deployment's relational database: one table pair per backend kind (`vector_store_qdrant_pt`/`_gc`, `vector_store_milvus_pt`/`_gc`), shared by every store on that kind of backend, a purge claim under `FOR UPDATE SKIP LOCKED` on PostgreSQL so concurrent purgers split a backlog, retired when the backend's filter-delete returns and kept when it raises, and a per-operation fence that is a registry lookup by incarnation. The registry collections the two stores kept inside their backends go, and with them the per-process lock tables that serialized creation within one process only, and `registry_replication_factor`. `QdrantConf` and `MilvusConf` gain a required `registry_database`, the name of a relational database under `resources.databases`; DatabaseManager hands its engine to the store, and `provision()` creates the two tables on it, surviving a racing provisioner. The `VectorStore` contract no longer restricts a partition to one process: every operation is safe from any process sharing the backend, and a store that cannot give that says so itself (the engine-backed SQLite store holds a partition's index in the process that opened it; the sqlite-vec store is shared by the processes of one node). The SQLite stores' on-disk layout changes (shared tables per collection in place of tables per partition); existing SQLite vector store files are not migrated. partition_lifecycle_contract.py holds the contract tests every backend mixes in: stale handles, empty re-creation, idempotent deletion, purge reclaiming what deletion deferred and leaving live partitions alone; run against Qdrant in local, REST and gRPC modes. test_sql_partition_registry.py runs the registry on SQLite and PostgreSQL, concurrent creators included. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The first time a vector store is handed out, the resource manager starts the same purge loop it runs for segment stores, one per (backend, collection); close() cancels both sets. Mechanical churn in the same change: the loop's interval and pause constants lose their SEGMENT_STORE_ prefix and the loop takes a label for its failure log line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

A Qdrant or Milvus handle read its incarnation as live and then made the remote call; a deletion committing between the two left the write under a dead incarnation and the operation reporting success. The handle now reads the row again after the call and raises the stale error if the incarnation died meanwhile, so a completed operation never reports success on a dead partition. No lock spans the remote call and the logical delete waits for nothing; a write that landed under a dead incarnation is the purge's to reclaim. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

One purge could not be the last: a Qdrant or Milvus write the registry read as live can land after the purge that followed the deletion, and the queue entry was retired at that first purge, so nothing would ever find those points. The queue entry is now the incarnation's tombstone. A purge round looks for one point under the incarnation and deletes by filter if there is one; a round that found points keeps the entry due; a round that found none stamps it clean; and the entry is removed only by a round that finds nothing again once `tombstone_retention_seconds` (a day unless configured, per store, on the database clock) has passed since the clean stamp. A round that finds a late write clears the stamp and the rounds start over. Until removal the incarnation is never re-minted. `purge_deleted_partitions` returns True when the round found points, so the sweeper comes back for another round, and False when it found none or nothing was due. The retention is the one clock in the design and it decides nothing about validity: a stale write is refused by the liveness check after the operation, and the retention only has to exceed, by orders of magnitude, the longest a request can be in flight, which request_timeout_seconds bounds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

A retention is set in days and compared on the database clock; a fraction of a second has no use, and the field then matches request_timeout_seconds and max_retry_interval_seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…tization `QdrantConf` gains `hnsw_config`, `optimizers_config` and `quantization_config`, plain mappings mirroring qdrant-client's `HnswConfigDiff`, `OptimizersConfigDiff` and `QuantizationConfig`, so qdrant-client stays optional for configuration parsing; the store's params validate them against qdrant's own models. They apply to the store's data collection, never to its registry collection. `m` must be 0 or unset: the collection is multi-tenant and disables the global graph in favor of per-partition payload indexing, so a deployment tunes `payload_m`, which defaults to 16 as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

edwinyyyu mentioned this pull request Sep 14, 2026

[vector store 1/12] Remove per-project filterable properties (speedkick) #1606

Merged

edwinyyyu marked this pull request as draft September 14, 2026 20:03

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 4 times, most recently from 28884f8 to 6ef5caf Compare September 14, 2026 20:33

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 6ef5caf to 1364e65 Compare September 14, 2026 21:40

edwinyyyu changed the title ~~(Depends on #1606) Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick)~~ [vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 14, 2026

edwinyyyu mentioned this pull request Sep 14, 2026

[vector store 14/14] Close the filter union, and make negation the complement on every backend (speedkick) #1616

Draft

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 5 times, most recently from 556eb26 to 2a1da2c Compare September 14, 2026 23:16

edwinyyyu changed the title ~~[vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick)~~ [vector store 11/12] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 14, 2026

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 2 times, most recently from 556eb26 to 918e7cb Compare September 15, 2026 17:26

edwinyyyu mentioned this pull request Sep 15, 2026

[vector store 10/14] Mint an incarnation per partition life in a SQL-arbitrated registry; delete logically, reclaim by purge (speedkick) #1631

Draft

edwinyyyu changed the title ~~[vector store 11/12] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick)~~ [vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 15, 2026

edwinyyyu mentioned this pull request Sep 15, 2026

Add session, source and expansion to EventMemory (speedkick) #1597

Open

edwinyyyu mentioned this pull request Sep 16, 2026

[vector store 11/14] Keep the Qdrant and Milvus partition registries in SQL, so any process may create, delete and purge (speedkick) #1656

Closed

edwinyyyu changed the title ~~[vector store 11/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick)~~ [vector store 12/14] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 16, 2026

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 1e51ab3 to ddedb9e Compare September 16, 2026 18:57

edwinyyyu changed the title ~~[vector store 12/14] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick)~~ [vector store 11/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 16, 2026

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from ddedb9e to cf97dd3 Compare September 16, 2026 19:02

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from cf97dd3 to 76f5f24 Compare September 16, 2026 19:21

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 76f5f24 to e7d8fac Compare September 16, 2026 19:44

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from e7d8fac to b31d854 Compare September 16, 2026 20:13

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from b31d854 to f5e85b6 Compare September 16, 2026 20:34

edwinyyyu changed the title ~~[vector store 11/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick)~~ [vector store 12/14] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 16, 2026

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from f5e85b6 to a0f3d0a Compare September 16, 2026 21:21

edwinyyyu mentioned this pull request Sep 16, 2026

[vector store 11/14] Check liveness after every remote operation; keep a deleted partition's tombstone until a clean sweep and a retention (speedkick) #1657

Draft

edwinyyyu and others added 13 commits September 16, 2026 14:37

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from a0f3d0a to 70aff66 Compare September 16, 2026 21:44

Navigation Menu

Uh oh!

[vector store 12/14] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) - #1618

[vector store 12/14] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick)#1618
edwinyyyu wants to merge 17 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/qdrant-collection-options-speedkick

edwinyyyu commented Sep 14, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

edwinyyyu commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the change

Stack

What changed

Tests

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

edwinyyyu commented Sep 14, 2026 •

edited

Loading