Skip to content

Commit 605f833

Browse files
fix(populate): make Dependencies picklable for multiprocessing
Parallel populate pickles the table (and its connection's Dependencies) to worker processes, which then reload dependencies and reconnect. Dependencies held an `itertools.count` (_node_alias_count) that is not picklable. This was latent on Linux through Python 3.13 because multiprocessing defaulted to `fork` (which inherits rather than pickles); Python 3.14 defaults to `forkserver`, which pickles the initializer args and surfaced the failure (`TypeError: cannot pickle 'itertools.count' object` in test_multi_processing). Add __getstate__/__setstate__ to drop the counter on pickle and rebuild it on load — workers reload dependencies anyway, so the value need not be carried.
1 parent 0a0aeb2 commit 605f833

1 file changed

Lines changed: 13 additions & 0 deletions

File tree

src/datajoint/dependencies.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,19 @@ def clear(self) -> None:
140140
self._node_alias_count = itertools.count() # reset alias IDs for consistency
141141
super().clear()
142142

143+
def __getstate__(self) -> dict:
144+
# itertools.count is not picklable. Multiprocessing populate pickles the
145+
# table (and thus its connection's Dependencies) to worker processes,
146+
# where dependencies are reloaded — so the counter is dropped here and
147+
# rebuilt in __setstate__ rather than carried across the pickle.
148+
state = self.__dict__.copy()
149+
state.pop("_node_alias_count", None)
150+
return state
151+
152+
def __setstate__(self, state: dict) -> None:
153+
self.__dict__.update(state)
154+
self._node_alias_count = itertools.count()
155+
143156
def load(self, force: bool = True, schema_names: set[str] | None = None) -> None:
144157
"""
145158
Load dependencies for the given schemas.

0 commit comments

Comments
 (0)