Repository navigation
emb-embed-explore Image Folder Import Error #35
Description
when I select a folder with three images to embed, this pops up (full traceback in slack) and it does not recognize the embeddings (clustering errors out stating that embedding must be performed first)
Error messages shared through slack.
Activity
NetZissou commentedon Jun 11, 2026
Investigation shows that the crash is caused due to source & OS conflicts.
The
ImageFolderDatasetcontains lambda functions which can't be serialized by Python during runtime. However, the multiprocessing methodspawnrequires serialization which caused the crash. More details please see issue.Additionally, the Streamlit application behavior should be fixed as it notify users embeddings complete whereas the images were failed to be preprocessed in the first place.
We might consider create a dedicated utility function for image preprocessing for
emb-explorerand removehpc-inferencefrom the dependency list.NetZissou commentedon Jun 12, 2026
^^^ also a blocker for #27 as
hpc-inferenceis agit+https://dependency, and PyPI rejects direct-URL deps.A
ImageFolderDatasetmodule that works for single machine, optimize for image preprocessing on multi-cores CPUs makes sense for the development cycle of this pkg. We shed transitive weight we never use foremb-explorer:matplotlib,nvidia-ml-py,psutil,pyyaml, distributed/rank code.