Skip to content

Embed & Explore folder handling and size-aware auto backend - #52

Open
NetZissou wants to merge 4 commits into
mainfrom
fix/embed-folder-and-auto-backend
Open

NetZissou wants to merge 4 commits into
mainfrom
fix/embed-folder-and-auto-backend

Conversation

@NetZissou

Copy link
Copy Markdown
Collaborator

Four independent fixes found while reading the app log for a small image folder, one commit each.


a1df5a7 Size-aware auto:

resolve_backend("auto") resolved to cuML whenever a GPU was present, making the size heuristics inside reduce_dim / run_kmeans unreachable from the apps.

Both now use one rule, CUML_AUTO_RANGE, with per-method bounds chosen from measured crossovers of the app's own paths: cuML for PCA from 500 samples and KMeans from 1000, UMAP only from 10000 (subprocess start-up has to be amortized), t-SNE only below 3000 (the collapse-safe exact solver is O(N^2)).

Explicit cuml / sklearn choices are honored unchanged.

Note, the measured numbers behind the bounds are machine-specific , but it should be applicable to general use case.


cf71f61 Progress context reports failure

StreamlitProgressContext showed its success message whenever the block exited without raising, so "Embedding complete!" appeared above the "no images" error; the save/repartition sections had the exit message overwrite their error.

New fail(message); all four failure sites use it.


87db000 Include subfolders

checkbox in the Embed section (the service already supported recursive listing; the UI never asked for it).

[xxx@xxx Goura_cristata]$ tree -L 1
.
├── eol
└── gbif
image

bdb71e1 No model load for 0 images

generate_embeddings returns before load_model_unified when the listing is empty.

see screenshot above for the verified fix.

NetZissou and others added 4 commits October 6, 2026 10:41
resolve_backend("auto") resolved to cuML whenever a GPU was present, which made the size heuristics inside reduce_dim and run_kmeans unreachable from the apps and sent small datasets through cuML start-up for no gain.

Replace both with one shared rule, CUML_AUTO_RANGE:

cuML for PCA from 500 samples and KMeans from 1000, UMAP only from 10000 (subprocess start-up has to be amortized), and t-SNE only below 3000 (the collapse-safe exact solver is O(N^2)).

The sidebars now pass the dataset size and method; explicit backend choices are honored unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
EmbeddingService.generate_embeddings already supported recursive listing, but the sidebar never passed it, so a folder organized b, for example,  species subfolders reported 0 images.

Add an Include subfolders checkbox (default on) and pass it through.
@NetZissou NetZissou self-assigned this Oct 12, 2026
@NetZissou NetZissou changed the title Embed & Explore folder handling and size-aware auto backend (#50) Embed & Explore folder handling and size-aware auto backend Oct 12, 2026
@NetZissou NetZissou linked an issue Oct 12, 2026 that may be closed by this pull request
@NetZissou NetZissou added bug Something isn't working enhancement New feature or request labels Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Embed & Explore: image-folder handling and backend auto-selection

1 participant