indexed nearest-by join #1

Draft

sezruby wants to merge 10 commits into

mainfrom

Commits on May 12, 2026

feat(knn): Phase 0 foundation — LanceProbe primitive + metric types

sezruby
and
claude
committed
feat(knn): staged RDD pipeline + IndexedNearestJoin.apply + bounded TopKHeap

sezruby
and
claude
committed
feat(knn): Phase 1.5 — fragment-grouped probing for multi-task parallelism

sezruby
and
claude
committed
feat(knn): 3-exec Catalyst-visible staged plan with AQE-visible merge shuffle

sezruby
and
claude
committed
feat(knn): df.kNearestJoin DataFrame extension method

sezruby
and
claude
committed
feat(knn): Phase 3 hardening — refineFactor, prefilter pushdown, IVF-PQ recall

sezruby
and
claude
committed
feat(knn): Spark 4.2 SQL integration — IndexedNearestByJoinRule

sezruby
and
claude
committed
test(knn-bench): benchmark suite — synthetic, Wikipedia perf, SIFT/Cohere recall, SQL

sezruby
and
claude
committed
docs(knn): design, impl plan, reviewer guide, ANN proposal, benchmark results

sezruby
and
claude
committed

Commits on May 13, 2026

test(knn-bench): closer-to-RewriteNearestByJoin baseline + cluster scaling sweep

sezruby
and
claude
committed