feat: relevance ranking, evals, chronicle, embeddings, skill drift-check#1754
Closed
BarrettHolien wants to merge 2 commits into
Closed
feat: relevance ranking, evals, chronicle, embeddings, skill drift-check#1754BarrettHolien wants to merge 2 commits into
BarrettHolien wants to merge 2 commits into
Conversation
Borrow the strongest ideas from tobi/qmd and garrytan/gbrain and graft them onto graphify's structural spine (five features, each: module + CLI + tests): 1. Ranking (graphify/ranking.py): rank query results by Reciprocal Rank Fusion over lexical + structural (proximity/centrality/community) + optional semantic backends, replacing the degree-only render order. New query flags --top-k, --explain, --semantic; also on the MCP query_graph tool. Shared core factored into serve._resolve_and_rank / serve.rank_query_nodes. 2. Evals (graphify/evals.py): `graphify bench` scores P@k/recall/MRR/nDCG against a JSONL fixture using the real ranking pipeline. --init scaffolds, --save/ --replay gate regressions (exit 1) via .graphify-evals/eval-results.jsonl. 3. Embeddings (graphify/embed.py): opt-in `graphify embed` builds a local embedding sidecar (Ollama / sentence-transformers, no API). --semantic fuses cosine similarity into ranking and seeds traversal when lexical finds nothing. 4. Chronicle (graphify/chronicle.py): `graphify chronicle OLD NEW` (or --rev via `git show`) diffs two graph snapshots — nodes/edges, god-nodes emerged/ vanished, community shifts. Generalizes prs.py across time, API-free. 5. Skill drift (graphify/skill_migrations.py): `graphify skill status|check-update` reports installed-skill-vs-package drift (exit 1 for CI) with a migration registry. Skill: document the new query flags and a new analysis.md reference (bench/embed/ chronicle/skill) via tools/skillgen fragments; regenerated + blessed all artifacts. skillgen --check/--audit-coverage/roundtrips all pass.
- Bump version 0.9.0 -> 0.10.0 (minor: five new features) + CHANGELOG entry, and register the 0.10.0 skill migration so `skill check-update` explains it. - Fix embedding quality: nomic-embed-text is trained with `search_query:` / `search_document:` instruction prefixes; apply them (model-keyed, so non-nomic backends are unaffected). Without them nomic's query/doc vectors are mis-calibrated and retrieval degrades. - Stop tracking the local `.graphify-evals/` eval-results log (gitignore + drop the file that `git add -A` swept into the feature commit).
BarrettHolien
force-pushed
the
feat/ranking-evals-chronicle-embeddings
branch
from
88423d1 to
86e0e80
Compare
Author
|
Superseded by #1760 (rebased onto current v8, opened from the HolienTech org fork). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grafts the strongest ideas from tobi/qmd (hybrid-search ranking) and garrytan/gbrain (evals, versioned skills, history) onto graphify's structural spine. Five features, each a module + CLI + tests. Graphify's differentiator: it fuses structural signals (graph proximity, hub centrality, community) that pure-retrieval tools can't compute.
Features
graphify/ranking.py) — rank query results by Reciprocal Rank Fusion over lexical + structural (proximity/centrality/community) + optional semantic backends, replacing the degree-only render order. Newqueryflags--top-k,--explain,--semantic; also on the MCPquery_graphtool.graphify/evals.py) —graphify benchscores P@k / recall / MRR / nDCG against a JSONL fixture using the real ranking pipeline.--initscaffolds;--save/--replaygate regressions (exit 1).graphify/embed.py) — opt-ingraphify embedbuilds a local sidecar (Ollama nomic-embed-text with propersearch_query:/search_document:prefixes, or sentence-transformers; no API).--semanticfuses cosine similarity and seeds traversal when lexical finds nothing.graphify/chronicle.py) —graphify chronicle OLD NEW(or--revviagit show) diffs two graph snapshots: nodes/edges, god-nodes emerged/vanished, community shifts. API-free.graphify/skill_migrations.py) —graphify skill status|check-updatereports installed-skill-vs-package drift (exit 1 for CI) with a migration registry.Version bumped 0.9.0 → 0.10.0 with CHANGELOG entry. Skill fragments regenerated + blessed (
skillgen --check/--audit-coverage/roundtrips pass). 136 new tests; full suite green apart from 4 pre-existing GEMINI_API_KEY-env failures unrelated to this change.