feat(kg-rag): KG-RAG semantic-graph layer on the new kb-server architecture#423
Open
AdriaGuilera wants to merge 41 commits into
Conversation
Ports the KG-RAG / semantic-graph stack from the legacy
lamb-kb-server-stable into the new pluggable lamb-kb-server (port 9092)
and wires it through Knowledge Stores instead of legacy Knowledge Bases.
KB server:
- New /graph and /benchmarks routers, mounted only when KG_RAG_ENABLED.
- Neo4j graph_store + LLM concept extractor adapted to string
collection IDs and request-scoped OpenAI credentials.
- New graph_indexing helper feeds Neo4j from the ChromaDB backend after
vector ingestion succeeds (fails open).
- KG-RAG query augmentation hangs off query_service.query_with_plugin
('simple_query' | 'kg_rag_query'), preserving the trace contract.
- Optional 'kg-rag' extra adds neo4j + openai.
LAMB backend:
- New knowledge_store_graph_router proxies graph + benchmark calls
through KnowledgeStoreClient using per-org token/URL resolution.
- KnowledgeStoreCreate threads graph_enabled to the KB server.
Frontend:
- New graphService.js + benchmarkService.js axios clients.
- New KnowledgeStoreGraphView + KnowledgeStoreBenchmarkView Svelte
components, mounted as tabs on KnowledgeStoreDetail when KG-RAG is
on. Curation actions (approve / reject concepts and relationships)
proxy through the LAMB router.
- CreateKnowledgeWizard exposes a Graph RAG opt-in checkbox.
Infra:
- docker-compose.next.yaml gets an optional neo4j service under
--profile kg-rag.
- .env.next.example documents the KG_RAG_* block.
Tests:
- tests/unit/test_kg_rag.py covers config parsing, concept-extraction
helpers and the plugin's three graceful-degradation paths.
Quality gates: lamb-kb-server-stable untouched; KB server starts and
all existing unit + integration tests pass without Neo4j; graph
features are off by default per-server AND per-collection.
Blocking fixes: - Benchmark route now uses a single body model with embedded ``embedding_credentials``, so the LAMB proxy's flat JSON validates (previously FastAPI's Body(embed=True) required wrapping under ``request``). Adds an EmbeddingCredentialsBody schema. - init_db now runs lightweight ALTER TABLE migrations so existing installations get the new ``graph_enabled`` column without an ``OperationalError: no such column``. Uses a direct sqlite3 connection (not the SQLAlchemy engine) so the migration check doesn't leave WAL state that breaks fork-based lock tests. Only runs against pre-existing DBs to keep the hot path zero-cost. Significant: - Permalinks now flow through to Neo4j Chunk nodes (permalink_original / permalink_full_markdown / permalink_page). Snapshot endpoint exposes them on chunk node data for citations. Moderate: - Vector backend gets public ``get_chunks_by_id``, ``get_chunks_by_source``, and ``iter_all_chunks`` surfaces. KG-RAG plugin and migration endpoint now call those instead of reaching into ChromaDB's private client cache. Default ``[]`` / NotImplementedError lets non-chromadb backends degrade cleanly. - Graph migration endpoint surfaces unsupported-backend (e.g. qdrant) as a clear 400 with explanation; the frontend hides the migrate button and explains why instead of letting the user click into an error. - ConceptExtractor passes a configurable per-request OpenAI timeout (default 60s) so a hung vendor can't pin an ingestion worker indefinitely. New ``KG_RAG_OPENAI_TIMEOUT_SECONDS`` env knob. - Graph + Benchmark tabs are now gated on ``ks.graph_enabled``, with a graceful fallback to "show tabs if the store could be migrated". Documented ingest-time slowdown in .env.next.example. Nits: - Drop unused ``_collection_dict`` helper in routers/graph.py. - Lazy-import OrganizationConfigResolver in the LAMB proxy router to match existing convention. - Move the ``reload_config`` fixture into tests/unit/conftest.py so any unit test can pick it up. Tests: - New regression tests for the proxy-flat body shape and the schema migration. Idempotent re-run check is included. Full unit (308) + integration (179 + 1 skipped) suites pass.
…reate toggle, vendor/model picker, verified-only retrieval, Sigma full-graph view) and drop the in-app benchmark surface fix(kb-v2): per-collection LLM extraction config + plugin system (OpenAI/Ollama), cascade-aware curation actions, and Cypher tightened to use only verified concepts + relationships
The question-time entity extractor was reading KG_RAG_QUESTION_EXTRACTION_MODEL env and falling back to the server-level chat_model, ignoring the per-collection extraction config picked by the user at KS creation. As a result the entity names produced at query time could be normalised differently from those stored at build time, silently zeroing the question-entity seeding path. It now resolves vendor/model/endpoint from the collection's stored extraction_* fields via LLMExtractionRegistry.build(...), the same path used by the build-time concept extractor. Cache key is now (vendor, model, question) so cross-collection lookups don't collide.
…ed KSs and surface extracted entities in Test Query UI
…ration filters
and pagination
- Fix concept verification state to be scoped per Knowledge Store via
MENTIONS.verification_state instead of the org-level Concept node,
so approving concepts in one KS does not affect other KSs
- Fix change-detection in curation tx to compare against
MENTIONS.verification_state (the per-collection value) rather than
concept.verification_state, which could be 'verified' from another KS
and silently suppress the write
- Delete associated graph data (Document, Chunk nodes, orphaned Concepts)
when a linked library item is removed from a Knowledge Store
- Add node drag support and zoom-adaptive labels to the Sigma.js graph view
- Add verification status filter (All/Unverified/Verified/Rejected) to
Concepts and Relationships sections in the curation panel
- Add client-side pagination (20 items/page) to both curation lists
- Show percentage verified in stats cards and section headers
- Optimistic in-place mutation on verify/unverify toggle to eliminate flicker
…b-integration' into feat/kg-rag-new-kbserver-arch
…m KB server refactor Lamb-Project#334
100% diff coverage for the KG-RAG / new-KB-server feature across backend, lamb-kb-server and library-manager. Lamb-Project#334
…b-integration' into feat/kg-rag-new-kbserver-arch # Conflicts: # backend/creator_interface/knowledge_store_router.py # lamb-kb-server/backend/plugins/vector_db/chromadb_backend.py # lamb-kb-server/backend/schemas/content.py # lamb-kb-server/backend/services/ingestion_service.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR ports the KG-RAG (Knowledge-Graph Retrieval-Augmented Generation) capability onto the new pluggable
lamb-kb-serverarchitecture introduced inprojects/refactor/kbserver-lamb-integration.It adds a semantic-graph layer (Neo4j) on top of the existing vector store, a
kg_rag_queryplugin that fuses graph traversal with vector retrieval (RRF), an auditable concept-extraction & curation pipeline, and a Svelte UI to explore and verify the per-collection knowledge graph.What's included
lamb-kb-server/backend) — Neo4j-backedgraph_store,graph_indexing,concept_extractionservices andkg_rag_queryplugin; graph/benchmark/curation routers; LLM extraction backends.backend/) — knowledge-store client, RAG processor and graph-proxy routers wiring KG-RAG into the LAMB core.frontend/svelte-app) — Knowledge Store UI: Graph RAG toggle (locked-at-creation), drag/zoom graph view, per-collection concept verification/curation./queryis routed through the KG-RAG plugin automatically for graph-enabled collections.docker-compose.next.yaml,.env.*.example, andDocumentation/kg-rag-deployment.mddeployment guide.lamb-kb-server,backendandlibrary-manager(100% diff coverage on feature-added lines).How to test
See
Documentation/kg-rag-deployment.md. In short: bring up the stack with thekg-ragprofile andKG_RAG_ENABLED=true, create a graph-enabled collection, ingest documents, and query — graph-enabled collections route throughkg_rag_query.Notes
projects/refactor/kbserver-lamb-integration(the new kb-server architecture this work was branched from), notmain.