Skip to content

kanchengw/NexusGraph

BranchesTags

Repository files navigation

NexusGraph

Production-grade GraphRAG Demo with 3-path retrieval, LLM reranker, 5-metric LLM-as-Judge, dual-layer memory, HIL approval, and automated data flywheel.

Graph DB: Neo4j 5. LLM: DashScope Qwen3.6-flash. Observability: Langfuse + Prometheus + Grafana.

Architecture

System Architecture

graph TB
    subgraph Client["Client Layer"]
        U[User] --> API["API Gateway
(port 8000)"]
    end
    subgraph Online["Online Agent Layer
(FastAPI + LangGraph)"]
        OA["Online Agent
3-Path Retriever
+ Memory"]
        API --> OA
        OA --> LF[("Langfuse
LLM Trace")]
        OA --> PG_RES[(PostgreSQL
Metrics)]
    end
    subgraph Backend["Backend Agent Layer
(Offline Analysis)"]
        JA["Judge Agent
5-Metric Eval"]
        OptA["Optimizer Agent
Analysis + HIL"]
        IA["Index Agent
KB Rebuild"]
        JA --> OptA --> IA
    end
    subgraph Storage["Data Layer"]
        Neo4j[("Neo4j
Knowledge Graph")]
        PG[("PostgreSQL
Eval + Memory")]
        Prom[("Prometheus
System Metrics")]
    end
    OA -.->|read| Neo4j
    OA --> Prom
    IA --> Neo4j
    JA & OptA --> PG
    Prom --> GF[Grafana]
    PG --> GF
    LF -.->|Periodic
Snapshot| PG
Loading

Online Query Flow

flowchart TB
    C[Client] -->|POST /query| GW[API Gateway]
    subgraph Agent["Online Agent"]
        GW -->|session_id + user_id| AG[LangGraph Agent]
        AG --> MEM[memory_service.search]
        MEM -->|relevant memories| AG
        AG --> RET[graphrag_search tool]
        RET --> V[Vector: 1024d cosine]
        RET --> BM[BM25: fulltext index]
        RET --> GR[Graph: entity expand 1-2 hops]
        V & BM & GR --> FUSION["Merge & Deduplicate"]
        FUSION --> RR["LLM Reranker\nscores each chunk 0-10"]
        RR --> TOPK[Top-k selection]
        TOPK --> CTX[Build context + system prompt]
        CTX --> GEN[LLM generate response]
        MEM --> GEN
        GEN -->|async| ADD[memory_service.add]
    end
    GW -->|trace| LF[Langfuse]
    GW -->|log| PG[(PostgreSQL)]
    GEN -->|answer| GW
    GW -->|response| C
Loading

Backend Data Flywheel

flowchart TB
    subgraph Source["Data Sources"]
        RB[("RAGBench")]
        FB[("Feedback / Ratings")]
        LF[("Langfuse")]
        PR[("Prometheus")]
        ER[("evalresult")]
    end

    subgraph Judge["① Judge Agent"]
        RM[(retrievalmetric)] -->|">=200 rows"| EV[run_eval]
        EV -->|"5 metrics"| ER
    end

    subgraph Optimizer["② Optimizer Agent (HIL A + HIL B)"]
        ANL[analyze] --> SUG[suggestion]
        SUG --> HILA{Approve?}
        HILA -->|no| XX[Discard]
        VF["LLM Judge cases"] --> REP[verification report]
        REP -.-> HILB{Pass?}
        HILB -->|yes| CL[Cleanup backup]
        HILB -->|no| RBK["Restore dump + revert"]
        SUG -.->|"inject"| ANL
        RBK -.->|"inject"| ANL
    end

    subgraph Index["③ Index Agent"]
        IDX["chunk + embed + extract"]
    end

    RB --> EV
    LF & PR & ER & FB --> ANL
    HILA -->|approve| IDX
    IDX --> VF
    CL --> NEXT["Next cycle"]
Loading

Project Structure

NexusGraph/
+-- app/core/
|   +-- langgraph/          # Online Agent (graph + tools)
|   +-- graphrag/           # Retriever, Indexer
|   +-- judge_agent/        # Evaluation pipeline
|   +-- optimizer_agent/    # Analysis + LLM opt + HIL
|   +-- index_agent/        # KB construction
+-- app/api/                # REST endpoints
+-- app/models/             # SQLModel
+-- app/services/           # LLM, embeddings, memory
+-- offline_agent/          # CLI entry point
+-- scripts/                # Utility scripts
+-- grafana/                # Dashboard provisioning
+-- prometheus/             # Config
+-- docker-compose.yml      # Docker profiles

Features

Base Services (always-on infrastructure)

  • PostgreSQL (pgvector): RetrievalMetric, Feedback, EvalResult, Memory vectors
  • Prometheus + Grafana: QPS, latency, container metrics, 2 dashboards
  • cAdvisor: Container resource usage

Online Agent

  • FastAPI server with LangGraph agent
  • Two-layer memory: Short-term (session checkpoints) + Long-term (mem0 + pgvector cross-session)
  • 3-path retrieval + LLM reranker: Vector (1024d cosine) + BM25 (fulltext) + Graph (entity expand, 1-2 hops) + LLM rerank
  • Langfuse: Full trace per query

Backend Agents

Agent Directory Responsibility
Judge Agent app/core/judge_agent/ RAGBench evaluation, triggered by conversation count (default 200, configurable)
Optimizer Agent app/core/optimizer_agent/ Metrics analysis + LLM optimization + HIL approval
Index Agent app/core/index_agent/ KB construction (chunk, embed, entity extract)

Data flywheel: Judge (triggered by conversation count) -> Optimizer (HIL A) -> Index -> Verification(HIL B) -> Cleanup/Rollback, all automated via workflow.

Offline Admin UI:

A web management panel for Backend Agents (Judge / Optimizer / Index), running as a standalone FastAPI server (http://localhost:8001/admin).

Page Route Function
Dashboard /admin Flywheel status display, manual trigger, real-time SSE event feed
Sessions /admin#sessions Full lifecycle records of each flywheel iteration
Eval Results /admin#eval RAGBench 5-metric evaluation history with per-sample detail
HIL A (Optimize) /admin#optimize Review LLM optimization suggestions; approve / modify parameters / reject
HIL B (Verify) /admin#verify Review verification before/after report; pass (cleanup) or degrade (rollback)
Rollback /admin#rollback Rollback event history with restore path and reason

Quick Start

Prerequisites

  • Python 3.12+
  • Docker and Docker Compose
  • LLM API key (DashScope)

Setup

git clone https://github.com/kanchengw/NexusGraph.git
cd NexusGraph
cp .env.example .env.development
# Configure your LLM API key and backend

1. Start Infrastructure (Docker)

# Full online serving
docker compose --profile online --env-file .env.production up -d

# Data layer + monitoring for offline analysis
docker compose --profile offline up -d

2. Start Application

# Online API + Chat UI (port 8000)
python run_server.py

# Backend Agents + Admin UI (port 8001)
python run_offline_server.py

Index Knowledge Base

Choose one of two approaches:

Option A - Download pre-built release (recommended) Download from GitHub Releases:Neo4j data dump - contains a fully built Neo4j graph database with RAGBench TechQA (galileo-ai/ragbench) demo data.

docker compose --profile offline up -d neo4j
docker cp backups/neo4j.dump graphrag-neo4j:/data/
docker exec graphrag-neo4j neo4j-admin database load --from-path=/data/neo4j.dump --overwrite-destination=true
docker restart graphrag-neo4j
Artifact Count Description
Documents 1,192 TechQA train split
Chunks 63,890 512-char chunks, 64-char overlap
Entity Nodes ~8K+ Extracted from ~50 docs (deepseek-r1:8b local / qwen3.6-flash cloud)
Relationships ~15K+ Entity relations from ~50 docs

Option B - Build from scratch

# Standalone index (debug/dev only)
python scripts/run_index_agent.py --split train --max-docs 200

Local Mode (Index Agent Only)

ENABLE_LOCAL=true replaces cloud API calls only for the Index Agent.

Component Cloud (default) Local (ENABLE_LOCAL=true)
Index Agent Embedding DashScope text-embedding-v3 mxbai-embed-large-v1 (sentence-transformers)
Index Agent Entity Extraction DashScope qwen3.6-flash deepseek-r1:8b (Ollama) 
# .env.development
ENABLE_LOCAL=true
LOCAL_OLLAMA_BASE_URL=http://127.0.0.1:11434/v1
LOCAL_LLM_MODEL=deepseek-r1:8b

3. Run Offline Pipeline

After infrastructure is up and KB is indexed, run the back-end data flywheel:

# Full flywheel: eval -> analyze -> optimize (HIL A) -> index -> verify (HIL B) -> cleanup/rollback
python -m offline_agent.cli flywheel

# Step-by-step execution
python -m offline_agent.cli eval              # Judge: RAGBench evaluation
python -m offline_agent.cli analyze           # Optimizer: metric analysis
python -m offline_agent.cli optimize          # Optimizer: LLM suggestions (HIL A)
python -m offline_agent.cli index             # Index: rebuild KB chunks

The Admin UI manages the full pipeline via web (python run_offline_server.py, port 8001):

  • Sessions & events timeline - real-time pipeline progress
  • Eval results - 5-metric LLM-as-Judge scores
  • HIL A - review/approve/reject optimization suggestions
  • HIL B - review verification reports after index rebuild
  • Suggestion history - all past optimization reports

Production Deployment

Docker Compose Profiles

Profile Services Use Case
online PostgreSQL + Neo4j + Prometheus + Grafana + cAdvisor + App (FastAPI) Full stack online serving
offline PostgreSQL + Neo4j + Prometheus + Grafana + cAdvisor Data layer + monitoring for offline analysis
base Prometheus + Grafana + cAdvisor Monitoring infrastructure only

Port Map

Port Service Public Purpose
8000 FastAPI + UI Yes Online API + Chat UI
3000 Grafana Yes Dashboard UI
5432 PostgreSQL No Internal DB
7474 Neo4j Browser No Graph management
7687 Neo4j Bolt No Graph DB
9090 Prometheus No Metrics
8080 cAdvisor No Container metrics

Data Persistence

Volume Service Purpose
neo4j_data Neo4j Graph database
postgres_data PostgreSQL Metrics, feedback, eval, memory
prometheus_data Prometheus Time-series metrics
grafana_data Grafana Dashboard settings

Backup & Restore

bash scripts/backup-data.sh
bash scripts/restore-data.sh <backup.tar.gz>

Configuration

Required (.env.development)

Variable Description
LLM_BASE_URL LLM API endpoint
LLM_API_KEY LLM API key
DEFAULT_LLM_MODEL Online LLM model
EMBEDDING_MODEL Embedding model

Optional

Variable Default Description
GRAPHRAG_CHUNK_SIZE 512 Chunk size (chars)
GRAPHRAG_CHUNK_OVERLAP 64 Chunk overlap
GRAPHRAG_TOP_K 5 Top chunks per path
GRAPHRAG_ENABLE_RERANKER true LLM reranker
EVALUATION_LLM qwen-plus Judge model for eval
EVALUATION_API_KEY (same as LLM) Separate API key for eval
EVALUATION_BASE_URL (same as LLM) Separate endpoint for eval
LANGFUSE_PUBLIC_KEY - Langfuse public key
LANGFUSE_SECRET_KEY - Langfuse secret key
LANGFUSE_HOST https://cloud.langfuse.com Langfuse endpoint

Optimizer-Tunable Parameters

The Optimizer Agent can suggest changes to the following 8 dimensions:

Parameter Default Range Description
GRAPHRAG_TOP_K 5 1 - 10 Per-path retrieval count (vector / BM25 / graph each return top-K)
GRAPHRAG_CHUNK_SIZE 512 128 - 1024 Document chunk size in characters
GRAPHRAG_CHUNK_OVERLAP 64 0 - 256 Chunk overlap window
GRAPHRAG_RERANK_TOP_K 5 1 - 10 Chunks passed to LLM after reranker scoring
GRAPH_EXPAND_SOURCE_K 3 1 - 10 Top-N vector chunks used as seeds for entity graph expansion
GRAPH_EXPAND_LIMIT 50 10 - 200 Max chunks returned from entity expansion
GRAPH_MAX_HOPS 2 1 - 3 Entity graph expansion depth
BM25 Index default rebuild / n-gram config Fulltext index configuration

API Endpoints

Method Path Description
POST /api/v1/graphrag/query Retrieval query
GET /api/v1/graphrag/health Service health
POST /api/v1/graphrag/feedback Submit feedback
GET /api/v1/graphrag/feedback/stats Feedback stats

License

Apache 2.0 License Copyright 2026 kanchengw

About

Production-grade GraphRAG Demo with 3-path retrieval, LLM reranker, offline evaluation, LLM-as-Judge, dual-layer memory, and automated data flywheel with visulization dashborad and Developing UI.

Topics

langgraph graphrag agentic-workflow

Resources

Readme

License

Apache-2.0 license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases 1

v1.0.0 - Demo Data Package Latest
Jun 30, 2026

Packages

 
 
 

Contributors

Languages