RAG in .NET Demo

A fully local Retrieval-Augmented Generation (RAG) system built with .NET 10, LLamaSharp, and SQLite. This application demonstrates how to build an intelligent question-answering system that can ingest documents from various sources and answer questions using a local LLM with retrieved context.

Related Blog Post

🎯 What is RAG?

Retrieval-Augmented Generation (RAG) is an AI technique that enhances Large Language Models (LLMs) by:

1. Retrieving relevant information from a knowledge base
2. Augmenting the LLM prompt with that retrieved context
3. Generating accurate, grounded answers based on your specific documents

This approach provides more accurate, up-to-date answers than relying solely on an LLM's training data.

✨ Features

• 🏠 Fully Local - No cloud APIs required, all processing happens on your machine
• 📁 Multi-Source Ingestion - Index documents from local folders and websites
• 📄 Multiple File Formats - Support for .txt, .md, .pdf, .docx, .xlsx, .pptx
• 🌐 Web Crawling - Automatically crawl and index websites
• 💾 Persistent Vector Store - SQLite-based storage for fast retrieval
• 🔍 Semantic Search - Find relevant context using vector embeddings
• 🎨 Beautiful Console UI - Built with Spectre.Console
• ⚙️ Highly Configurable - Extensive configuration options via JSON

🏗️ Architecture

┌─────────────────┐
│  Document       │
│  Sources        │
│  (Folders/URLs) │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Document       │
│  Chunking       │
│  (Overlap)      │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Embedding      │
│  Model          │
│  (nomic-embed)  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Vector Store   │
│  (SQLite)       │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  User Question  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Semantic       │
│  Search         │
│  (Top-K)        │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  LLM            │
│  (Llama 3.1)    │
│  + Context      │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Answer         │
└─────────────────┘

🚀 Getting Started

Prerequisites

• .NET 10 SDK
• Two GGUF model files:
  • Embedding Model: nomic-embed-text-v1.5.Q8_0.gguf (~275 MB)
  • LLM Model: Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf (~4.9 GB)

Installation

1. Clone the repository

   git clone <repository-url>
   cd RAGInDotNetDemo

2. Download the models

   Create a models directory and download the GGUF files:

   mkdir models
   # Download models from HuggingFace or other GGUF sources

   Example sources:

   • Nomic Embed: https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF
   • Llama 3.1: https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF

3. Configure the application

   Edit rag-config.json and update the model paths:

   {
     "model": {
       "modelPath": "C:\\path\\to\\your\\models\\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
     },
     "embedding": {
       "modelPath": "C:\\path\\to\\your\\models\\nomic-embed-text-v1.5.Q8_0.gguf"
     }
   }

   Note: The included rag-config.json uses tuned parameters optimized for comprehensive retrieval:

   • Higher topK (15) for broader context
   • Higher minScore (0.50) for quality filtering
   • Stronger repetition penalties for cleaner output

   These differ from code defaults but provide better results for most use cases.

4. Add your documents

   Place documents in the docs folder or configure additional sources in rag-config.json

5. Build and run

   dotnet build
   dotnet run --project src/RAGInDotNetDemo

⚙️ Configuration Guide: rag-config.json

The application is configured through a single JSON file with six main sections:

📚 sources - Document Sources

Defines where to ingest documents from. Supports both local folders and web URLs.

Folder Source Example:

{
  "id": "local-docs",
  "type": "folder",
  "path": ".\\docs",
  "recursive": true,
  "includePatterns": [ "*.txt", "*.md", "*.pdf", "*.docx", "*.xlsx", "*.pptx" ]
}

Parameters:

• id (string, required) - Unique identifier for this source
• type (string, required) - Must be "folder" for local directories
• path (string, required) - Path to the folder (absolute or relative)
• recursive (boolean, optional, default: true) - Whether to scan subdirectories
• includePatterns (array, optional) - File patterns to include (glob syntax)

URL Source Example:

{
  "id": "example-site",
  "type": "url",
  "url": "https://blog.example.com",
  "urlIncludePattern": "https://blog.example.com*",
  "maxDepth": 4
}

Parameters:

• id (string, required) - Unique identifier for this source
• type (string, required) - Must be "url" for web sources
• url (string, required) - Starting URL to crawl
• urlIncludePattern (string, optional) - Only crawl URLs matching this pattern (supports * wildcard)
• maxDepth (integer, optional, default: 1) - Maximum crawl depth from the starting URL

---

🤖 model - LLM Configuration

Configures the large language model used for generating answers.

{
  "modelPath": "C:\\models\\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",
  "contextSize": 4096,
  "gpuLayerCount": 0,
  "seed": 1337,
  "inferenceParameters": { ... }
}

Parameters:

• modelPath (string, required) - Full path to the GGUF model file
• contextSize (integer, default: 4096) - Maximum context window size in tokens
  • Larger values allow more context but use more memory
  • Must not exceed the model's maximum context size
• gpuLayerCount (integer, default: 0) - Number of model layers to offload to GPU
  • 0 = CPU only
  • Higher values = more GPU usage, faster inference
  • Set based on your GPU's VRAM capacity
• seed (integer, default: 1337) - Random seed for reproducible outputs

inferenceParameters - Text Generation Controls

Fine-tune how the model generates text:

{
  "maxTokens": 512,
  "temperature": 0.2,
  "topP": 0.9,
  "topK": 40,
  "repeatPenalty": 1.3,
  "frequencyPenalty": 0.5,
  "presencePenalty": 0.3,
  "penaltyCount": 256
}

Parameters:

• maxTokens (integer, default: 512) - Maximum length of generated response in tokens
  • Longer = more detailed answers but slower
• temperature (float, default: 0.2) - Controls randomness (0.0 to 2.0)
  • Lower (0.1-0.3) = more focused, deterministic responses (recommended for RAG)
  • Higher (0.7-1.0) = more creative, varied responses
• topP (float, default: 0.9) - Nucleus sampling threshold (0.0 to 1.0)
  • Only considers tokens with cumulative probability up to this value
  • Lower = more conservative word choices
• topK (integer, default: 40) - Only sample from the top K most likely tokens
  • Lower = more deterministic
  • Higher = more diverse
• repeatPenalty (float, default: 1.1) - Penalty for repeating tokens (1.0 = no penalty)
  • Higher values discourage repetition
  • The example config uses 1.3 for stronger repetition prevention
• frequencyPenalty (float, default: 0.0) - Reduces likelihood of frequently used tokens (0.0 to 2.0)
  • The example config uses 0.5 for more varied responses
• presencePenalty (float, default: 0.0) - Encourages new topics (0.0 to 2.0)
  • The example config uses 0.3 to encourage topic diversity
• penaltyCount (integer, default: 64) - How many recent tokens to consider for penalties
  • The example config uses 256 for longer-term penalty tracking

---

🔢 embedding - Embedding Model Configuration

Configures the model that converts text into vector embeddings for semantic search.

{
  "modelPath": "C:\\models\\nomic-embed-text-v1.5.Q8_0.gguf",
  "batchSize": 2048,
  "dimensions": 768
}

Parameters:

• modelPath (string, required) - Full path to the embedding model GGUF file
• batchSize (integer, required) - Context size for the embedding model
  • Affects how much text can be embedded at once
  • Typically set to 2048 for most embedding models
• dimensions (integer, required) - Vector embedding dimensionality
  • Must match your embedding model's output size
  • nomic-embed-text-v1.5 uses 768 dimensions

---

✂️ ingestion - Document Processing

Controls how documents are split into chunks for embedding.

{
  "chunkSize": 1000,
  "chunkOverlap": 200
}

Parameters:

• chunkSize (integer, default: 1000) - Maximum size of each text chunk in characters
  • Larger chunks = more context per chunk but fewer chunks
  • Smaller chunks = more granular retrieval
  • Recommended: 500-1500 characters
• chunkOverlap (integer, default: 200) - Number of overlapping characters between chunks
  • Prevents context from being lost at chunk boundaries
  • Recommended: 10-20% of chunk size

Example:

Document: "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
ChunkSize: 10, Overlap: 3

Chunk 1: "ABCDEFGHIJ"
Chunk 2: "HIJKLMNOPQ"  (HIJ overlaps)
Chunk 3: "OPQRSTUVWX"  (OPQ overlaps)

---

💾 vectorStore - Vector Database

Configures persistent storage for embedded document chunks.

{
  "persistPath": ".\\vectorstore.db",
  "defaultTopK": 15
}

Parameters:

• persistPath (string, default: ".\\vectorstore.db") - Path to SQLite database file
  • The vector store is saved here after ingestion
  • Can be absolute or relative path
• defaultTopK (integer, default: 3) - Default number of similar chunks to retrieve
  • Can be overridden by the rag.topK setting
  • The example config uses 15 for broader context retrieval

---

🎯 rag - RAG Pipeline Settings

Controls the overall RAG behavior and query answering process.

{
  "forceReingest": true,
  "topK": 15,
  "minScore": 0.50,
  "promptTemplate": "",
  "exitCommands": [ "exit", "quit" ],
  "maxSourcesDisplay": 10
}

Parameters:

• forceReingest (boolean, default: false) - Force re-ingestion on startup

  • true = Always rebuild the vector store (ignores existing cache)
  • false = Only ingest if sources changed or vector store doesn't exist
  • The example config uses true for development/testing
  • Set to false in production for better startup performance

• topK (integer, default: 3) - Number of most relevant chunks to retrieve

  • Higher values = more context but potentially more noise
  • Lower values = more focused but may miss relevant information
  • The example config uses 15 for comprehensive context retrieval
  • Recommended starting point: 3-5, then adjust based on answer quality

• minScore (float, default: 0.15) - Minimum similarity score threshold (0.0 to 1.0)

  • Only chunks with similarity score above this are used
  • Higher = stricter relevance filtering
  • Lower = more permissive (may include less relevant context)
  • The example config uses 0.50 for high-quality matches only
  • Start with 0.15 and increase if you get irrelevant results

• promptTemplate (string, optional) - Custom system prompt template

  • If empty, uses the default built-in template
  • Use {context} and {question} placeholders
  • Example:

    "promptTemplate": "You are a helpful assistant. Use this context: {context}\n\nAnswer: {question}"

• exitCommands (array, default: ["exit", "quit"]) - Commands to exit the application

  • Case-insensitive matching

• maxSourcesDisplay (integer, default: 10) - Maximum number of source citations to display

  • Limits the sources table in the UI for readability
  • All sources are still used for answer generation
  • Set to a higher value to see more sources, or lower to keep output concise

---

📖 Example Configuration Scenarios

Scenario 1: Memory-Constrained System (CPU Only)

{
  "model": {
    "modelPath": "path/to/model.gguf",
    "contextSize": 2048,
    "gpuLayerCount": 0,
    "inferenceParameters": {
      "maxTokens": 256,
      "temperature": 0.2
    }
  },
  "rag": {
    "topK": 2
  }
}

Scenario 2: High-Performance System (GPU Accelerated)

{
  "model": {
    "modelPath": "path/to/model.gguf",
    "contextSize": 8192,
    "gpuLayerCount": 35,
    "inferenceParameters": {
      "maxTokens": 1024,
      "temperature": 0.3
    }
  },
  "rag": {
    "topK": 5
  }
}

Scenario 3: Technical Documentation (Precise Answers)

{
  "model": {
    "inferenceParameters": {
      "temperature": 0.1,
      "topP": 0.8,
      "topK": 20
    }
  },
  "rag": {
    "topK": 3,
    "minScore": 0.25
  }
}

Scenario 4: Creative Content (More Variety)

{
  "model": {
    "inferenceParameters": {
      "temperature": 0.7,
      "topP": 0.95,
      "topK": 50
    }
  },
  "rag": {
    "topK": 5,
    "minScore": 0.10
  }
}

---

🧪 Testing

The project includes comprehensive unit tests covering:

• Vector Store Operations - In-memory and SQLite implementations
• Document Chunking - Overlap chunker logic
• Ingestion Pipeline - Document source processing
• RAG Pipeline - End-to-end answer generation

Run tests:

dotnet test

Run specific test class:

dotnet test --filter "FullyQualifiedName~SqliteVectorStore_Tests"

---

📁 Project Structure

RAGInDotNetDemo/
├── src/
│   └── RAGInDotNetDemo/
│       ├── Abstractions/       # Interfaces and models
│       ├── Chunking/           # Document chunking logic
│       ├── Configuration/      # Configuration models
│       ├── Embedding/          # Embedding service
│       ├── Ingestion/          # Document ingestion
│       ├── Pipeline/           # RAG pipeline
│       ├── VectorStore/        # Vector database
│       └── Program.cs          # Application entry point
├── tests/
│   └── RAGInDotNetDemo.Tests/
├── docs/                       # Sample documents for ingestion
├── models/                     # GGUF model files (not in repo)
├── rag-config.json            # Configuration file
└── vectorstore.db             # Persisted vector store

---

🔧 Troubleshooting

Issue: "Model file not found"

• Ensure modelPath and embedding.modelPath point to valid GGUF files
• Use absolute paths or ensure relative paths are correct from the executable location

Issue: Out of memory errors

• Reduce model.contextSize
• Reduce model.gpuLayerCount to 0 (CPU only)
• Reduce ingestion.chunkSize
• Reduce rag.topK

Issue: Slow inference

• Increase model.gpuLayerCount if you have a GPU
• Use a smaller quantized model (Q4_K_M instead of Q8_0)
• Reduce inferenceParameters.maxTokens

Issue: Poor answer quality

• Increase rag.topK to retrieve more context
• Lower rag.minScore to be less strict
• Adjust ingestion.chunkSize and chunkOverlap
• Lower temperature for more focused answers

Issue: Answers not based on documents

• Set forceReingest: true to rebuild the vector store
• Verify documents are in the configured source paths
• Check that file patterns in includePatterns match your files
• Increase rag.topK to retrieve more chunks

---

🤝 Contributing

Contributions are welcome! Areas for improvement:

• Additional document format support
• More vector store implementations (Qdrant, Milvus, etc.)
• Advanced chunking strategies
• Reranking support
• Hybrid search (keyword + semantic)

---

📝 License

This project is licensed under the Apache License 2.0. See terms in LICENSE.txt.

---

🙏 Acknowledgments

• LLamaSharp - .NET bindings for llama.cpp
• Spectre.Console - Beautiful console UI
• HtmlAgilityPack - HTML parsing for web ingestion
• PdfPig & DocumentFormat.OpenXml - Document parsing

---

📚 Additional Resources

• LLamaSharp Documentation
• Understanding RAG
• GGUF Model Format
• Vector Embeddings Explained

---

Built with ❤️ using .NET 10