diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..715ea337 --- /dev/null +++ b/.gitignore @@ -0,0 +1,72 @@ +# Environment variables and secrets +.env +.env.local +.env.*.local +*.env + +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +pip-wheel-metadata/ +share/python-wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# Virtual environments +venv/ +env/ +ENV/ +env.bak/ +venv.bak/ + +# IDEs +.vscode/ +.idea/ +*.swp +*.swo +*~ +.DS_Store + +# Database files +chroma_db/ +*.db +*.sqlite +*.sqlite3 + +# Data files (do not commit large datasets) +Data/ + +# Logs +*.log +logs/ + +# Jupyter Notebook +.ipynb_checkpoints + +# pytest +.pytest_cache/ +.coverage +htmlcov/ + +# mypy +.mypy_cache/ +.dmypy.json +dmypy.json + diff --git a/README.md b/README.md new file mode 100644 index 00000000..d2b1c751 --- /dev/null +++ b/README.md @@ -0,0 +1,111 @@ +# MedGuard: FDA-Label-Grounded Medication Safety Assistant + +## Problem Statement + +Medication errors are a leading cause of preventable harm in healthcare. Patients frequently misunderstand drug labels, take incorrect doses, ignore food-related instructions, or rely on unreliable online sources. Traditional AI chatbots are unsafe in this domain because they may hallucinate or provide unverified medical advice. + +MedGuard solves this by using **Retrieval-Augmented Generation (RAG)** over **official FDA drug labels**. Every response is grounded in regulatory-grade documentation, ensuring that users receive accurate, traceable, and safe medication information. + +## Current Status + +✅ **Working MVP** - Core RAG backend is working end-to-end (FastAPI + LangChain + ChromaDB). All API endpoints (`/ask`, `/validate`, `/schedule`) are returning the expected JSON responses, and ingestion/chunking has been tuned to improve FDA-label coverage and answer quality. + +## Getting Started + +For detailed setup instructions, please refer to [SETUP.md](SETUP.md). + +### Quickstart (Windows + Conda) + +The full openFDA label dataset is large. For a demo/hackathon, ingest a small subset (4–5 partitions) and cap the total chunks so setup finishes quickly while still proving the end-to-end RAG flow. + +## System Architecture + +MedGuard follows a safety-first Retrieval-Augmented Generation (RAG) pipeline: + +1. The user submits a medication-related question along with optional patient context (age, pregnancy, etc.). + +2. The system retrieves relevant sections (Dosage, Warnings, Contraindications, ADRs) from FDA drug labels stored in ChromaDB. + +3. A Safety & Conflict Analyzer checks for contradictions or risk factors. + +4. The retrieved FDA text is passed to the LLM via LangChain. + +5. The LLM generates a grounded answer and a structured reminder plan. + +6. The system returns a JSON response with citations and confidence. + +This architecture ensures that every output is verifiable, explainable, and compliant with medical safety requirements. + +## Technology Stack + +| Layer | Technology | +|--------------|------------| +| Language | Python | +| LLM | OpenAI GPT / Google Gemini | +| Framework | LangChain | +| Vector Store | ChromaDB | +| API Layer | FastAPI | +| Data Source | openFDA Drug Label Dataset | +| Validation | Pydantic | + +## System Flow + +1. The user submits a medication-related question. + +2. The system enriches the query with basic patient context (age, pregnancy, etc.). + +3. Relevant sections from FDA drug labels are retrieved from the vector database. + +4. Safety and conflict checks are applied to validate dosage and warnings. + +5. The LLM generates a grounded response using only the retrieved FDA data. + +6. A structured medication reminder plan is created from the dosage instructions. + +7. The final answer is returned as a JSON response with citations and confidence. + +## Future Innovations + +- **Adherence + escalation workflow**: reminders + missed-dose logic (label-only) and an optional escalation workflow (email/SMS) when high-risk keywords appear. +- **Dose Safety Engine (label-first)**: parse dose limits, frequency, max daily dose, food/alcohol rules from labels and return a structured “safe/unsafe + why” verdict with citations. +- **Multilingual label-grounded mode**: translate retrieved FDA text first, then answer in the user’s language while keeping citations to the original English sections. + +## Screenshots + +### System architecture (RAG + safety-first pipeline) +![Architecture flow](Screenshots/Flowchart.jpeg) + +### Optional workflow automation (n8n) +![n8n workflow](Screenshots/n8n%20Workflow.jpeg) + +### FastAPI docs (endpoints) +![FastAPI docs - endpoints](Screenshots/api-docs-endpoints.png) + +Swagger UI showing the available endpoints: `/ask`, `/validate`, `/schedule`. + +### FastAPI docs (schemas) +![FastAPI docs - schemas](Screenshots/api-docs-schemas.png) + +Request/response schemas used by the API. + +### Testing phase (Swagger UI) +#### Debug: `/debug/chroma` +![Debug chroma - parameters](Screenshots/testing/debug-chroma-params.png) + +![Debug chroma - response](Screenshots/testing/debug-chroma-response.png) + +#### Chat: `/ask` +![Ask - request](Screenshots/testing/ask-request.png) + +![Ask - response](Screenshots/testing/ask-response.png) + +#### Dosage validation: `/validate` +![Validate - request](Screenshots/testing/validate-request.png) + +![Validate - response](Screenshots/testing/validate-response.png) + +#### Schedule generator: `/schedule` +![Schedule - request](Screenshots/testing/schedule-request.png) + +![Schedule - response](Screenshots/testing/schedule-response.png) + diff --git a/SETUP.md b/SETUP.md new file mode 100644 index 00000000..6fd10722 --- /dev/null +++ b/SETUP.md @@ -0,0 +1,160 @@ +# Local Setup Guide + +## ⚠️ Important: Python Version Requirement + +**ChromaDB requires Python 3.8-3.12**. Python 3.13+ may have compatibility issues. + +If you're using Python 3.14, please use **Python 3.11 or 3.12** instead. See `LOCAL_SETUP.md` for details. + +## Prerequisites + +- Python 3.8-3.12 (3.11 or 3.12 recommended) +- OpenAI API key ([Get one here](https://platform.openai.com/api-keys)) +- Virtual environment (recommended) + +## Step-by-Step Setup + +### 1. Activate Virtual Environment + +**Windows (PowerShell):** +```powershell +.\venv\Scripts\Activate.ps1 +``` + +**Windows (Command Prompt):** +```cmd +venv\Scripts\activate.bat +``` + +**Linux/Mac:** +```bash +source venv/bin/activate +``` + +### 2. Install Dependencies + +```bash +pip install -r requirements.txt +``` + +### 3. Set OpenAI API Key + +**Windows (PowerShell):** +```powershell +$env:OPENAI_API_KEY="your-api-key-here" +``` + +**Windows (Command Prompt):** +```cmd +set OPENAI_API_KEY=your-api-key-here +``` + +**Linux/Mac:** +```bash +export OPENAI_API_KEY="your-api-key-here" +``` + +**Or create a `.env` file** (recommended for persistence): +```env +OPENAI_API_KEY=your-api-key-here +``` + +Then install `python-dotenv` and load it: +```bash +pip install python-dotenv +``` + +### 4. Run Data Ingestion (First Time Only) + +This populates your ChromaDB with drug label data: + +```bash +# For testing (downloads 1 partition - faster, ~5-10 minutes) +python ingest_cloud_embeddings.py --limit-partitions 1 + +# For full dataset (downloads all partitions - slower, ~30-60 minutes) +python ingest_cloud_embeddings.py +``` + +**Note:** This only needs to be run once. The data persists in `./chroma_db/` directory. + +### 5. Start the API Server + +```bash +uvicorn app.main:app --reload +``` + +The API will be available at: +- **API Base**: http://localhost:8000 +- **Interactive Docs**: http://localhost:8000/docs +- **Alternative Docs**: http://localhost:8000/redoc + +### 6. Test the API + +Visit http://localhost:8000/docs to test the endpoints interactively, or use curl: + +```bash +# Test question answering +curl -X POST "http://localhost:8000/ask" \ + -H "Content-Type: application/json" \ + -d '{"drug_name": "Aspirin", "query": "What are the side effects?"}' +``` + +## Troubleshooting + +### Missing Dependencies + +If you get import errors, make sure all dependencies are installed: +```bash +pip install -r requirements.txt --upgrade +``` + +### API Key Not Found + +Make sure the environment variable is set: +```powershell +# Check if set (PowerShell) +$env:OPENAI_API_KEY + +# Check if set (Linux/Mac) +echo $OPENAI_API_KEY +``` + +### ChromaDB Errors + +If you get ChromaDB errors, try clearing and re-ingesting: +```bash +python ingest_cloud_embeddings.py --clear-existing --limit-partitions 1 +``` + +### Port Already in Use + +If port 8000 is in use, specify a different port: +```bash +uvicorn app.main:app --reload --port 8001 +``` + +## Project Structure + +``` +supervity/ +├── app/ +│ ├── __init__.py +│ ├── main.py # FastAPI application +│ ├── chains.py # LangChain chains (QA, validation, schedule) +│ └── rag.py # Vector store configuration +├── ingest_cloud_embeddings.py # Data ingestion script +├── requirements.txt # Python dependencies +├── chroma_db/ # ChromaDB database (created after ingestion) +└── Data/ # Downloaded FDA data (created after ingestion) +``` + +## Next Steps + +1. ✅ Set up environment +2. ✅ Install dependencies +3. ✅ Set API key +4. ✅ Run ingestion +5. ✅ Start server +6. 🎉 Use the API! + diff --git a/Screenshots/Flowchart.jpeg b/Screenshots/Flowchart.jpeg new file mode 100644 index 00000000..fec3b542 Binary files /dev/null and b/Screenshots/Flowchart.jpeg differ diff --git a/Screenshots/api-docs-endpoints.png b/Screenshots/api-docs-endpoints.png new file mode 100644 index 00000000..89a7faa8 Binary files /dev/null and b/Screenshots/api-docs-endpoints.png differ diff --git a/Screenshots/api-docs-schemas.png b/Screenshots/api-docs-schemas.png new file mode 100644 index 00000000..c72eb499 Binary files /dev/null and b/Screenshots/api-docs-schemas.png differ diff --git a/Screenshots/n8n Workflow.jpeg b/Screenshots/n8n Workflow.jpeg new file mode 100644 index 00000000..e8f3f04b Binary files /dev/null and b/Screenshots/n8n Workflow.jpeg differ diff --git a/Screenshots/testing/ask-request.png b/Screenshots/testing/ask-request.png new file mode 100644 index 00000000..065e4d43 Binary files /dev/null and b/Screenshots/testing/ask-request.png differ diff --git a/Screenshots/testing/ask-response.png b/Screenshots/testing/ask-response.png new file mode 100644 index 00000000..8e128239 Binary files /dev/null and b/Screenshots/testing/ask-response.png differ diff --git a/Screenshots/testing/debug-chroma-params.png b/Screenshots/testing/debug-chroma-params.png new file mode 100644 index 00000000..b4288cea Binary files /dev/null and b/Screenshots/testing/debug-chroma-params.png differ diff --git a/Screenshots/testing/debug-chroma-response.png b/Screenshots/testing/debug-chroma-response.png new file mode 100644 index 00000000..0994669d Binary files /dev/null and b/Screenshots/testing/debug-chroma-response.png differ diff --git a/Screenshots/testing/schedule-request.png b/Screenshots/testing/schedule-request.png new file mode 100644 index 00000000..e314e314 Binary files /dev/null and b/Screenshots/testing/schedule-request.png differ diff --git a/Screenshots/testing/schedule-response.png b/Screenshots/testing/schedule-response.png new file mode 100644 index 00000000..43fe73df Binary files /dev/null and b/Screenshots/testing/schedule-response.png differ diff --git a/Screenshots/testing/validate-request.png b/Screenshots/testing/validate-request.png new file mode 100644 index 00000000..4262a4cc Binary files /dev/null and b/Screenshots/testing/validate-request.png differ diff --git a/Screenshots/testing/validate-response.png b/Screenshots/testing/validate-response.png new file mode 100644 index 00000000..d39e8336 Binary files /dev/null and b/Screenshots/testing/validate-response.png differ diff --git a/app/__init__.py b/app/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/app/chains.py b/app/chains.py new file mode 100644 index 00000000..655a821d --- /dev/null +++ b/app/chains.py @@ -0,0 +1,279 @@ +from langchain_core.prompts import PromptTemplate, ChatPromptTemplate +from langchain_core.runnables import RunnablePassthrough, RunnableLambda +from langchain_core.output_parsers import StrOutputParser, JsonOutputParser +from langchain_openai import ChatOpenAI +from app.rag import get_vector_store +from operator import itemgetter +import os +import re + +try: + from dotenv import load_dotenv # type: ignore + load_dotenv(override=False) +except Exception: + pass + + +def _norm(s: str) -> str: + return " ".join("".join(ch.lower() if ch.isalnum() else " " for ch in (s or "")).split()) + +# Risk classification rules (deterministic, based on FDA label text). +_WS_RE = re.compile(r"\s+") + + +def _norm_text(s: str) -> str: + return _WS_RE.sub(" ", (s or "").lower()).strip() + + +def compute_risk_level_from_label_text(label_text: str) -> str: + """ + Rules: + - HIGH if label mentions bleeding, allergy, pregnancy risk, organ damage, or overdose + - MEDIUM if mentions mild side effects + - else LOW + """ + t = _norm_text(label_text) + + high_terms = [ + # bleeding + "bleeding", + "bleed", + "hemorrhage", + "haemorrhage", + # allergy + "allergy", + "allergic", + "hypersensitivity", + "anaphylaxis", + # pregnancy risk + "pregnancy", + "pregnant", + "fetal", + "foetal", + "teratogenic", + # organ damage + "organ damage", + "liver damage", + "hepatic injury", + "hepatotoxic", + "hepatic failure", + "kidney damage", + "renal injury", + "renal failure", + # overdose + "overdose", + "toxicity", + "poisoning", + ] + if any(term in t for term in high_terms): + return "HIGH" + + # MEDIUM: mentions mild side effects (keep close to the requirement wording) + medium_signals = [ + "mild side effect", + "mild side effects", + "mild adverse reaction", + "mild adverse reactions", + ] + if any(sig in t for sig in medium_signals): + return "MEDIUM" + + return "LOW" +# Initialize LLM - using OpenAI only +def get_llm(): + """Get the LLM instance. Uses OpenAI gpt-4o-mini.""" + if not os.getenv("OPENAI_API_KEY"): + raise ValueError("OPENAI_API_KEY environment variable is required") + + return ChatOpenAI(model="gpt-4o-mini", temperature=0) + +# Prompts +QA_SYSTEM_PROMPT = ( + "You must only answer using the FDA drug label text provided to you. " + "Do not use any external medical knowledge. " + "If the answer is not found in the FDA label text, return this JSON exactly: " + "{{\"error\":\"Information not found in FDA label\"}}. " + "Otherwise, return a single-line valid JSON object and NOTHING else. " + "Do NOT return markdown. Do NOT use bullet symbols. Do NOT include newline characters. " + "Do NOT include explanations outside JSON." +) + +QA_USER_PROMPT = ( + "Drug: {drug}\n" + "Question: {question}\n" + "FDA Label Context: {context}\n\n" + "Required JSON format when answer IS found:\n" + "{{\"drug\":\"\",\"section\":\"\"," + "\"key_points\":[\"point 1\",\"point 2\",\"point 3\"],\"risk_level\":\"LOW | MEDIUM | HIGH\"}}\n\n" + "Rules when answer IS found:\n" + "- drug MUST equal the provided Drug value exactly\n" + "- section must be one best matching FDA section from the context (e.g., Warnings, Dosage and Administration, Contraindications)\n" + "- key_points must be 1-3 short points derived from the context (no bullets, no newlines)\n" + "- risk_level must be LOW, MEDIUM, or HIGH\n" + "Return JSON only." +) + +VALIDATION_PROMPT = """ +Check if the following dosage is safe for the given drug based on the FDA label. +Context: +{context} + +Drug: {drug} +Dosage: {dosage} + +Return a JSON with "safe": boolean and "reason": string. +""" + +SCHEDULE_PROMPT = """ +You are a smart scheduler. Your goal is to create a medication reminder schedule list based on the dosage instructions found in the FDA context. + +Context: +{context} + +Drug: {drug} +Start Time: {start_time} (ISO format or HH:MM) + +Instructions: +1. Analyze the context to find the standard frequency for the drug (e.g., "once daily", "every 6 hours", "twice a day"). +2. Calculate the specific times for reminders starting from the provided Start Time for the next 24 hours. +3. Return ONLY a JSON object with a key "schedule" containing a list of strings (times). + +Example Output: +{{ + "frequency_found": "every 6 hours", + "schedule": ["08:00", "14:00", "20:00", "02:00"] +}} + +If frequency is not found, return empty list. +""" + +def retrieve_with_filter(inputs): + """ + Custom retrieval function to apply metadata filtering for drug name. + """ + drug_name = inputs.get("drug_name") + query = inputs.get("question") + + vectorstore = get_vector_store() + + # search_kwargs filter + # Note: This assumes drug_name matches exact metadata key "drug_name" in Chroma + # We add a fallback to empty dict if null content for robustness + docs = [] + if drug_name: + # Prefer normalized matching (new ingestion stores drug_name_norm). + dn = _norm(drug_name) + try: + docs = vectorstore.similarity_search(query, k=4, filter={"drug_name_norm": dn}) + except Exception: + docs = [] + # Back-compat: older collections only have "drug_name" + if not docs: + docs = vectorstore.similarity_search(query, k=4, filter={"drug_name": drug_name}) + # Last resort: search without filter then post-filter by parsing the "Drug Name:" prefix. + if not docs: + candidates = vectorstore.similarity_search(query, k=12) + dn_norm = dn + filtered = [] + for d in candidates: + text = (d.page_content or "") + first = text.splitlines()[0] if text else "" + if first.lower().startswith("drug name:"): + name = first.split(":", 1)[1].strip() + if _norm(name) == dn_norm or (dn_norm and dn_norm in _norm(name)): + filtered.append(d) + docs = filtered[:4] + else: + # If no drug name is provided, search across all. + docs = vectorstore.similarity_search(query, k=4) + + if not docs: + return "No relevant FDA label data found." + + # Format context using metadata so the LLM sees FDA text + which section it came from. + parts = [] + for d in docs: + section = (d.metadata or {}).get("section") or "Unknown" + text = (d.page_content or "").strip() + if not text: + continue + parts.append(f"Section: {section}\n{text}") + return "\n\n".join(parts) if parts else "No relevant FDA label data found." + + +def get_label_context(drug_name: str, question: str) -> str: + """Convenience wrapper to fetch the same context string used for /ask.""" + return retrieve_with_filter({"drug_name": drug_name, "question": question}) + +def get_retrieval_chain(): + """ + Returns a runnable that takes a dictionary {drug_name, question} and returns the context string. + """ + return RunnableLambda(retrieve_with_filter) + +def get_qa_chain(): + prompt = ChatPromptTemplate.from_messages( + [ + ("system", QA_SYSTEM_PROMPT), + ("human", QA_USER_PROMPT), + ] + ) + llm = get_llm() + + # The chain expects input: {"drug_name": "...", "question": "..."} + chain = ( + { + "context": get_retrieval_chain(), + "drug": itemgetter("drug_name"), + "question": itemgetter("question"), + } + | prompt + | llm + | JsonOutputParser() + ) + return chain + +def get_validation_chain(): + prompt = PromptTemplate.from_template(VALIDATION_PROMPT) + llm = get_llm() + + # We need to construct a "question" for the retriever to find dosage info + def prepare_retrieval_input(inputs): + return { + "drug_name": inputs["drug_name"], + "question": "dosage and administration maximum daily dose precautions" + } + + chain = ( + { + "context": prepare_retrieval_input | get_retrieval_chain(), + "drug": itemgetter("drug_name"), + "dosage": itemgetter("dosage") + } + | prompt + | llm + | JsonOutputParser() + ) + return chain + +def get_schedule_chain(): + prompt = PromptTemplate.from_template(SCHEDULE_PROMPT) + llm = get_llm() + + def prepare_retrieval_input(inputs): + return { + "drug_name": inputs["drug_name"], + "question": "dosage and administration frequency schedule" + } + + chain = ( + { + "context": prepare_retrieval_input | get_retrieval_chain(), + "drug": itemgetter("drug_name"), + "start_time": itemgetter("start_time") + } + | prompt + | llm + | JsonOutputParser() + ) + return chain diff --git a/app/main.py b/app/main.py new file mode 100644 index 00000000..239fe991 --- /dev/null +++ b/app/main.py @@ -0,0 +1,147 @@ +from fastapi import FastAPI, HTTPException +from pydantic import BaseModel +from app.chains import ( + get_qa_chain, + get_validation_chain, + get_schedule_chain, + get_label_context, + compute_risk_level_from_label_text, +) +from app.rag import get_vector_store + +app = FastAPI(title="MedGuard", description="Label-safe Medication Reminder Chatbot") + +class ChatRequest(BaseModel): + drug_name: str # Added drug_name as it's required for our specific retrieval + query: str + +class ValidationRequest(BaseModel): + drug_name: str + dosage: str + +class ScheduleRequest(BaseModel): + drug_name: str + start_time: str + +@app.get("/") +async def root(): + return {"message": "MedGuard API is running. Access docs at /docs"} + +@app.get("/debug/chroma") +async def debug_chroma(sample: int = 25, scan: int = 2000): + """ + Debug endpoint: shows what's inside the Chroma collection so you can pick valid drug_name values. + + Notes: + - Each drug can generate many chunks, so the "first N" records often all belong to the same drug. + - This endpoint scans up to `scan` records (paged) to collect up to `sample` unique drug names. + """ + try: + vs = get_vector_store() + col = vs._collection + total = int(col.count()) + + target_unique = max(0, min(int(sample), 100)) + scan_limit = max(0, min(int(scan), 20000)) + if target_unique == 0: + return {"count": total, "scan": scan_limit, "sample_drugs": []} + + drugs: list[str] = [] + seen: set[str] = set() + + offset = 0 + page_size = 500 + scanned = 0 + while scanned < scan_limit and len(drugs) < target_unique and offset < total: + page = min(page_size, scan_limit - scanned) + res = col.get(limit=page, offset=offset, include=["metadatas"]) + metas = res.get("metadatas", []) or [] + for m in metas: + if not isinstance(m, dict): + continue + dn = m.get("drug_name") + if isinstance(dn, str) and dn and dn not in seen: + seen.add(dn) + drugs.append(dn) + if len(drugs) >= target_unique: + break + offset += page + scanned += page + + return {"count": total, "scan": scanned, "sample_drugs": drugs} + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) + + +@app.get("/debug/search_drug") +async def debug_search_drug(name: str, k: int = 12): + """ + Debug endpoint: search the vector store and return the drug names that appear in the top results. + Useful when you want to test a drug (e.g. "aspirin") and confirm if it's present. + """ + try: + kk = max(1, min(int(k), 50)) + vs = get_vector_store() + docs = vs.similarity_search(name, k=kk) + out = [] + seen = set() + for d in docs: + dn = (d.metadata or {}).get("drug_name") + if isinstance(dn, str) and dn and dn not in seen: + seen.add(dn) + out.append(dn) + return {"query": name, "k": kk, "matches": out} + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) + +@app.post("/ask") +async def chat_endpoint(request: ChatRequest): + """ + answers questions about a specific drug based on its FDA label. + """ + chain = get_qa_chain() + try: + response = chain.invoke({"drug_name": request.drug_name, "question": request.query}) + + # If model says "not found", return the required error JSON as-is. + if isinstance(response, dict) and response.get("error") == "Information not found in FDA label": + return response + + # Deterministically override risk_level based on FDA label text. + label_context = get_label_context(request.drug_name, request.query) + risk = compute_risk_level_from_label_text(label_context) + + if isinstance(response, dict): + response["risk_level"] = risk + # Ensure drug echoes the requested drug name (schema requirement) + response["drug"] = request.drug_name + return response + + # Fallback: if parser didn't return dict, return a safe error. + return {"error": "Information not found in FDA label"} + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) + +@app.post("/validate") +async def validate_dosage(request: ValidationRequest): + """ + Validates if a dosage is safe according to the FDA label. + """ + chain = get_validation_chain() + try: + result = chain.invoke({"drug_name": request.drug_name, "dosage": request.dosage}) + return result + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) + +@app.post("/schedule") +async def generate_schedule(request: ScheduleRequest): + """ + Generates a generic reminder schedule based on FDA frequency recommendations. + """ + chain = get_schedule_chain() + try: + result = chain.invoke({"drug_name": request.drug_name, "start_time": request.start_time}) + return result + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) diff --git a/app/rag.py b/app/rag.py new file mode 100644 index 00000000..9ca3f598 --- /dev/null +++ b/app/rag.py @@ -0,0 +1,27 @@ +import chromadb +import os +from langchain_chroma import Chroma +from langchain_openai import OpenAIEmbeddings + +try: + from dotenv import load_dotenv # type: ignore + load_dotenv(override=False) +except Exception: + pass + +PERSIST_DIRECTORY = "./chroma_db" + +def get_embedding_function(): + """Get OpenAI embedding function.""" + if not os.getenv("OPENAI_API_KEY"): + raise ValueError("OPENAI_API_KEY environment variable is required") + + return OpenAIEmbeddings(model="text-embedding-3-small") + +def get_vector_store(): + embedding_func = get_embedding_function() + return Chroma( + persist_directory=PERSIST_DIRECTORY, + embedding_function=embedding_func, + collection_name="fda_labels" + ) diff --git a/ingest_cloud_embeddings.py b/ingest_cloud_embeddings.py new file mode 100644 index 00000000..f75b1a03 --- /dev/null +++ b/ingest_cloud_embeddings.py @@ -0,0 +1,782 @@ +""" +Script to download openFDA drug label data, chunk it, generate embeddings +using OpenAI (cloud-based), and store them in ChromaDB. +""" + +import json +import glob +import os +import sys +import argparse +import requests +import zipfile +import io +import time +import math +import re +import heapq +import random +from typing import List, Optional +from langchain_text_splitters import RecursiveCharacterTextSplitter +from langchain_core.documents import Document +from langchain_chroma import Chroma +import chromadb + +try: + from dotenv import load_dotenv # type: ignore + load_dotenv(override=False) +except Exception: + pass + +DATA_DIR = "./Data" +PERSIST_DIRECTORY = "./chroma_db" +COLLECTION_NAME = "fda_labels" +DOWNLOAD_INDEX_URL = "https://api.fda.gov/download.json" + +CHUNK_SIZE = 1000 +CHUNK_OVERLAP = 200 + + +_NON_ALNUM_RE = re.compile(r"[^a-z0-9]+") + + +def normalize_drug_name(value: str) -> str: + """Normalize drug names for more reliable matching across casing/punctuation.""" + if not value: + return "" + v = value.strip().lower() + v = _NON_ALNUM_RE.sub(" ", v) + v = re.sub(r"\s+", " ", v).strip() + return v + + +def get_embedding_function(): + """ + Returns an embedding function using OpenAI. + """ + openai_key = os.getenv("OPENAI_API_KEY") + + if not openai_key: + print("ERROR: OPENAI_API_KEY environment variable is required!") + print("Please set OPENAI_API_KEY environment variable.") + sys.exit(1) + + print("Using OpenAI embeddings (text-embedding-3-small)") + from langchain_openai import OpenAIEmbeddings + return OpenAIEmbeddings(model="text-embedding-3-small") + + +def download_data(limit_partitions: Optional[int] = None): + """ + Downloads OpenFDA drug label data if not present. + + Args: + limit_partitions: If set, only download this many partitions (for testing) + """ + if not os.path.exists(DATA_DIR): + os.makedirs(DATA_DIR) + + # Check if we already have JSON files + existing_files = glob.glob(os.path.join(DATA_DIR, "**/*.json"), recursive=True) + # Some extracted datasets may contain directories named like "*.json" (e.g. a folder + # called "drug-label-0001-of-0013.json" that contains the actual file). Filter to files. + existing_files = [p for p in existing_files if os.path.isfile(p)] + if existing_files: + print(f"Found {len(existing_files)} existing JSON files in {DATA_DIR}. Skipping download.") + return + + print("Fetching OpenFDA download index...") + try: + response = requests.get(DOWNLOAD_INDEX_URL, timeout=30) + response.raise_for_status() + index_data = response.json() + + # Navigate to drug -> label + partitions = index_data.get('results', {}).get('drug', {}).get('label', {}).get('partitions', []) + + if not partitions: + print("No partitions found in OpenFDA index.") + return + + # Limit partitions if specified (for testing) + if limit_partitions: + partitions = partitions[:limit_partitions] + print(f"Limiting to first {limit_partitions} partition(s) for testing...") + + print(f"Found {len(partitions)} partition(s). Downloading...") + + for idx, part in enumerate(partitions, 1): + download_url = part['file'] + file_name = os.path.basename(download_url) + save_path = os.path.join(DATA_DIR, file_name) + + # Skip if already exists + if os.path.exists(save_path): + print(f" [{idx}/{len(partitions)}] {file_name} already exists, skipping...") + continue + + print(f" [{idx}/{len(partitions)}] Downloading {file_name}...") + file_resp = requests.get(download_url, stream=True, timeout=60) + file_resp.raise_for_status() + + # OpenFDA files are often Zipped JSONs + if download_url.endswith('.zip') or file_name.endswith('.zip'): + with zipfile.ZipFile(io.BytesIO(file_resp.content)) as z: + z.extractall(DATA_DIR) + print(f" Extracted to {DATA_DIR}") + else: + with open(save_path, 'wb') as f: + for chunk in file_resp.iter_content(chunk_size=8192): + f.write(chunk) + print(f" Saved to {save_path}") + + print("Download complete.") + + except Exception as e: + print(f"Failed to download data: {e}") + raise + + +def extract_drug_info(entry): + """ + Extracts relevant fields from a single drug entry. + """ + info = {} + + # Drug Name - try multiple fields + keep aliases for matching + drug_name = None + aliases: List[str] = [] + if 'openfda' in entry: + openfda = entry['openfda'] + if 'brand_name' in openfda and openfda['brand_name']: + if isinstance(openfda['brand_name'], list): + aliases.extend([x for x in openfda['brand_name'] if isinstance(x, str)]) + drug_name = openfda['brand_name'][0] + elif isinstance(openfda['brand_name'], str): + aliases.append(openfda['brand_name']) + drug_name = openfda['brand_name'] + elif 'generic_name' in openfda and openfda['generic_name']: + if isinstance(openfda['generic_name'], list): + aliases.extend([x for x in openfda['generic_name'] if isinstance(x, str)]) + drug_name = openfda['generic_name'][0] + elif isinstance(openfda['generic_name'], str): + aliases.append(openfda['generic_name']) + drug_name = openfda['generic_name'] + elif 'substance_name' in openfda and openfda['substance_name']: + if isinstance(openfda['substance_name'], list): + aliases.extend([x for x in openfda['substance_name'] if isinstance(x, str)]) + drug_name = openfda['substance_name'][0] + elif isinstance(openfda['substance_name'], str): + aliases.append(openfda['substance_name']) + drug_name = openfda['substance_name'] + + # Also check product_ndc + if not drug_name and 'product_ndc' in entry: + drug_name = entry['product_ndc'] + aliases.append(drug_name) + + if not drug_name: + return None # Skip if no name + + info['drug_name'] = drug_name + # Always include the primary name as an alias + if drug_name and drug_name not in aliases: + aliases.insert(0, drug_name) + # De-dupe aliases while keeping order + seen = set() + deduped: List[str] = [] + for a in aliases: + if not isinstance(a, str): + continue + key = normalize_drug_name(a) + if not key or key in seen: + continue + seen.add(key) + deduped.append(a) + info["aliases"] = deduped + + # Extract label sections. More sections = better coverage in the backend. + def _as_text(value, max_chars: int = 12000) -> Optional[str]: + if value is None: + return None + if isinstance(value, str): + v = value.strip() + return v[:max_chars] if v else None + if isinstance(value, list): + parts = [] + for x in value: + if isinstance(x, str) and x.strip(): + parts.append(x.strip()) + if len(parts) >= 3: # keep it bounded + break + if not parts: + return None + joined = "\n".join(parts) + return joined[:max_chars] + return None + + # Map openFDA keys -> (output key, human label) + section_fields = [ + ("boxed_warning", "Boxed Warning"), + ("warnings", "Warnings"), + ("warnings_and_precautions", "Warnings and Precautions"), + ("contraindications", "Contraindications"), + ("dosage_and_administration", "Dosage and Administration"), + ("dosage_forms_and_strengths", "Dosage Forms and Strengths"), + ("indications_and_usage", "Indications and Usage"), + ("adverse_reactions", "Adverse Reactions"), + ("drug_interactions", "Drug Interactions"), + ("use_in_specific_populations", "Use in Specific Populations"), + ("pregnancy", "Pregnancy"), + ("lactation", "Lactation"), + ("pediatric_use", "Pediatric Use"), + ("geriatric_use", "Geriatric Use"), + ("overdosage", "Overdosage"), + ("how_supplied", "How Supplied"), + ("storage_and_handling", "Storage and Handling"), + ("patient_counseling_information", "Patient Counseling Information"), + ("clinical_pharmacology", "Clinical Pharmacology"), + ("mechanism_of_action", "Mechanism of Action"), + ] + + text_parts = [] + has_any_section = False + for key, label in section_fields: + if key in entry: + txt = _as_text(entry.get(key)) + if txt: + info[key] = txt + text_parts.append(f"{label}: {txt}") + has_any_section = True + + # Back-compat: keep these common keys used elsewhere + if "dosage_and_administration" in info and "dosage" not in info: + info["dosage"] = info["dosage_and_administration"] + + # Full label text (very large) — keep a bounded sample as a last resort + if not has_any_section and "spl_product_data_elements" in entry: + spl_txt = _as_text(entry.get("spl_product_data_elements"), max_chars=12000) + if spl_txt: + info["full_label"] = spl_txt + text_parts.append(f"Full Label Text: {spl_txt}") + has_any_section = True + + if text_parts: + info["text_content"] = "\n\n".join(text_parts) + else: + return None + + return info + + +def create_documents(extracted_data: List[dict]) -> List[Document]: + """ + Creates LangChain Document objects from extracted drug data. + """ + documents = [] + for item in extracted_data: + drug_name = item["drug_name"] + drug_name_norm = normalize_drug_name(drug_name) + aliases = item.get("aliases", []) + # Chroma metadata must be scalar types (no lists/dicts). Keep a compact string for debugging. + aliases_str = " | ".join([a for a in aliases if isinstance(a, str)][:8]) + + # Section-based docs improve retrieval quality vs one huge merged blob. + # Include many FDA sections for better coverage. + section_order = [ + ("boxed_warning", "Boxed Warning"), + ("warnings", "Warnings"), + ("warnings_and_precautions", "Warnings and Precautions"), + ("contraindications", "Contraindications"), + ("dosage_and_administration", "Dosage and Administration"), + ("dosage_forms_and_strengths", "Dosage Forms and Strengths"), + ("indications_and_usage", "Indications and Usage"), + ("adverse_reactions", "Adverse Reactions"), + ("drug_interactions", "Drug Interactions"), + ("use_in_specific_populations", "Use in Specific Populations"), + ("pregnancy", "Pregnancy"), + ("lactation", "Lactation"), + ("pediatric_use", "Pediatric Use"), + ("geriatric_use", "Geriatric Use"), + ("overdosage", "Overdosage"), + ("how_supplied", "How Supplied"), + ("storage_and_handling", "Storage and Handling"), + ("patient_counseling_information", "Patient Counseling Information"), + ("clinical_pharmacology", "Clinical Pharmacology"), + ("mechanism_of_action", "Mechanism of Action"), + ("full_label", "Full Label Text"), + ("text_content", "FDA Label Excerpt"), + ] + + sections: List[tuple[str, str]] = [] + for key, label in section_order: + txt = item.get(key) + if isinstance(txt, str) and txt.strip(): + sections.append((label, txt)) + + # If we still have nothing, skip + if not sections: + continue + + for section_name, section_text in sections: + # IMPORTANT: + # Avoid creating header-only chunks. We store the FDA section text as the document body + # and keep drug/section identifiers in metadata. The retriever will format the context. + page = section_text + metadata = { + "drug_name": drug_name, + "drug_name_norm": drug_name_norm, + "section": section_name, + "aliases": aliases_str, + } + documents.append(Document(page_content=page, metadata=metadata)) + + return documents + + +def load_and_chunk_data( + limit_files: Optional[int] = None, + max_entries_total: Optional[int] = None, + max_entries_per_file: Optional[int] = None, + per_file_selection: str = "top", + random_seed: Optional[int] = 42, + chunk_size: int = CHUNK_SIZE, + chunk_overlap: int = CHUNK_OVERLAP, + max_chunks_total: Optional[int] = None, + include_drugs: Optional[List[str]] = None, +) -> List[Document]: + """ + Loads JSON files, extracts drug information, and chunks the documents. + """ + json_files = glob.glob(os.path.join(DATA_DIR, "**/*.json"), recursive=True) + # Filter out directories that happen to end with ".json" + json_files = sorted([p for p in json_files if os.path.isfile(p)]) + + # If the user requested a limited test run, only process the first N files. + # Note: `--limit-partitions` historically only affected downloading; we also + # use it to limit ingestion so test runs don't accidentally embed the full dataset. + if limit_files is not None: + json_files = json_files[: max(0, int(limit_files))] + print(f"Limiting ingestion to first {len(json_files)} JSON file(s) for testing...") + + if not json_files: + print(f"No JSON files found in {DATA_DIR}") + return [] + + all_documents = [] + entries_seen = 0 + include_norms = {normalize_drug_name(x) for x in (include_drugs or []) if normalize_drug_name(x)} + found_targets: set[str] = set() + + # Text splitter configuration + splitter = RecursiveCharacterTextSplitter( + chunk_size=chunk_size, + chunk_overlap=chunk_overlap, + length_function=len, + separators=["\n\n", "\n", ". ", " ", ""] + ) + + print(f"Processing {len(json_files)} JSON file(s)...") + + for file_path in json_files: + print(f" Processing {os.path.basename(file_path)}...") + per_file_seen = 0 + try: + with open(file_path, 'r', encoding='utf-8') as f: + data = json.load(f) + results = data.get('results', []) + + if not results: + print(f" No 'results' found in {file_path}") + continue + + # If we cap per-file entries, choose *better* entries per file to maximize demo quality. + # "top" (default): keep the most informative entries (warnings/contraindications/dosage + longer text) + # "first": take the first K matching entries + # "random": reservoir sample K matching entries + extracted_items = [] + k = max_entries_per_file if max_entries_per_file is not None else None + heap: list[tuple[float, int, dict]] = [] # (score, tie, info) + tie = 0 + + def score_info(info: dict) -> float: + score = 0.0 + if info.get("warnings"): + score += 5.0 + if info.get("warnings_and_precautions"): + score += 5.0 + if info.get("contraindications"): + score += 4.0 + if info.get("boxed_warning"): + score += 6.0 + if info.get("dosage"): + score += 2.0 + if info.get("overdosage"): + score += 4.0 + if info.get("pregnancy") or info.get("lactation"): + score += 3.0 + txt = info.get("text_content") or "" + score += min(3.0, len(txt) / 2000.0) + return score + + for entry in results: + if max_entries_total is not None and entries_seen >= max_entries_total: + break + info = extract_drug_info(entry) + if info: + # If an include list is provided, only keep matching entries. + if include_norms: + aliases = info.get("aliases", []) + alias_norms = [normalize_drug_name(a) for a in aliases if isinstance(a, str)] + # match if any include appears in any alias (exact or substring) + matched = False + for inc in include_norms: + for an in alias_norms: + # word-boundary-ish match to avoid overly broad substring matches + if an == inc or f" {inc} " in f" {an} ": + matched = True + found_targets.add(inc) + break + if matched: + break + if not matched: + continue + + if k is None: + extracted_items.append(info) + entries_seen += 1 + per_file_seen += 1 + else: + tie += 1 + if per_file_selection == "first": + if per_file_seen < k: + extracted_items.append(info) + entries_seen += 1 + per_file_seen += 1 + elif per_file_selection == "random": + if random_seed is not None: + random.seed(random_seed) + if per_file_seen < k: + extracted_items.append(info) + entries_seen += 1 + per_file_seen += 1 + else: + j = random.randint(0, tie - 1) + if j < k: + extracted_items[j] = info + else: + s = score_info(info) + if len(heap) < k: + heapq.heappush(heap, (s, tie, info)) + else: + if s > heap[0][0]: + heapq.heapreplace(heap, (s, tie, info)) + + if k is not None and per_file_selection == "top": + extracted_items = [t[2] for t in sorted(heap, key=lambda x: (-x[0], x[1]))] + entries_seen += len(extracted_items) + per_file_seen = len(extracted_items) + + if extracted_items: + docs = create_documents(extracted_items) + all_documents.extend(docs) + print(f" Extracted {len(extracted_items)} drug entries, created {len(docs)} documents") + + if max_entries_total is not None and entries_seen >= max_entries_total: + print(f"Reached max entries limit ({max_entries_total}). Stopping ingestion early.") + break + + if max_entries_per_file is not None and per_file_seen >= max_entries_per_file: + print(f" Reached per-file entry limit ({max_entries_per_file}). Moving to next file...") + + if include_norms and found_targets == include_norms: + print("Found all requested demo drugs. Stopping early.") + break + + except json.JSONDecodeError as e: + print(f" Error parsing JSON in {file_path}: {e}") + except Exception as e: + print(f" Error loading {file_path}: {e}") + + if not all_documents: + print("No documents created from the data files.") + return [] + + print(f"\nTotal documents before chunking: {len(all_documents)}") + print(f"Chunking with size={chunk_size}, overlap={chunk_overlap}...") + + # Chunk the documents + chunked_docs = splitter.split_documents(all_documents) + + if max_chunks_total is not None and len(chunked_docs) > max_chunks_total: + chunked_docs = chunked_docs[: max(0, int(max_chunks_total))] + print(f"Limiting to first {len(chunked_docs)} chunks for demo run...") + + print(f"Total chunks after splitting: {len(chunked_docs)}") + + return chunked_docs + + +def ingest_to_chromadb( + documents: List[Document], + embedding_function, + clear_existing: bool = False, + batch_size: int = 100, + max_retries: int = 3, + retry_wait_seconds: float = 2.0, +): + """ + Stores documents with embeddings in ChromaDB. + + Args: + documents: List of Document objects to ingest + embedding_function: Embedding function to use + clear_existing: If True, clear existing collection before ingesting + """ + if not documents: + print("No documents to ingest.") + return + + print(f"\nInitializing ChromaDB at {PERSIST_DIRECTORY}...") + + # Create or get the vector store + vectorstore = Chroma( + persist_directory=PERSIST_DIRECTORY, + embedding_function=embedding_function, + collection_name=COLLECTION_NAME + ) + + # Check if collection already has data + existing_count = vectorstore._collection.count() + if existing_count > 0: + print(f"Collection '{COLLECTION_NAME}' already has {existing_count} documents.") + + if clear_existing: + print("Clearing existing collection and re-ingesting...") + # Delete the collection and recreate + chroma_client = chromadb.PersistentClient(path=PERSIST_DIRECTORY) + try: + chroma_client.delete_collection(COLLECTION_NAME) + except: + pass + vectorstore = Chroma( + persist_directory=PERSIST_DIRECTORY, + embedding_function=embedding_function, + collection_name=COLLECTION_NAME + ) + else: + print("Appending to existing collection...") + + # Batch add documents + # Smaller batches reduce rate-limit / timeout risks for embedding APIs. + total_docs = len(documents) + total_batches = max(1, math.ceil(total_docs / max(1, batch_size))) + + print(f"\nIngesting {total_docs} chunks to ChromaDB...") + print("This may take a while depending on the number of documents and API rate limits...") + start_all = time.time() + + for batch_idx, i in enumerate(range(0, total_docs, batch_size), start=1): + batch = documents[i:i + batch_size] + batch_start = time.time() + print(f" [{batch_idx}/{total_batches}] Embedding + storing {len(batch)} chunks...", flush=True) + + success = False + for attempt in range(1, max_retries + 1): + try: + vectorstore.add_documents(batch) + success = True + break + except Exception as e: + wait = retry_wait_seconds * (2 ** (attempt - 1)) + print( + f" Error on batch {batch_idx} attempt {attempt}/{max_retries}: {e}. " + f"Retrying in {wait:.1f}s...", + flush=True, + ) + time.sleep(wait) + + if success: + done = min(i + batch_size, total_docs) + elapsed_batch = time.time() - batch_start + elapsed_all = max(0.001, time.time() - start_all) + rate = done / elapsed_all + print( + f" Ingested {done}/{total_docs} chunks " + f"(batch {batch_idx} took {elapsed_batch:.1f}s, avg {rate:.2f} chunks/s)", + flush=True, + ) + else: + print(f" Giving up on batch {batch_idx} after {max_retries} retries. Skipping...", flush=True) + + # Persist the collection (older LangChain Chroma wrappers exposed .persist()). + # Newer versions persist automatically for persistent clients, so this may not exist. + if hasattr(vectorstore, "persist"): + try: + vectorstore.persist() + except Exception as e: + print(f"Warning: failed to call vectorstore.persist(): {e}. Continuing...", flush=True) + + final_count = vectorstore._collection.count() + print(f"\nIngestion complete! Total documents in ChromaDB: {final_count}") + + +def main(): + """ + Main function to orchestrate the entire ingestion process. + """ + parser = argparse.ArgumentParser( + description="Download openFDA drug label data, chunk it, generate embeddings, and store in ChromaDB" + ) + parser.add_argument( + "--clear-existing", + action="store_true", + help="Clear existing ChromaDB collection before ingesting" + ) + parser.add_argument( + "--limit-partitions", + type=int, + default=None, + help="Limit the number of data partitions to download AND the number of local JSON files to ingest (for testing)" + ) + parser.add_argument( + "--include-drugs", + type=str, + default=None, + help="Comma-separated list of drug names to ingest (demo mode). Example: \"aspirin,ibuprofen,acetaminophen\"" + ) + parser.add_argument( + "--max-entries", + type=int, + default=None, + help="Maximum number of drug entries to process total (recommended for a 5-minute demo)" + ) + parser.add_argument( + "--max-entries-per-file", + type=int, + default=None, + help="Maximum number of drug entries to process per JSON file (recommended to spread ingestion across many partitions without taking too long)" + ) + parser.add_argument( + "--per-file-selection", + type=str, + default="top", + choices=["top", "first", "random"], + help="How to pick entries within each file when --max-entries-per-file is set. 'top' prefers richer label sections (best quality)." + ) + parser.add_argument( + "--random-seed", + type=int, + default=42, + help="Random seed used when --per-file-selection random is chosen." + ) + parser.add_argument( + "--max-chunks", + type=int, + default=None, + help="Maximum number of chunks to embed/store total (recommended for a 5-minute demo)" + ) + parser.add_argument( + "--chunk-size", + type=int, + default=CHUNK_SIZE, + help="Chunk size used for splitting label text (larger = fewer chunks)" + ) + parser.add_argument( + "--chunk-overlap", + type=int, + default=CHUNK_OVERLAP, + help="Chunk overlap used for splitting label text" + ) + parser.add_argument( + "--batch-size", + type=int, + default=100, + help="Batch size for embedding + storage (smaller is safer for rate limits)" + ) + parser.add_argument( + "--max-retries", + type=int, + default=3, + help="Max retries per batch on transient API/network errors" + ) + parser.add_argument( + "--retry-wait-seconds", + type=float, + default=2.0, + help="Initial wait before retrying a failed batch (uses exponential backoff)" + ) + + args = parser.parse_args() + + print("=" * 60) + print("OpenFDA Drug Label Data Ingestion with Cloud Embeddings") + print("=" * 60) + print() + + # Check for API key + openai_key = os.getenv("OPENAI_API_KEY") + + if not openai_key: + print("ERROR: OPENAI_API_KEY environment variable is required!") + print("Please set OPENAI_API_KEY environment variable.") + sys.exit(1) + + # Get embedding function + embedding_function = get_embedding_function() + print() + + # Step 1: Download data + print("Step 1: Downloading OpenFDA drug label data...") + print("-" * 60) + try: + download_data(limit_partitions=args.limit_partitions) + except Exception as e: + print(f"Download failed: {e}") + sys.exit(1) + print() + + # Step 2: Load and chunk data + print("Step 2: Loading and chunking data...") + print("-" * 60) + include_list = None + if args.include_drugs: + include_list = [x.strip() for x in args.include_drugs.split(",") if x.strip()] + documents = load_and_chunk_data( + limit_files=args.limit_partitions, + max_entries_total=args.max_entries, + max_entries_per_file=args.max_entries_per_file, + per_file_selection=args.per_file_selection, + random_seed=args.random_seed, + chunk_size=args.chunk_size, + chunk_overlap=args.chunk_overlap, + max_chunks_total=args.max_chunks, + include_drugs=include_list, + ) + + if not documents: + print("No documents to process. Exiting.") + sys.exit(1) + print() + + # Step 3: Generate embeddings and store in ChromaDB + print("Step 3: Generating embeddings and storing in ChromaDB...") + print("-" * 60) + ingest_to_chromadb( + documents, + embedding_function, + clear_existing=args.clear_existing, + batch_size=args.batch_size, + max_retries=args.max_retries, + retry_wait_seconds=args.retry_wait_seconds, + ) + + print() + print("=" * 60) + print("Ingestion process completed successfully!") + print("=" * 60) + + +if __name__ == "__main__": + main() + diff --git a/instructions.pdf b/instructions.pdf new file mode 100644 index 00000000..6bbfe247 Binary files /dev/null and b/instructions.pdf differ diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 00000000..ee1fcd35 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,20 @@ + +fastapi>=0.104.0 +uvicorn[standard]>=0.24.0 + + +langchain-core>=0.1.0 +langchain>=0.1.0 +langchain-community>=0.0.20 +langchain-openai>=0.0.5 +langchain-chroma>=0.1.0 +langchain-text-splitters>=0.0.1 + + +chromadb>=0.4.0 + +# Data handling +pydantic>=2.0.0 +pydantic-settings>=2.0.0 +python-multipart>=0.0.6 +requests>=2.31.0 diff --git a/setup_local.bat b/setup_local.bat new file mode 100644 index 00000000..b65b7ba3 --- /dev/null +++ b/setup_local.bat @@ -0,0 +1,54 @@ +@echo off +REM MedGuard Local Setup Script for Windows + +echo ============================================================ +echo MedGuard Local Setup +echo ============================================================ +echo. + +REM Check if virtual environment is activated +python -c "import sys; exit(0 if hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix) else 1)" 2>nul +if %errorlevel% neq 0 ( + echo Activating virtual environment... + call venv\Scripts\activate.bat +) + +echo. +echo Step 1: Installing dependencies... +echo ------------------------------------------------------------ +pip install -r requirements.txt +if %errorlevel% neq 0 ( + echo ERROR: Failed to install dependencies + pause + exit /b 1 +) + +echo. +echo Step 2: Checking setup... +echo ------------------------------------------------------------ +python check_setup.py +if %errorlevel% neq 0 ( + echo. + echo WARNING: Some checks failed. Please review the output above. + pause + exit /b 1 +) + +echo. +echo ============================================================ +echo Setup Complete! +echo ============================================================ +echo. +echo Next steps: +echo 1. Set your OpenAI API key: +echo set OPENAI_API_KEY=your-api-key-here +echo. +echo 2. Run data ingestion (first time only): +echo python ingest_cloud_embeddings.py --limit-partitions 1 +echo. +echo 3. Start the API server: +echo uvicorn app.main:app --reload +echo. +pause + + diff --git a/start_server.ps1 b/start_server.ps1 new file mode 100644 index 00000000..d1c4a23d --- /dev/null +++ b/start_server.ps1 @@ -0,0 +1,8 @@ +# Start MedGuard Server with Conda Environment +# Usage: .\start_server.ps1 + +Write-Host "Starting MedGuard API server with medguard conda environment..." -ForegroundColor Cyan + +# Use conda run to execute commands in the medguard environment +conda run -n medguard uvicorn app.main:app --reload +