-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathrag-problems-world.html
More file actions
143 lines (134 loc) · 9.99 KB
/
Copy pathrag-problems-world.html
File metadata and controls
143 lines (134 loc) · 9.99 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
<style>
.cat { font-size:13px; font-weight:500; color:var(--color-text-secondary); text-transform:uppercase; letter-spacing:.04em; margin:1.5rem 0 .6rem; padding-left:4px; }
.prob-grid { display:grid; grid-template-columns:repeat(auto-fit,minmax(280px,1fr)); gap:10px; margin-bottom:.5rem; }
.prob { border:0.5px solid var(--color-border-tertiary); border-radius:var(--border-radius-md); padding:12px 14px; cursor:pointer; transition:border-color .15s; }
.prob:hover { border-color:var(--color-border-secondary); }
.prob-title { font-size:13px; font-weight:500; color:var(--color-text-primary); margin:0 0 4px; display:flex; align-items:center; gap:8px; }
.prob-desc { font-size:12px; color:var(--color-text-secondary); margin:0; line-height:1.5; }
.badge { font-size:10px; font-weight:500; padding:2px 7px; border-radius:20px; white-space:nowrap; }
.sev-high { background:var(--color-background-danger); color:var(--color-text-danger); }
.sev-med { background:var(--color-background-warning); color:var(--color-text-warning); }
.sev-low { background:var(--color-background-secondary); color:var(--color-text-secondary); }
</style>
<h2 class="sr-only">All RAG problems the world currently faces, grouped by category</h2>
<div class="cat">Retrieval quality</div>
<div class="prob-grid">
<div class="prob" onclick="sendPrompt('Deep dive: pure vector search fails on enterprise corpora')">
<p class="prob-title"><span class="badge sev-high">Critical</span> Pure vector search misses exact matches</p>
<p class="prob-desc">Semantic embeddings fail on codes, product IDs, names, and domain-specific terms. BM25 ignored by most teams.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: top-k retrieval returns wrong chunks')">
<p class="prob-title"><span class="badge sev-high">Critical</span> Wrong chunks returned at top-k</p>
<p class="prob-desc">Cosine similarity doesn't equal relevance. Peripheral paragraphs outrank the actual answer chunk.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: multi-hop retrieval in RAG')">
<p class="prob-title"><span class="badge sev-high">Critical</span> Multi-hop queries fail completely</p>
<p class="prob-desc">Questions requiring two or more retrieval steps — "who owns the service that handles X" — break standard RAG.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: query rewriting in RAG')">
<p class="prob-title"><span class="badge sev-med">High</span> Query-document vocabulary mismatch</p>
<p class="prob-desc">User asks "how do I reset my password" but docs say "account recovery". No query rewriting = miss.</p>
</div>
</div>
<div class="cat">Context assembly</div>
<div class="prob-grid">
<div class="prob" onclick="sendPrompt('Deep dive: context window poisoning in RAG')">
<p class="prob-title"><span class="badge sev-high">Critical</span> Context window poisoning</p>
<p class="prob-desc">Retrieved chunks are individually relevant but collectively contradictory or incoherent. LLM hallucinates to fill gaps.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: duplicate chunks in RAG context')">
<p class="prob-title"><span class="badge sev-high">Critical</span> Duplicate and redundant chunks</p>
<p class="prob-desc">Same fact appears 4 times in context, wasting tokens and confusing the model's weighting of evidence.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: lost in the middle problem in LLMs')">
<p class="prob-title"><span class="badge sev-med">High</span> Lost-in-the-middle ordering</p>
<p class="prob-desc">LLMs attend to beginning and end of context. Key evidence buried in the middle is effectively invisible.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: chunking destroys document structure')">
<p class="prob-title"><span class="badge sev-med">High</span> Chunking destroys structure</p>
<p class="prob-desc">Fixed-size chunking splits tables, severs headers from content, and breaks parent-child relationships in docs.</p>
</div>
</div>
<div class="cat">Ingestion pipeline</div>
<div class="prob-grid">
<div class="prob" onclick="sendPrompt('Deep dive: parsing complex enterprise documents for RAG')">
<p class="prob-title"><span class="badge sev-high">Critical</span> Unstructured format chaos</p>
<p class="prob-desc">PDFs, scanned docs, Excel tables, email threads, Slack exports — all parsed differently, all losing signal.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: metadata poverty in RAG systems')">
<p class="prob-title"><span class="badge sev-med">High</span> Metadata poverty</p>
<p class="prob-desc">Chunks have no author, date, source, confidence, or domain tag. Retrieval can't filter or weight by recency or authority.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: stale embeddings and knowledge freshness in RAG')">
<p class="prob-title"><span class="badge sev-med">High</span> No incremental sync</p>
<p class="prob-desc">Document updated on Monday, embedding updated never. Agents confidently return stale answers.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: embedding model choice and its impact')">
<p class="prob-title"><span class="badge sev-low">Medium</span> Embedding model mismatch</p>
<p class="prob-desc">General-purpose embeddings trained on web text fail on legal, medical, or code-heavy enterprise corpora.</p>
</div>
</div>
<div class="cat">Latency and reliability</div>
<div class="prob-grid">
<div class="prob" onclick="sendPrompt('Deep dive: RAG latency killing agentic applications')">
<p class="prob-title"><span class="badge sev-high">Critical</span> Retrieval latency kills agents</p>
<p class="prob-desc">800ms–3s per retrieval call. Agents doing 10 calls per task = 30s response time. Unusable in production.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: no fallback when RAG retrieval fails')">
<p class="prob-title"><span class="badge sev-med">High</span> No graceful fallback</p>
<p class="prob-desc">When retrieval returns nothing useful, the agent either hallucinates or returns an empty response. No fallback chain.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: RAG systems under concurrent load')">
<p class="prob-title"><span class="badge sev-med">High</span> No semantic caching</p>
<p class="prob-desc">"What is our refund policy?" asked 200 times a day triggers 200 vector searches. Zero reuse of prior results.</p>
</div>
</div>
<div class="cat">Observability and eval</div>
<div class="prob-grid">
<div class="prob" onclick="sendPrompt('Deep dive: RAG evaluation and measuring retrieval quality')">
<p class="prob-title"><span class="badge sev-high">Critical</span> No way to measure retrieval quality</p>
<p class="prob-desc">Teams can't tell if retrieval recall is 40% or 90%. Quality degrades silently over months, discovered only by complaints.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: hallucination detection in RAG')">
<p class="prob-title"><span class="badge sev-high">Critical</span> Hallucination attribution is blind</p>
<p class="prob-desc">LLM gives a wrong answer. Was it bad retrieval? Bad context assembly? Bad generation? Nobody knows.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: RAG production monitoring and drift detection')">
<p class="prob-title"><span class="badge sev-med">High</span> No drift detection</p>
<p class="prob-desc">Corpus grows, query patterns shift, embedding model changes — no system monitors performance regression over time.</p>
</div>
</div>
<div class="cat">Enterprise and multi-tenancy</div>
<div class="prob-grid">
<div class="prob" onclick="sendPrompt('Deep dive: multi-tenant RAG data isolation')">
<p class="prob-title"><span class="badge sev-high">Critical</span> Data leakage across tenants</p>
<p class="prob-desc">Shared vector indexes mean tenant A's query can surface tenant B's documents. Catastrophic for enterprise.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: access control in RAG systems')">
<p class="prob-title"><span class="badge sev-high">Critical</span> No document-level ACLs</p>
<p class="prob-desc">HR policy docs and financial data flow through the same retrieval pipeline with no permission checks.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: PII in RAG pipelines')">
<p class="prob-title"><span class="badge sev-med">High</span> PII leaking through retrieval</p>
<p class="prob-desc">Customer emails indexed verbatim. Agent returns chunks containing SSNs, phone numbers, and addresses.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: audit trails and compliance in RAG')">
<p class="prob-title"><span class="badge sev-med">High</span> No audit trail</p>
<p class="prob-desc">Compliance teams can't answer "what documents was this response based on?" No retrieval provenance logging.</p>
</div>
</div>
<div class="cat">Agentic-specific failures</div>
<div class="prob-grid">
<div class="prob" onclick="sendPrompt('Deep dive: RAG in agentic loops and multi-step reasoning')">
<p class="prob-title"><span class="badge sev-high">Critical</span> Not designed for agentic loops</p>
<p class="prob-desc">Standard RAG is one-shot. Agents need iterative retrieval — retrieve, reason, decide what to retrieve next.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: tool coupling in agentic RAG')">
<p class="prob-title"><span class="badge sev-med">High</span> Tight coupling to single framework</p>
<p class="prob-desc">LangChain RAG doesn't plug into CrewAI. LlamaIndex patterns don't transfer. Every framework = rewrite.</p>
</div>
<div class="prob" onclick="sendPrompt('Deep dive: routing in multi-corpus RAG')">
<p class="prob-title"><span class="badge sev-med">High</span> No intelligent corpus routing</p>
<p class="prob-desc">Agent doesn't know whether to search the HR wiki, the product docs, or the Slack history. Searches all or picks wrong one.</p>
</div>
</div>