Skip to content

Clear the rest of the fabricated case studies: ratchet 16 to 9 #591

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pftg merged 1 commit into from
Aug 22, 2026
Merged

Clear the rest of the fabricated case studies: ratchet 16 to 9 #591

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
349 changes: 0 additions & 349 deletions content/blog/2025/complete-guide-ruby-rails-ai-integration-2025.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,42 +13,6 @@ Ruby on Rails developers face a critical decision in 2025: **Which AI SDK should

This guide solves that problem. You'll learn how to integrate AI into Rails apps using battle-tested patterns, avoid costly mistakes, and deploy with confidence.

## Why Rails AI Integration Failed in 2023: Lessons from $180K in Mistakes

Before showing you how to succeed, let me share three catastrophic failures we witnessed at JetThoughts. These aren't hypothetical - they're real projects that burned serious capital.

### The $12,000 Weekend That Killed a Startup

A fintech startup integrated OpenAI into their Rails app for automated investment advice. They launched Friday afternoon. By Monday morning, their AWS bill showed $12,387 in unexpected charges.

**What happened**: Their chat endpoint had no rate limiting. A single bug caused an infinite retry loop, making 47,000 GPT-4 API calls over 36 hours. Each call cost $0.26 (8K context tokens × $0.03/1K tokens).

**The real tragedy**: They had implemented response caching - but only for `temperature: 0`. Their production code used `temperature: 0.7`, so every retry bypassed the cache. The company shut down three weeks later.

**Key lesson**: Rate limiting isn't optional. Cache keys must match ALL parameters (model, temperature, max_tokens). Never deploy AI features without cost monitoring.

### Why Caching AI Responses is Sometimes WRONG

Conventional wisdom says "cache everything to save money." We learned this is dangerously wrong.

An e-commerce client cached customer support responses for 24 hours. Their Black Friday sale changed return policies from "30 days" to "60 days." But their AI chatbot kept telling customers the old policy for 24 hours because responses were cached.

Result: 230 customer complaints, manual intervention on 150+ orders, $18K in goodwill refunds.

**Contrarian take**: Never cache AI responses that reference time-sensitive business logic. Instead, cache the underlying data (product details, policy documents) and regenerate responses when data changes.

### The Hallucination That Cost $40K in Lost Revenue

A SaaS company built an AI-powered onboarding wizard using GPT-3.5. The AI would "read" their documentation and answer setup questions.

Customer report: "The AI told me to enable `enable_legacy_mode: true` in config. This broke our entire deployment pipeline. We lost 3 days of development."

**The problem**: Their documentation had zero mentions of `enable_legacy_mode`. GPT-3.5 hallucinated a plausible-sounding configuration option. It was 100% fabricated.

**Cost**: 12 customers hit this hallucination bug during free trial. Zero converted to paid. Lost ARR: $40K (12 customers × $3.3K/year).

**Key lesson**: Implement hallucination detection BEFORE production. Validate AI outputs against source data. Use function calling (structured outputs) instead of free-form text generation.

## The Ruby AI Landscape in 2025

The Ruby community now has **three primary paths** for AI integration:
Expand Down Expand Up @@ -751,319 +715,6 @@ class BatchAiJob < ApplicationJob
end
```

## Real-World Use Cases

### Case Study 1: Semantic Search for SaaS Knowledge Base

**Client**: HR tech SaaS (anonymous, 15K customers)
**Timeline**: 6 weeks from prototype to production
**Challenge**: Users couldn't find relevant help articles (keyword search failed)

**The problem in detail**: Customer support reported 400+ weekly tickets asking "Where's the documentation for X?" Their Elasticsearch keyword search required exact terminology matches. Search for "employee onboarding" returned zero results, but "new hire setup" found the right article.

**What we tried first (and failed)**:
1. **Week 1**: Added Elasticsearch synonyms (200+ manual mappings). Improved search by 15% but maintenance nightmare.
2. **Week 2**: Tried full-text search with better ranking. Marginally better, still missed semantic matches.
3. **Week 3**: Decided to implement semantic search with LangChain.rb + pgvector.

**Implementation**:
```ruby
# 1. Add pgvector extension
rails generate migration EnablePgvector
# In migration: enable_extension 'vector'

# 2. Migrate knowledge base to vector embeddings
class Article < ApplicationRecord
include Langchain::Vectorsearch::Pgvector
vectorsearch vectorizer: :openai, model: "text-embedding-3-small"

after_commit :async_embed, on: [:create, :update]

private

def async_embed
EmbedArticleJob.perform_later(id)
end
end

# 3. Background job for embedding (avoid blocking saves)
class EmbedArticleJob < ApplicationJob
def perform(article_id)
Article.find(article_id).embed!
rescue => e
Rails.logger.error "Failed to embed article #{article_id}: #{e.message}"
retry_job wait: 5.minutes
end
end

# 4. Replace keyword search with semantic search
def search(query)
Article.similarity_search(query, k: 5)
end
```

**What went wrong during rollout**:
- **Embedding costs exceeded budget**: Initially embedded on every save. $1,200 in OpenAI costs first week (10K articles × 50 updates/day).
- **Fix**: Changed to async jobs + deduplication (only embed if content changed).
- **Search latency**: First implementation had 800ms p95 latency (users noticed).
- **Fix**: Added HNSW index (dropped to 120ms p95).

**Final results** (after 3 months):
- 60% improvement in search relevance (A/B tested user surveys)
- 40% reduction in "where's the docs?" support tickets (240 fewer/week)
- $8K/month cost (embedding + vector database)
- ROI: $25K/month saved in support agent time

**Key lesson**: Budget for 2-3x your estimated embedding costs in first month. Users update content more than you expect.

### Case Study 2: AI-Powered Customer Support Automation

**Client**: Mid-sized e-commerce platform (anonymous, $40M ARR)
**Timeline**: 8 weeks pilot → 4 months full rollout
**Challenge**: 50K monthly support tickets, 70% were repetitive FAQs burning out support team

**The business pain**: Support team turnover hit 40% annually (industry average: 25%). Exit interviews revealed "answering the same shipping questions 100 times per day" as top complaint. Traditional chatbots had 15% resolution rate (too rigid).

**What we tried first**:
1. **Zendesk macros**: Helped, but required agents to manually select templates. Saved ~10 minutes per ticket.
2. **Rule-based chatbot**: 15% auto-resolution, but generated customer complaints ("bot doesn't understand me").
3. **GPT-3.5 experiment**: Better comprehension but hallucinated order details (dangerous).

**Why we chose Claude with function calling**:
- Function calling prevented hallucinations (AI can only return real database data)
- 200K context window handles entire conversation history
- Constitutional AI reduced toxic responses

**Implementation**:
```ruby
# app/services/support_agent_service.rb
class SupportAgentService
CONFIDENCE_THRESHOLD = 0.8

def handle_ticket(ticket)
tools = [
order_lookup_tool(ticket.user),
refund_policy_tool,
shipping_status_tool,
product_info_tool
]

response = ANTHROPIC_CLIENT.messages.create(
model: "claude-3-5-sonnet-latest",
max_tokens: 1024,
tools: tools,
messages: conversation_history(ticket),
system: support_agent_system_prompt
)

# Only auto-respond if high confidence
if auto_resolvable?(response)
ticket.update(
status: :resolved,
response: format_customer_response(response),
resolved_by: "ai_agent",
resolution_time: Time.current - ticket.created_at
)
track_ai_resolution(ticket, response)
else
# Escalate with AI draft (helps human agents)
ticket.update(
status: :needs_human_review,
ai_draft: response.content.first["text"]
)
end
end

private

def auto_resolvable?(response)
response.stop_reason == "end_turn" &&
confidence_score(response) > CONFIDENCE_THRESHOLD &&
!response.content.any? { |c| c["type"] == "tool_use" && c["name"] == "escalate_to_human" }
end

def order_lookup_tool(user)
{
name: "get_order_status",
description: "Look up real-time order status and shipping info",
input_schema: {
type: "object",
properties: {
order_number: { type: "string", pattern: "^ORD-[0-9]{6}$" }
},
required: ["order_number"]
}
}
end

def track_ai_resolution(ticket, response)
AiResolutionMetric.create!(
ticket_id: ticket.id,
confidence_score: confidence_score(response),
tokens_used: response.usage["total_tokens"],
category: ticket.category
)
end
end
```

**What went wrong during pilot**:
- **Week 2**: AI auto-resolved a ticket asking for refund. Customer was furious - they wanted to CANCEL order, not get refund policy. Lost $400 sale.
- **Fix**: Added `escalate_to_human` tool for financial requests >$100.
- **Week 4**: AI gave outdated shipping policy (4-6 weeks vs new 2-3 weeks).
- **Fix**: Implemented daily policy sync job pulling from single source of truth.
- **Week 6**: Support agents complained AI drafts were "too robotic."
- **Fix**: Updated system prompt to match brand voice, added personality guidelines.

**Pilot results** (first 60 days on 10% of tickets):
- 38% auto-resolution rate (better than expected)
- 2.1 minute average resolution time (vs 4 hours human)
- Customer satisfaction: 4.2/5 (human agents: 4.5/5 - surprisingly close!)

**Full rollout results** (6 months):
- 45% of tickets auto-resolved (22,500/month)
- Support team reduced from 25 → 18 agents (through attrition, zero layoffs)
- $15K/month AI costs (Claude API + infrastructure)
- $180K/month labor costs saved (7 agents × $26K annual fully-loaded)
- **ROI**: 1100% ($180K saved / $15K cost)
- **Unexpected benefit**: Remaining agents report higher job satisfaction (handling complex cases, not repetitive FAQs)

**Key lessons**:
1. **Start with 10% pilot**: Catch edge cases before full rollout.
2. **Add "escalate to human" tool**: Let AI decide when it's unsure.
3. **Track confidence scores**: Adjust threshold based on customer satisfaction data.
4. **Treat AI as junior agent**: Review resolutions weekly, retrain on failures.

### Case Study 3: Content Generation Pipeline

**Client**: B2B SaaS content marketing agency (8-person team)
**Timeline**: 4 weeks prototype → 3 months optimization
**Challenge**: Manual blog post creation took 8-12 hours per post, bottleneck limiting agency growth

**The business context**: Agency charged $1,200/blog post (2,000+ words). Could produce max 40 posts/month with current team. Had demand for 100+ posts/month but couldn't hire fast enough (content quality control was bottleneck).

**What we tried first**:
1. **Hired more writers**: Quality inconsistent, onboarding took 6 weeks.
2. **Used GPT-3.5 drafts**: Output was generic, required 6+ hours editing (not much faster than writing from scratch).
3. **Experimented with GPT-4**: Much better quality, but needed workflow optimization.

**Why GPT-4 + Human Editorial worked**:
- GPT-4 maintains consistent brand voice with detailed prompts
- Human editors catch hallucinations and add expert insights
- 80/20 split: AI handles structure + research, humans add unique value

**Implementation**:
```ruby
# app/services/content_generator_service.rb
class ContentGeneratorService
def generate_blog_post(topic:, keywords:, target_audience:, brand_voice:, length: 2000)
# Step 1: Generate outline with keyword integration
outline = generate_outline(topic, keywords, target_audience)

# Step 2: Generate sections with examples and stats
sections = outline.map do |section_title|
generate_section(
section_title,
length: length / outline.size,
brand_voice: brand_voice,
include_examples: true
)
end

# Step 3: Generate SEO-optimized title and meta
{
title: generate_title(topic, keywords),
outline: outline,
content: sections.join("\n\n"),
seo_meta: generate_seo_metadata(topic, keywords),
internal_links: suggest_internal_links(topic),
fact_check_flags: identify_claims_to_verify(sections)
}
end

private

def generate_outline(topic, keywords, audience)
system_prompt = <<~PROMPT
You are an expert B2B SaaS content strategist.
Create outlines that address #{audience} pain points.
Integrate keywords naturally: #{keywords.join(', ')}
PROMPT

response = OpenAI::Client.new.chat(
parameters: {
model: "gpt-4o",
messages: [
{ role: "system", content: system_prompt },
{ role: "user", content: "Create detailed 5-section outline for: #{topic}" }
],
temperature: 0.7
}
)

parse_outline(response.dig("choices", 0, "message", "content"))
end

def generate_section(title, length:, brand_voice:, include_examples:)
prompt = <<~PROMPT
Write #{length}-word section titled "#{title}".
Brand voice: #{brand_voice}
#{include_examples ? 'Include 1-2 specific examples or case studies.' : ''}
Include relevant statistics (mark with [VERIFY] if unsure).
PROMPT

OpenAI::Client.new.chat(
parameters: {
model: "gpt-4o",
messages: [{ role: "user", content: prompt }],
temperature: 0.7,
max_tokens: (length * 1.5).to_i # Words to tokens conversion
}
).dig("choices", 0, "message", "content")
end

def identify_claims_to_verify(sections)
# Extract stats and claims for human fact-checking
sections.flat_map do |section|
section.scan(/\[VERIFY\].*?\./).map { |claim| claim.gsub('[VERIFY]', '').strip }
end
end
end
```

**What went wrong during prototype**:
- **Week 1**: Generated 5 test posts. 3 contained fabricated statistics. One cited a non-existent "Gartner 2024 report."
- **Fix**: Added `[VERIFY]` tags forcing human fact-checking. Implemented claim extraction.
- **Week 2**: Client complained posts were "too generic, could be about any SaaS product."
- **Fix**: Enhanced prompts with specific brand voice guidelines, added client-specific examples to system prompts.
- **Week 3**: SEO keywords felt "stuffed" and unnatural.
- **Fix**: Changed from "include these keywords X times" to "naturally integrate these concepts."

**Production workflow** (after optimization):
1. **AI Draft** (30 min): GPT-4 generates outline + initial draft
2. **Human Fact-Check** (45 min): Editor verifies all [VERIFY] claims, adds citations
3. **Human Enhancement** (90 min): Editor adds unique insights, client-specific examples, fixes voice
4. **Total**: 2.5 hours vs 8-12 hours manual writing

**Results** (after 6 months):
- 70% reduction in content creation time (8 hours → 2.5 hours average)
- 3x content output increase (40 → 120 posts/month with same team)
- $200/post AI costs (GPT-4 API usage)
- $800/post writer labor saved (5.5 hours × $145/hour fully-loaded cost)
- **Revenue impact**: $96K additional monthly revenue (80 more posts × $1,200)
- **Quality metrics**: Client retention 95% (vs 88% pre-AI), revision requests down 30%

**Unexpected challenges**:
- **AI detection tools**: Some clients worried about "AI-generated content" SEO penalties. Had to educate on human-AI collaboration model.
- **Writer morale**: Junior writers felt threatened. Had to reposition as "AI handles research, you add expert insights."
- **Overreliance risk**: Editors started skipping fact-checks (trusting AI too much). Instituted mandatory verification audits.

**Key lessons**:
1. **AI + human is better than either alone**: AI handles structure/research, humans add expertise/credibility.
2. **Always verify AI-generated facts**: Implement systematic fact-checking workflow (don't rely on manual vigilance).
3. **Brand voice requires training**: Generic prompts = generic output. Invest in voice guidelines.
4. **Measure quality, not just speed**: Track client retention and revision requests alongside production metrics.

## Monitoring and Observability

### Track AI Performance in Production
Expand Down
Loading
Loading