Skip to content

UNIFIED_DISCORD_API.md

Latest commit

 

History

History
239 lines (191 loc) · 9.26 KB

UNIFIED_DISCORD_API.md

File metadata and controls

239 lines (191 loc) · 9.26 KB
Raw

Task: Unified Discord API Implementation

Task Overview

Objective: Refactor Discord integration to use a unified internal API for both manual (/discord-push) and automatic (hourly) synchronization, eliminating code duplication and improving architecture.

Update: The integration now uses Gateway buffering with hourly summarization. API polling is reserved for backfill/debug, but the unified syncRecentMessages() path remains the shared entry point.

Status: ✅ COMPLETED

Problem Statement

Issues Identified

  1. Code Duplication: Manual /discord-push command and automatic hourly sync had separate, duplicated logic
  2. Circular Dependencies: DiscordCommandService trying to create DiscordService instances caused "not a constructor" errors
  3. Inefficient Architecture: Webhook listener approach caused high server load and complexity
  4. Poor Message Quality: Bot messages and empty content were being processed
  5. Inconsistent Filtering: Different filtering logic between manual and automatic sync

Performance Issues

  • High CPU Usage: Frontend container using 160% CPU due to webpack dev server + ESLint warnings
  • Memory Usage: 815MB reduced to 203MB (60% improvement)
  • Complex Message Caching: Unnecessary message buffering with webhook listeners

Solution Implemented

1. Unified Internal API

Created: DiscordService.syncRecentMessages(timeRangeHours = 1)

Used By:

  • Manual command: handlePushCommand() → syncRecentMessages(1)
  • Automatic sync: SchedulerService.syncAllDiscordIntegrations() → syncRecentMessages(1)

Benefits:

  • Single source of truth for Discord message processing
  • Consistent filtering and summarization logic
  • Easier testing and maintenance
  • No code duplication

2. Fixed Circular Dependencies

Problem:

// This failed - circular dependency
const DiscordService = require('./discordService');
const discordService = new DiscordService(integrationId);

Solution:

// Pass existing instance instead of creating new one
case 'discord-push':
  return await this.commandService.handlePushCommand(this);

async handlePushCommand(discordServiceInstance = null) {
  const syncResult = await discordServiceInstance.syncRecentMessages(1);
}

3. API Polling Architecture

Before: Webhook listeners + message buffering After: Direct Discord API polling

Implementation:

// Enhanced fetchMessages with filtering
async fetchMessages(options = {}) {
  const response = await axios.get(`https://discord.com/api/v10/channels/${channelId}/messages`, {
    headers: { Authorization: `Bot ${process.env.DISCORD_BOT_TOKEN}` },
    params: { limit: options.limit || 50 }
  });

  // Filter out bot messages immediately
  const filteredMessages = response.data.filter(msg => !msg.author?.bot);
  return filteredMessages;
}

4. Improved Message Quality

Filtering Logic:

const recentMessages = messages.filter(msg => {
  const msgTime = new Date(msg.timestamp);
  const isInTimeRange = msgTime >= timeAgo;
  const isHuman = !msg.author?.bot;           // Exclude Discord bots
  const hasContent = msg.content && msg.content.trim().length > 0;  // Non-empty
  
  return isInTimeRange && isHuman && hasContent;
});

5. Integrated Periodic Scheduler

Added to existing SchedulerService.runSummarizer():

// Step 1: Sync Discord integrations (NEW)
console.log('Step 1: Syncing Discord integrations...');
await SchedulerService.syncAllDiscordIntegrations();

// Step 2: Summarize individual chat rooms  
console.log('Step 2: Summarizing individual chat rooms...');
const chatRoomSummaries = await chatSummarizerService.summarizeAllActiveChats();

6. Performance Optimizations

Docker Development:

  • Moved npm install to build time (not runtime)
  • Disabled resource-intensive ESLint warnings in development
  • Optimized webpack file watching settings
  • Reduced memory usage from 815MB → 203MB

ESLint Configuration:

rules: {
  'react/prop-types': 'off',           // Disable prop-types validation
  'react-hooks/exhaustive-deps': 'off', // Disable exhaustive deps warnings  
  'no-unused-vars': 'off',             // Disable unused vars warnings
  'max-len': 'off',                    // Disable max line length warnings
}

Code Changes Made

Files Modified

Core Services

  • backend/services/discordService.js: Added syncRecentMessages(), improved fetchMessages()
  • backend/services/discordCommandService.js: Refactored handlePushCommand() to use unified API
  • backend/services/schedulerService.js: Added syncAllDiscordIntegrations() method

Configuration & Optimization

  • frontend/Dockerfile.dev: Optimized build settings and environment variables
  • docker-compose.dev.yml: Updated development server settings
  • frontend/.env.development: Added performance optimization flags
  • frontend/.eslintrc.js: Relaxed development rules to reduce CPU usage

Removed Legacy Code

  • Removed updateMessageHistory() method (message caching no longer needed)
  • Updated terminology: "Webhook Listener" → "Auto Sync" in user-facing messages

Key Methods

New: DiscordService.syncRecentMessages(timeRangeHours = 1)

Purpose: Unified method for both manual and automatic Discord sync Features:

  • Fetches messages from Discord API
  • Applies comprehensive filtering (bots, empty content, time range)
  • Creates AI summaries via Gemini
  • Enqueues agent events for the Commonly Bot external runtime
  • Saves to DiscordSummaryHistory
  • Returns standardized result object

Updated: SchedulerService.syncAllDiscordIntegrations()

Purpose: Process all active Discord integrations during hourly sync Logic:

  1. Find all integrations with config.webhookListenerEnabled: true
  2. Initialize DiscordService for each integration
  3. Call syncRecentMessages(1) for each
  4. Log results and handle errors gracefully

Testing Results

Manual Testing

  1. ✅ /discord-push command works without "Unknown command" error
  2. ✅ No more "DiscordService is not a constructor" errors
  3. ✅ Bot messages are properly filtered out
  4. ✅ Meaningful content successfully queued and posted to Commonly pods via the Commonly Bot agent
  5. ✅ Performance improvements: memory usage reduced 60%

Architecture Verification

  1. ✅ Both manual and automatic sync use same syncRecentMessages() method
  2. ✅ No code duplication between command and scheduler
  3. ✅ Message filtering consistent across both sync types
  4. ✅ Proper error handling and logging

Deployment Considerations

Environment Setup

# Required environment variables
DISCORD_BOT_TOKEN=your_bot_token
DISCORD_CLIENT_ID=your_client_id
DISCORD_CLIENT_SECRET=your_client_secret
DISCORD_PUBLIC_KEY=your_public_key

# Development optimizations
GENERATE_SOURCEMAP=false
ESLINT_NO_DEV_ERRORS=true
CHOKIDAR_USEPOLLING=false

Service Dependencies

  • Discord API: Requires bot token with proper permissions
  • MongoDB: For integration configuration and summary history
  • Gemini API: For AI-powered message summarization
  • ngrok: For Discord webhook callbacks during development

Monitoring

  • Check Docker container CPU/memory usage: docker stats
  • Monitor Discord API rate limits (50 requests/second)
  • Track hourly sync success rates in logs
  • Verify the Commonly Bot agent posts appear in linked pods

Future Improvements

Immediate Opportunities

  1. Configurable Sync Intervals: Allow custom frequencies beyond hourly
  2. Enhanced Filtering: Add keyword-based filtering, user role filtering
  3. Better Error Recovery: Retry logic for failed Discord API calls
  4. Performance Monitoring: Track sync performance metrics

Long-term Enhancements

  1. Discord Thread Support: Handle threaded conversations
  2. Rich Media Processing: Better handling of embeds, attachments, reactions
  3. User Mapping: Link Discord users to Commonly users
  4. Advanced Analytics: Sentiment analysis, engagement scoring

Lessons Learned

Architecture Decisions

  • Prefer API polling over webhooks for predictable, scheduled tasks
  • Always check for existing code before writing new functions
  • Use unified internal APIs to eliminate duplication
  • Pass instances instead of creating new ones to avoid circular dependencies

Performance Optimization

  • Move heavy operations to build time (npm install)
  • Disable development-only linting to reduce CPU usage
  • Use volume mounting for development to avoid rebuilds
  • Monitor container resource usage regularly

Development Best Practices

  • Read existing code thoroughly before implementing
  • Refactor existing functions instead of writing duplicates
  • Test both manual and automatic flows when changing shared code
  • Update user-facing documentation when changing terminology

Conclusion

The unified Discord API implementation successfully eliminates code duplication, improves performance, and provides a cleaner architecture for Discord integration. Both manual and automatic Discord synchronization now use the same high-quality filtering and processing logic, ensuring consistent behavior and easier maintenance.

Key Achievement: From separate, duplicated code paths to a single, unified API that serves both use cases efficiently.