AgentWatch: Production-Grade AI Agent Observability Platform

AgentWatch is a high-performance, open-source AI Agent Observability & Telemetry Platform engineered for distributed multi-agent systems. It tracks LLM token consumption, real-time USD costs, process CPU/RAM footprints, task execution lifecycles, and exception stack traces—delivering zero-overhead monitoring via a non-blocking Python SDK, a high-throughput FastAPI REST & WebSockets server, Redis Pub/Sub streaming, automated multi-channel alerts, and an interactive live Terminal (TUI) Dashboard.

📋 Table of Contents

Executive Overview
Problem Statement & Business Challenges
Who Uses AgentWatch & Core Use Cases
System Architecture Blueprint
Codebase Technical Implementation Deep-Dive
Quick Start & Step-by-Step Usage Guide
Running the Terminal (TUI) Dashboard
Docker Compose Self-Hosting
Testing & Verification
Performance & Latency Benchmarks
Repository Structure Map
Documentation Hub
License & Contributing

📌 Executive Overview

Autonomous AI agents (powered by LangChain, AutoGen, CrewAI, LlamaIndex, or custom Python loops) execute multi-step reasoning, tool invocations, and API calls. Without real-time observability, agents operate as black boxes—making it impossible to track token spend, debug infinite loops, monitor memory leaks, or catch silent task failures.

AgentWatch bridges this gap by providing an end-to-end telemetry stack:

Non-Blocking SDK: Enqueues telemetry events onto background worker threads without blocking main agent execution (<0.1ms thread overhead).
Multi-Provider LLM Cost Engine: Calculates exact USD spend dynamically across OpenAI (GPT-4o), Anthropic (Claude 3.5), Google (Gemini 1.5), DeepSeek (V3/R1), Meta (Llama 3), and custom pricing models.
Resource Footprint Tracking: Auto-collects CPU % and RSS Memory (MB) using psutil.
Live Terminal (TUI) Dashboard: Renders real-time agent tables, streaming log consoles, alert panels, and cost charts directly in the terminal via Textual.
Pluggable Alerting: Dispatches multi-channel alerts (Slack, Discord, Email, Webhooks, Console) on high CPU/RAM, token spikes, task timeouts, or missing heartbeats.

🎯 Problem Statement & Business Challenges

Building and deploying multi-agent AI systems presents unique operational risks:

Black-Box Agent Execution: Traditional logging fails to capture hierarchical task execution, agent state transitions, and step-by-step reasoning.
Unpredictable LLM API Spend: Autonomous loops can consume millions of prompt/completion tokens unnoticed, causing unexpected API billing spikes.
Process Resource Bloat: Unbounded agent memory accumulation or infinite execution loops freeze host instances.
Silent Exceptions & Task Failures: Tool call failures or unhandled LLM exceptions break agent workflows without triggering alerts.
Heavyweight Enterprise Tool Overhead: Existing APM solutions require complex cloud SaaS sign-ups, high per-event pricing, or invasive code refactoring.

AgentWatch solves these challenges with a self-hostable, zero-friction, open-source platform that integrates into any Python agent script with 3 lines of code.

👥 Who Uses AgentWatch & Core Use Cases

Target Personas

AI Engineers & LLMOps Teams: Instrument multi-agent pipelines to monitor cost per agent and debug tool execution paths.
Software Engineers & Developers: Self-host local agent telemetry during development using SQLite and the Textual TUI dashboard.
DevOps & Infrastructure Teams: Deploy production monitoring using Docker Compose, PostgreSQL, Redis, and WebSockets.

Primary Use Cases

Real-Time Token & USD Cost Tracking: Monitor API spend live per model, agent, or task.
Task Lifecycle & Exception Auditing: Capture start, completion, failure states, and full Python stack traces.
System Health & Heartbeat Monitoring: Detect offline agents or hanging execution loops instantly.
Multi-Channel Alert Dispatching: Send immediate Slack/Discord notifications when an agent encounters errors or exceeds memory thresholds.

🏛 System Architecture Blueprint

flowchart TD
    subgraph AGENT_RUNTIME [Agent Execution Context]
        A1[AI Agent Script / CrewAI / LangChain] --> A2[AgentWatch Python SDK Client]
        A2 -->|Non-Blocking Queue | A3[Background Telemetry Worker]
    end

    subgraph SERVER_BACKEND [AgentWatch Backend Server]
        A3 -->|HTTP REST / JSON| B1[FastAPI API Gateway]
        B1 --> B2[SQLAlchemy 2.0 Async ORM]
        B1 --> B3[Telemetry & Cost Engine]
        B1 --> B4[Alert Threshold Engine]
        B1 --> B5[Redis Pub/Sub Manager]
    end

    subgraph STORAGE_LAYER [Persistence & Streaming]
        B2 --> C1[(PostgreSQL / SQLite WAL)]
        B5 --> C2[(Redis Event Bus)]
    end

    subgraph PRESENTATION_PLUGINS [Visualization & Notifications]
        C2 -->|WebSockets| D1[Textual Terminal TUI Dashboard]
        C2 -->|WebSockets| D2[Web Clients / Custom Dashboards]
        B4 -->|Notifications| E1[Slack / Discord / Webhook / Email]
    end

🔬 Codebase Technical Implementation Deep-Dive

The repository is organized under agentwatch/ with modular sub-systems:

1. Non-Blocking Python SDK (`agentwatch/sdk/`)

SDK Client (client.py): Provides the AgentWatch client class. It uses a thread-safe queue.Queue and background daemon thread (_telemetry_worker) to dispatch telemetry payload asynchronously over HTTP using httpx. Main-thread latency is $< 0.1\text{ms}$.
Context Manager Support: Supports with AgentWatch(...) as client: syntax for automatic agent registration and teardown.
SDK Methods:
- start_task(name, task_id=None): Registers and tracks a new task lifecycle.
- complete_task(task_id=None, result=None): Marks task completion with execution duration metrics.
- fail_task(task_id=None, error=None): Logs task failures.
- log(message, level="INFO"): Sends structured log records linked to current agent and task IDs.
- metric(cpu=None, memory=None, tokens=0, prompt_tokens=0, completion_tokens=0, cost=None, model=None): Reports resource utilization and token counts.
- error(error_type, message, stack_trace=None): Captures exception events.
System Resource Collector (collector.py): Uses psutil to sample current process CPU utilization (%) and Resident Set Size (RSS Memory in MB).
SDK Configuration (config.py): Pydantic-backed configuration model managing API targets, agent names, models, auto-system metrics, and heartbeat intervals.

2. FastAPI REST & WebSocket Backend Server (`agentwatch/server/`)

API Gateway (main.py): Initializes the FastAPI ASGI application with CORS middleware, lifespan database initialization, and router inclusions.
Pydantic v2 Schemas (schemas.py): Strict request/response validation schemas for agents, tasks, metrics, logs, errors, analytics, and alerts.
API Sub-Routers (server/api/):
- /agents: Agent registration, active listing, and /agents/heartbeat ping handling.
- /tasks: Start, complete, fail, and search agent tasks.
- /metrics: Ingest process CPU/RAM metrics and token counts.
- /logs: Query and ingest structured agent execution logs.
- /errors: Track error events and exception stack traces.
- /analytics: Aggregate token consumption, total cost, and task success/failure ratios.
- /alerts: Query active alerts and trigger evaluation rules.
- /health: System liveness and readiness probe endpoints.

3. Async SQLAlchemy ORM & Dual Database Layer (`agentwatch/database/`)

Database Models (models.py): Declarative SQLAlchemy 2.0 ORM mappings with relationships and cascade deletes: Agent, Task, Metric, Log, ErrorEvent, Alert.
Database Connection Manager (connection.py): Manages async sessions for PostgreSQL (asyncpg) and SQLite (aiosqlite configured with Write-Ahead Logging PRAGMA journal_mode=WAL and busy_timeout=5000).

4. Real-Time Token Cost & Threshold Alert Engine (`agentwatch/telemetry/`)

Multi-Model Cost Engine (cost.py): Dynamically calculates USD token costs based on model-specific prompt and completion pricing per 1M tokens:
- OpenAI: gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-3.5-turbo.
- Anthropic: claude-3-5-sonnet, claude-3-haiku, claude-3-opus.
- Google: gemini-1.5-pro, gemini-1.5-flash.
- DeepSeek: deepseek-v3, deepseek-r1.
- Meta / Open Source: llama-3-70b, llama-3-8b.
- Custom Pricing: Supports override configurations for enterprise fine-tuned models.
Threshold Alert Engine (alerts.py): Evaluates ingested metrics against configurable thresholds:
- HIGH_CPU: Triggered when process CPU % exceeds threshold (e.g. $> 80%$).
- HIGH_MEMORY: Triggered when RSS RAM exceeds limit (e.g. $> 500\text{ MB}$).
- TOKEN_SPIKE: Triggered on sudden token surges.
- TASK_TIMEOUT: Triggered when task execution duration exceeds allowed SLA.
- ERROR_EVENT: Triggered on captured exceptions.
- MISSING_HEARTBEAT: Triggered when an agent fails to ping within configured interval.

5. Redis Pub/Sub & WebSockets Streaming (`agentwatch/websocket/`)

WebSocket Connection Manager (manager.py): Manages active client connections (/ws/telemetry), broadcasting JSON telemetry events to all connected clients.
Redis Pub/Sub Bus (redis_pubsub.py): Connects server nodes across scaled instances using Redis channels (agentwatch:events), with automatic fallback to in-memory event queues when Redis is unavailable.

6. Interactive Textual Terminal (TUI) Dashboard (`agentwatch/dashboard/` & `app.py`)

Textual Terminal UI App (app.py, dashboard/app.py): Rich terminal application providing a live visual console:
- Header & Footer: Live system clock, status indicators, and keyboard shortcuts (q: Quit, r: Refresh).
- Agent DataTable: Live grid displaying active agents, current task, CPU %, RAM MB, total tokens, and USD cost.
- Real-Time Log Stream: Rich log console displaying colored agent log messages and levels.
- Alert Panel: High-visibility warning widget displaying active threshold violations.
Dashboard API Client (dashboard/api_client.py): Asynchronous HTTP/WebSocket client connecting the TUI to the backend server.

7. Outbound Plugin Notification System (`agentwatch/plugins/`)

Plugin Interface Base (base.py): Extensible base class for alert channels.
Built-In Channel Plugins:
- ConsolePlugin (console.py): Formatted stdout log outputs.
- SlackPlugin (slack.py): Incoming webhook rich Block Kit payloads.
- DiscordPlugin (discord.py): Discord Webhook embeds.
- WebhookPlugin (webhook.py): Custom HTTP POST JSON payloads.
- EmailPlugin (email.py): SMTP email alert delivery.

8. Docker Containerization & Multi-Agent Simulator (`docker/` & `examples/`)

Server Container (Dockerfile.server): Production Uvicorn/FastAPI build.
Dashboard Container (Dockerfile.dashboard): TUI dashboard container environment.
Docker Compose Spec (docker-compose.yml): Complete multi-service setup linking PostgreSQL 16, Redis 7, AgentWatch Server, and Multi-Agent Simulator.
Examples: quickstart.py (basic SDK usage) and simulated_agent.py (multi-agent workload generator).

🚀 Quick Start & Step-by-Step Usage Guide

System Prerequisites

Python: 3.10+
Git

1. Installation

# Clone Repository
git clone https://github.com/Tarunjit45/agentwatch.git
cd agentwatch

# Create & Activate Virtual Environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install Dependencies
pip install -r requirements.txt
pip install -e .

2. Launch Backend Telemetry Server

uvicorn agentwatch.server.main:app --host 0.0.0.0 --port 8000 --reload

The OpenAPI documentation will be available at http://localhost:8000/docs.

3. Instrument Your Python AI Agent

Add AgentWatch telemetry to any Python script:

import time
from agentwatch import AgentWatch

# Initialize SDK with automatic registration and background telemetry queue
with AgentWatch(agent_name="AnalysisAgent", model="gpt-4o") as client:
    # 1. Start a task
    task_id = client.start_task("Process Customer Feedback")
    client.log("Loading dataset records into memory...", level="INFO")
    
    # 2. Simulate processing and report LLM token usage & system metrics
    time.sleep(0.5)
    client.metric(
        tokens=1500,
        prompt_tokens=1000,
        completion_tokens=500,
        cpu=18.5,
        memory=240.0
    )
    client.log("Summarization completed successfully.", level="INFO")
    
    # 3. Mark task as complete
    client.complete_task(task_id, result="Processed 150 records.")

🖥️ Running the Terminal (TUI) Dashboard

Launch the live interactive terminal console:

python app.py

The TUI provides real-time streaming updates for active agents, token counts, USD costs, logs, and alerts without leaving your terminal.

🐳 Docker Compose Self-Hosting

Spin up the full production stack (PostgreSQL, Redis, AgentWatch Server, and Multi-Agent Simulator):

docker-compose -f docker/docker-compose.yml up -d

🧪 Testing & Verification

Run the full automated test suite using pytest:

pytest

Verification Summary

Total Tests: 11 passed, 0 failed (100% pass rate).
Test Modules:
- test_alerts.py: CPU/RAM thresholds and heartbeat timeout assertions.
- test_api_extensive.py: REST API validation and error response handling.
- test_cost.py: Token calculation accuracy for default and custom models.
- test_database.py: SQLAlchemy async model relationships and cascades.
- test_plugins.py: Alert plugin broadcast functionality.
- test_sdk.py: Non-blocking SDK queue and registration flow.
- test_server.py: Server health probes and telemetry lifecycle.
- test_websocket.py: WebSocket streaming and multi-client broadcasting.

⚡ Performance & Latency Benchmarks

Component	Benchmark Metric	Result	Technical Note
SDK Main Thread Latency	Overhead per log/metric	< 0.1 ms	Asynchronous non-blocking background queue puts
SDK Memory Footprint	RSS Memory Overhead	< 15 MB	Lightweight daemon thread footprint
SQLite DB Performance	Concurrent Writes	WAL Mode Active	Non-blocking multi-thread reads & fast writes
WebSocket Throughput	Real-Time Broadcasting	High Throughput	Non-blocking async event loops with Redis Pub/Sub

📁 Repository Structure Map

agentwatch/
├── .github/                # GitHub Issue & Pull Request templates
├── agentwatch/             # Core Python package
│   ├── dashboard/          # Textual TUI dashboard (app.py, api_client.py)
│   ├── database/           # Async SQLAlchemy ORM models & connection managers
│   ├── plugins/            # Outbound alert channels (Slack, Discord, Webhook, Email, Console)
│   ├── sdk/                # Non-blocking Python SDK (client.py, collector.py, config.py)
│   ├── server/             # FastAPI REST gateway & sub-routers (agents, tasks, metrics, etc.)
│   ├── telemetry/          # Multi-model cost engine & threshold alert evaluator
│   └── websocket/          # Real-time WebSocket connection manager & Redis Pub/Sub
├── docker/                 # Dockerfiles & multi-container Docker Compose definitions
├── docs/                   # Complete documentation suite (API, ARCHITECTURE, SDK, DEPLOYMENT, etc.)
├── examples/               # Sample agent scripts (quickstart.py, simulated_agent.py)
├── tests/                  # Automated pytest test suites
├── .env.example            # Sample environment variables config
├── app.py                  # Single-command launcher for Textual TUI Dashboard
├── llms.txt                # LLM context reference document
├── pyproject.toml          # Package build setup & pytest configurations
├── RELEASE_REPORT.md       # Release readiness audit report
└── requirements.txt        # Production dependencies

📖 Documentation Hub

Explore the full documentation suite in docs/:

🏛 Architecture: docs/ARCHITECTURE.md
🔌 REST & WebSocket API: docs/API.md
🐍 Python SDK Guide: docs/SDK.md
🚀 Deployment Guide: docs/DEPLOYMENT.md
⚙️ Configuration: docs/CONFIGURATION.md
🧩 Plugin System: docs/PLUGIN_GUIDE.md
🔒 Security Policy: docs/SECURITY.md
🗺️ Roadmap: docs/ROADMAP.md
❓ FAQ & Troubleshooting: docs/FAQ.md | docs/TROUBLESHOOTING.md

🤝 License & Contributing

Contributing

Contributions are welcome! Please read CONTRIBUTING.md and CODE_OF_CONDUCT.md before submitting pull requests.

License

Distributed under the MIT License. See LICENSE for details.

Navigation Menu

Repository files navigation

AgentWatch: Production-Grade AI Agent Observability Platform

📋 Table of Contents

📌 Executive Overview

🎯 Problem Statement & Business Challenges

👥 Who Uses AgentWatch & Core Use Cases

Target Personas

Primary Use Cases

🏛 System Architecture Blueprint

🔬 Codebase Technical Implementation Deep-Dive

1. Non-Blocking Python SDK (`agentwatch/sdk/`)

2. FastAPI REST & WebSocket Backend Server (`agentwatch/server/`)

3. Async SQLAlchemy ORM & Dual Database Layer (`agentwatch/database/`)

4. Real-Time Token Cost & Threshold Alert Engine (`agentwatch/telemetry/`)

5. Redis Pub/Sub & WebSockets Streaming (`agentwatch/websocket/`)

6. Interactive Textual Terminal (TUI) Dashboard (`agentwatch/dashboard/` & `app.py`)

7. Outbound Plugin Notification System (`agentwatch/plugins/`)

8. Docker Containerization & Multi-Agent Simulator (`docker/` & `examples/`)

🚀 Quick Start & Step-by-Step Usage Guide

System Prerequisites

1. Installation

2. Launch Backend Telemetry Server

3. Instrument Your Python AI Agent

🖥️ Running the Terminal (TUI) Dashboard

🐳 Docker Compose Self-Hosting

🧪 Testing & Verification

Verification Summary

⚡ Performance & Latency Benchmarks

📁 Repository Structure Map

📖 Documentation Hub

🤝 License & Contributing

Contributing

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github		.github
agentwatch		agentwatch
docker		docker
docs		docs
examples		examples
tests		tests
.editorconfig		.editorconfig
.env.example		.env.example
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
CITATIONS.cff		CITATIONS.cff
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
RELEASE_REPORT.md		RELEASE_REPORT.md
app.py		app.py
llms.txt		llms.txt
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

AgentWatch: Production-Grade AI Agent Observability Platform

📋 Table of Contents

📌 Executive Overview

🎯 Problem Statement & Business Challenges

👥 Who Uses AgentWatch & Core Use Cases

Target Personas

Primary Use Cases

🏛 System Architecture Blueprint

🔬 Codebase Technical Implementation Deep-Dive

1. Non-Blocking Python SDK (agentwatch/sdk/)

2. FastAPI REST & WebSocket Backend Server (agentwatch/server/)

3. Async SQLAlchemy ORM & Dual Database Layer (agentwatch/database/)

4. Real-Time Token Cost & Threshold Alert Engine (agentwatch/telemetry/)

5. Redis Pub/Sub & WebSockets Streaming (agentwatch/websocket/)

6. Interactive Textual Terminal (TUI) Dashboard (agentwatch/dashboard/ & app.py)

7. Outbound Plugin Notification System (agentwatch/plugins/)

8. Docker Containerization & Multi-Agent Simulator (docker/ & examples/)

🚀 Quick Start & Step-by-Step Usage Guide

System Prerequisites

1. Installation

2. Launch Backend Telemetry Server

3. Instrument Your Python AI Agent

🖥️ Running the Terminal (TUI) Dashboard

🐳 Docker Compose Self-Hosting

🧪 Testing & Verification

Verification Summary

⚡ Performance & Latency Benchmarks

📁 Repository Structure Map

📖 Documentation Hub

🤝 License & Contributing

Contributing

License

About

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

1. Non-Blocking Python SDK (`agentwatch/sdk/`)

2. FastAPI REST & WebSocket Backend Server (`agentwatch/server/`)

3. Async SQLAlchemy ORM & Dual Database Layer (`agentwatch/database/`)

4. Real-Time Token Cost & Threshold Alert Engine (`agentwatch/telemetry/`)

5. Redis Pub/Sub & WebSockets Streaming (`agentwatch/websocket/`)

6. Interactive Textual Terminal (TUI) Dashboard (`agentwatch/dashboard/` & `app.py`)

7. Outbound Plugin Notification System (`agentwatch/plugins/`)

8. Docker Containerization & Multi-Agent Simulator (`docker/` & `examples/`)