openclaw-langsmith

LangSmith tracing plugin for OpenClaw. Automatically traces agent turns, tool calls, and LLM invocations to LangSmith for observability, debugging, and cost tracking.

Features

• Agent turn tracing — Each agent turn becomes a LangSmith run with prompt, response, and token usage
• Token tracking — Prompt tokens, completion tokens, and total tokens displayed in LangSmith dashboard
• Smart tagging — Auto-tags traces with source (cron, discord, slack, telegram), job names, channel IDs
• Tool call tracing — Tool calls nested under their parent agent run
• Engram LLM tracing — Memory extraction/consolidation calls appear as LLM runs with full prompts
• Batch queue — Operations batched for efficient API usage (configurable interval and size)
• Per-feature toggles — Enable/disable each trace type independently
• Zero runtime dependencies — Uses native fetch and crypto.randomUUID()
• Error isolation — Tracing errors never affect gateway operation

Quick Start

1. Get a LangSmith API Key

1. Sign up at smith.langchain.com
2. Go to Settings > API Keys
3. Create a new API key (starts with lsv2_pt_...)

2. Install the Plugin

cd ~/.openclaw/extensions
git clone https://github.com/joshuaswarren/openclaw-langsmith.git
cd openclaw-langsmith
npm install && npm run build

3. Add API Key to Gateway Environment

The gateway needs the API key in its environment. Choose your platform:

macOS (launchd)

Edit ~/Library/LaunchAgents/ai.openclaw.gateway.plist and add inside EnvironmentVariables:

<key>LANGSMITH_API_KEY</key>
<string>lsv2_pt_your_key_here</string>

Linux (systemd)

Edit ~/.config/systemd/user/openclaw-gateway.service and add to the [Service] section:

Environment="LANGSMITH_API_KEY=lsv2_pt_your_key_here"

Or create an environment file at ~/.config/openclaw/env:

LANGSMITH_API_KEY=lsv2_pt_your_key_here

Then reference it in the service file:

EnvironmentFile=%h/.config/openclaw/env

Docker

Add to your docker-compose.yml or pass via -e:

environment:
  - LANGSMITH_API_KEY=lsv2_pt_your_key_here

4. Enable in openclaw.json

{
  "plugins": {
    "allow": ["openclaw-langsmith"],
    "entries": {
      "openclaw-langsmith": {
        "enabled": true,
        "config": {
          "langsmithApiKey": "${LANGSMITH_API_KEY}",
          "projectName": "openclaw"
        }
      }
    }
  }
}

5. Restart Gateway

macOS:

launchctl kickstart -k gui/$(id -u)/ai.openclaw.gateway

Linux:

systemctl --user restart openclaw-gateway

Docker:

docker compose restart openclaw-gateway

Verify (all platforms):

tail -f ~/.openclaw/logs/gateway.log | grep langsmith
# Should see: [langsmith] langsmith tracing active

Configuration

Option	Type	Default	Description
langsmithApiKey	string	$LANGSMITH_API_KEY	LangSmith API key
langsmithEndpoint	string	https://api.smith.langchain.com	API endpoint
projectName	string	openclaw	LangSmith project name
traceAgentTurns	boolean	true	Trace agent turns
traceToolCalls	boolean	true	Trace tool calls
traceEngramLlm	boolean	true	Trace engram LLM calls
batchIntervalMs	number	1000	Batch flush interval (ms)
batchMaxSize	number	20	Max operations before flush
debug	boolean	false	Enable debug logging

Filtering Traces

Traces are automatically tagged for easy filtering in LangSmith:

Tag	Description	Example
cron	Cron job runs	Filter all scheduled jobs
discord	Discord messages	Filter Discord conversations
slack	Slack messages	Filter Slack conversations
telegram	Telegram messages	Filter Telegram conversations
job:<id>	Specific cron job	job:96b7720d-02b1-4373-8846-33306c9913fc
name:<name>	Cron job name	name:X Bookmarks → Insights pipeline
channel:<id>	Discord channel	channel:1467253309348909241
guild:#<name>	Discord guild	guild:#proj-deckard

How It Works

Agent Turns

Hooks into before_agent_start and agent_end. Creates LangSmith runs with:

• Prompt content
• Response messages
• Token usage (prompt, completion, total)
• Duration
• Auto-generated tags based on session source

Tool Calls

Hooks into before_tool_call and after_tool_call. Tool runs are nested under the parent agent run using LangSmith's trace_id and dotted_order for proper hierarchy.

Engram LLM Calls

The engram memory plugin emits LlmTraceEvent objects via globalThis.__openclawEngramTrace. This plugin subscribes and creates LLM runs with:

• Full prompt text (no truncation)
• Full output text (no truncation)
• Model and operation type
• Token usage and duration

Error Isolation

• No API key? No problem. If you install the plugin without configuring an API key, it simply logs a warning and disables itself — OpenClaw continues running normally
• All LangSmith API calls wrapped in try/catch
• Network failures log warnings but never affect gateway operation
• Invalid API keys or LangSmith outages won't break your agents

Development

npm install
npm run build    # Build with tsup
npm run dev      # Watch mode

Related Projects

• OpenClaw — The AI agent gateway
• openclaw-engram — Local-first memory plugin
• LangSmith — LLM observability platform

License

MIT © Joshua Warren