Tracing and observability

AgentExecutor.execute() and FlowExecutor.execute() emit spans through a TraceExporter. Span names and attributes follow the OpenTelemetry GenAI semantic conventions, so GenAI-aware observability backends can recognise them without custom mapping.

Stability. The GenAI conventions are in Development status (not stable). Attribute names have changed between versions of the spec before (for example gen_ai.system became gen_ai.provider.name) and may change again. This SDK follows the spec as retrieved on 2026-10-01 from the semantic-conventions-genai repository (where the conventions now live). Pin your SDK version if dashboards depend on these names.

Quick start

Pass an exporter to createAgent(); every run of the agent (send(), stream(), session turns, resume() and runs continued by agent.approvals.resolve()) is traced, sub-agents included. To keep traces on disk and browse them with npx lousho traces, use fileTraceExporter() (see Local traces). The vendor/ model prefix chooses the provider; with OpenRouter use openrouter/<vendor>/<model> (e.g. openrouter/openai/gpt-4o-mini).

import { createAgent } from '@lousho/build-ai-agent';
import { fileTraceExporter } from '@lousho/build-ai-agent/traces';

const agent = createAgent({
  model: 'openai/gpt-4o-mini',
  instructions: 'You are a helpful assistant.',
  exporter: fileTraceExporter(), // writes .lousho/traces/<day>/<traceId>.jsonl
});

Any object with onSpanStart and onSpanEnd is an exporter. The run does not wait for them: a hook that throws, or an async one whose promise rejects, is reported once per exporter with console.warn and otherwise ignored.

Advanced: the executor API

With the executor, pass the exporter to AgentExecutor.execute():

import { AgentExecutor } from '@lousho/build-ai-agent/executor';
import { type TraceExporter } from '@lousho/build-ai-agent';

const exporter: TraceExporter = {
  onSpanStart: (span) => console.log('[start]', span.kind, span.name),
  onSpanEnd: (span) => console.log('[end]', span.name, span.status?.code ?? 'ok', span.attributes),
};

await AgentExecutor.execute({ agent, input, provider, toolRegistry, exporter });

For real OpenTelemetry spans, import createOtelTraceExporter() from @lousho/build-ai-agent/otel (needs the optional peer dependency @opentelemetry/api and a registered TracerProvider). It carries span kind and error status over to OpenTelemetry. See examples/tracing (npm run example:tracing:console, npm run example:tracing:otel, which also prints metrics).

Span tree

invoke_agent {agent name}          INTERNAL
  chat {model}                     CLIENT     one per provider.generate() call
  execute_tool {tool name}         INTERNAL   one per tool call

Each send() / stream() call is a trace of its own unless you parent it. To put several runs in one trace, pass a span id of yours as parentSpanId; the run's invoke_agent span becomes its child, and its cost rolls up into it:

import { withSpan, type SimpleAgent, type TraceExporter } from '@lousho/build-ai-agent';

declare const exporter: TraceExporter;
declare const planner: SimpleAgent;
declare const writer: SimpleAgent;

await withSpan(exporter, 'research pipeline', {}, async (span) => {
  await planner.send('Plan the report.', { parentSpanId: span.id });
  await writer.send('Write it.', { parentSpanId: span.id });
});

A run continued after an approval (agent.approvals.resolve()) is a trace of its own: its invoke_agent span holds the approved call's execute_tool span (first, as the call runs before the model is called again), then the chat spans of the rest of the run. A rejected call has no execute_tool span.

A flow run (FlowExecutor.execute) produces:

invoke_workflow {flow name}        INTERNAL
  flow.node {node type}            INTERNAL   one per node execution, nested like the flow
    chat {model}                   CLIENT     inside an llmCall node
    execute_tool {tool name}       INTERNAL   inside a toolCall node

To nest a flow under an agent span, pass that span's id as parentSpanId (withSpan hands the span to its callback); the flow span then becomes its child.

Attributes

Constants for every name live in src/execution/semconv.ts (exported as GenAiAttr, GenAiOperation, GenAiMetric, ErrorAttr, FlowAttr, SdkAttr, LegacyAttr).

Agent run: `invoke_agent {agent name}`

Attribute	Value
`gen_ai.operation.name`	`invoke_agent`
`gen_ai.agent.name`	The agent's `name`. An unnamed sub-agent uses the key the lead knows it by (`subagents: { researcher }` gives `researcher`), and the span is named after it
`gen_ai.agent.id`	The agent's `id`, when set
`gen_ai.provider.name`	The provider's `name`
`gen_ai.request.model`	The agent's model (`settings.model`, else the provider's default), when known
`gen_ai.conversation.id`	`sessionId`, when set
`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`	The run's total tokens: every model call, including those of delegated child runs; absent when no call reported usage. These repeat the `chat` spans' tokens: see Summing token usage over spans. The `gen_ai.client.token.usage` metric still counts each model call once
`lousho.cost_usd`	Cumulative estimated USD of the run (every model call and delegated child run, including a background child that finishes before the run ends); absent when any model used has no known price
`lousho.usage.estimated`	`true` when any of the run's tokens were estimated
`lousho.subagent.name`	On a sub-agent's run: the name the lead knows it by (the `task` tool's `agent`)
`lousho.subagent.depth`	On a sub-agent's run: `1` for a sub-agent of the top-level run, `2` for its sub-agent, and so on
`lousho.task.id`	On a sub-agent's run started by `task`: the task's `taskId`

Summing token usage over spans

The totals on invoke_agent spans count tokens that the run's chat spans already carry, and a lead run's total also counts its sub-agents' runs. A backend that adds up gen_ai.usage.* over every span of a trace (some OpenTelemetry LLM cost dashboards do) therefore counts each token at least twice, and once more per level of sub-agent nesting: a lead with one sub-agent whose model calls used 33 + 7 input tokens shows 40 on the chat spans, 7 on the sub-agent's invoke_agent span and 40 on the lead's, 87 in all.

To get the true total, sum chat spans only (gen_ai.operation.name = chat), read the top-level invoke_agent span alone, or use the gen_ai.client.token.usage metric. If the backend cannot filter, pass createOtelTraceExporter({ agentUsageTotals: false }): the usage attributes then stay on chat spans only (lousho.cost_usd stays on every span).

Model call: `chat {model}` (CLIENT)

Attribute	Value
`gen_ai.operation.name`	`chat`
`gen_ai.provider.name`	The provider's `name`
`gen_ai.request.model`	The requested model
`gen_ai.request.temperature`, `gen_ai.request.max_tokens`	When set
`gen_ai.response.model`	When the provider reports it
`gen_ai.response.id`	The provider's response id, when it reports one
`gen_ai.response.finish_reasons`	e.g. `["stop"]`, `["tool_call"]`
`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`	Token usage
`lousho.cost_usd`	Estimated USD of this step, when the built-in price table (or `registerModel`) knows the model; absent otherwise
`lousho.usage.estimated`	`true` when the provider reported no usage and the tokens were estimated
`lousho.retry.count`	How many times `withRetry()` (the agent's `retry`) retried this call; absent when it was not retried
`lousho.retry.errors`	Each retried failure's error category and HTTP status, e.g. `["timeout", "unknown 500"]`

Tool call: `execute_tool {tool}`

Attribute	Value
`gen_ai.operation.name`	`execute_tool`
`gen_ai.tool.name`	The tool name
`gen_ai.tool.call.id`	The model's tool-call id (agent runs)
`gen_ai.tool.description`	The tool's description, when it has one
`gen_ai.tool.type`	`function`
`gen_ai.agent.name`, `gen_ai.conversation.id`	The calling agent / session
`lousho.tool.parent_call_id`	On a call a code mode script made: the `run_code` call's id (its span is the parent)

Flows

The GenAI spec defines no convention for flow nodes, so node spans use the lousho.flow.* namespace. The run span uses the spec's workflow convention.

Span	Attribute	Value
`invoke_workflow {name}`	`gen_ai.operation.name`	`invoke_workflow`
	`gen_ai.workflow.name`	The flow's `name`
	`lousho.flow.code`	The flow's `code`
	`lousho.flow.outcome`	`success` or `error`
`flow.node {type}`	`lousho.flow.node.id`	The node's id
	`lousho.flow.node.type`	The node type (`sequence`, `llmCall`, ...)
	`lousho.flow.outcome`	`success` or `error`

Errors

A failed span gets error.type and an error span status (OpenTelemetry ERROR, message = the error message). error.type is the thrown error's name (_OTHER when a non-Error is thrown). A tool call that returns an error to the model (isError) is marked error.type = tool_error.

Metrics

createOtelTraceExporter() also records the OpenTelemetry GenAI client metrics through @opentelemetry/api's metrics API, using the global MeterProvider (metrics.getMeter(...)), or the meter option:

Metric	Type, unit	Recorded	Attributes
`gen_ai.client.token.usage`	Histogram, `{token}`	Once per token type for every model call that reports (or estimates) usage	`gen_ai.operation.name` (`chat`), `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.response.model` (when known), `gen_ai.token.type` (`input` or `output`)
`gen_ai.client.operation.duration`	Histogram, `s`	Every model call (`chat`) and tool call (`execute_tool`)	`gen_ai.operation.name`, `gen_ai.provider.name` and `gen_ai.request.model` (model calls), `gen_ai.response.model` (when known), `error.type` (when it failed)

Both use the spec's recommended bucket boundaries. They follow the gen-ai-metrics page of semantic-conventions v1.40 (Development stability): the newest conventions split the token histogram into per-type counters, which this SDK does not emit yet. Agent and flow spans are not counted in the duration metric, only the model and tool calls inside them.

import { metrics } from '@opentelemetry/api';
import { ConsoleMetricExporter, MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { createOtelTraceExporter } from '@lousho/build-ai-agent/otel';

// Swap ConsoleMetricExporter for an OTLP exporter to reach any backend.
metrics.setGlobalMeterProvider(
  new MeterProvider({ readers: [new PeriodicExportingMetricReader({ exporter: new ConsoleMetricExporter() })] })
);

const exporter = createOtelTraceExporter(); // metrics on by default
// createOtelTraceExporter({ metrics: false })  // spans only
// createOtelTraceExporter({ meter })           // an already-obtained Meter

Without a registered MeterProvider, OpenTelemetry's no-op meter discards the records. The metrics API lives in @opentelemetry/api, so nothing is recorded (and nothing is loaded) when that optional peer is not installed. The cost is an estimate from the price table (see registerModel), not a bill.

Message and argument content is opt-in

Prompts, model output and tool arguments/results are sensitive and large, so the gen_ai.* content attributes are never recorded by default. Opt in per agent with createAgent({ exporter, captureContent: true }).

Advanced: the executor API

With the executor, opt in per run:

import { AgentExecutor } from '@lousho/build-ai-agent/executor';
import { type TraceExporter } from '@lousho/build-ai-agent';

declare const exporter: TraceExporter;

await AgentExecutor.execute({ agent, input, provider, exporter, captureContent: true });

Flows take the same flag on their context (captureContent). When captureContent is not set, the OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true environment variable (the name the spec uses as its example) turns it on. An explicit captureContent: false always wins.

With it on, content is recorded as JSON strings (the spec's fallback for spans) in the spec's message schema:

Attribute	On	Content
`gen_ai.input.messages`	`invoke_agent`, `chat`	`[{ role, parts: [{ type: 'text', content }, { type: 'tool_call', id, name, arguments }, { type: 'tool_call_response', id, response }] }]`
`gen_ai.system_instructions`	`chat`	`[{ type: 'text', content }]`
`gen_ai.output.messages`	`chat`	`[{ role: 'assistant', parts: [...] }]`
`gen_ai.tool.call.arguments`	`execute_tool`	The arguments as JSON
`gen_ai.tool.call.result`	`execute_tool`	The result as JSON (not set when the tool failed)

Deprecated attribute names

Before the GenAI conventions, spans used ad-hoc names and the span names agent.run, llm.generate and tool.call. The old attribute names are still emitted next to the new ones so existing dashboards keep working, but they are deprecated and will be removed in a future major version. Span names changed (to what the spec requires); update queries that matched on them.

Deprecated	Use instead
span `agent.run`	span `invoke_agent {name}`, `gen_ai.operation.name = invoke_agent`
span `llm.generate`	span `chat {model}`, `gen_ai.operation.name = chat`
span `tool.call`	span `execute_tool {tool}`, `gen_ai.operation.name = execute_tool`
`model`	`gen_ai.request.model`
`promptTokens`	`gen_ai.usage.input_tokens`
`completionTokens`	`gen_ai.usage.output_tokens`
`totalTokens`	(sum the two usage attributes)
`finishReason`	`gen_ai.response.finish_reasons` (an array; `tool_calls` is now `tool_call`)
`toolName`	`gen_ai.tool.name`
`error` (the error message; absent on success)	`error.type` and the span status
`input`, `prompt`, `args`, `result`	`gen_ai.input.messages`, `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result` (opt-in, see above)

With createOtelTraceExporter({ legacyAttributes: false }) the OpenTelemetry spans carry only the GenAI names, not the deprecated ones next to them (fewer attributes per span); the default true keeps both.

The deprecated content attributes (input, prompt, args, result) are off by default: redactContent defaults to true. Pass redactContent: false to record them as before; captureContent governs only the gen_ai.* content attributes.

Local traces

fileTraceExporter() from @lousho/build-ai-agent/traces writes each run to a file, and lousho traces shows the files in the terminal. No OpenTelemetry backend, no extra package.

import { createAgent, defineTool } from '@lousho/build-ai-agent';
import { fileTraceExporter } from '@lousho/build-ai-agent/traces';
import { z } from 'zod';

const getWeather = defineTool({
  name: 'get_weather',
  description: 'Current weather for a city',
  input: z.object({ city: z.string() }),
  execute: async ({ city }) => `Sunny in ${city}`,
});

const agent = createAgent({
  model: 'openai/gpt-4o-mini',
  tools: [getWeather],
  exporter: fileTraceExporter({ dir: '.lousho/traces' }), // the default dir
});

Then, from the project directory:

npx lousho traces                 # the 20 most recent runs, newest first
npx lousho traces --limit 50 --json
npx lousho traces 3f2a            # one run as a tree; a unique id prefix is enough
npx lousho traces 3f2a --content  # also the captured messages and tool arguments

TIME    AGENT    DURATION  MODEL CALLS  TOOL CALLS  TOKENS IN/OUT  COST       STATUS  ID
2h ago  weather  1.65s     2            1           159/32         $0.000043  ok      3f2a9c1e-5b7d-4e8a-9c0f-1a2b3c4d5e6f

Trace 3f2a9c1e-5b7d-4e8a-9c0f-1a2b3c4d5e6f  4 spans, 1.65s
invoke_agent weather         |████████████████████████|     1.65s  $0.000043
├─ chat gpt-4o-mini          |█████████████           |     900ms  openai/gpt-4o-mini  in 62 out 18  $0.000020
├─ execute_tool get_weather  |             █          |      22ms  tool get_weather
└─ chat gpt-4o-mini          |             ██████████ |     705ms  openai/gpt-4o-mini  in 97 out 14  $0.000023

Options: --dir D reads another directory (default .lousho/traces), --limit N lists N runs (default 20), --json prints the summaries or the spans as JSON. An id prefix that matches several runs lists them and exits 1. Colour is off when NO_COLOR is set or the output is not a terminal. The token counts and the cost are the sums of the run's chat spans, sub-agents' included (background ones too); the cost is blank (-) when a model's price is unknown.

Files. One file per run at <dir>/<YYYY-MM-DD>/<traceId>.jsonl (the run's local start date; the trace id is the id of its invoke_agent span), so old traces can be deleted by day. Each line is one finished span, written as it ends:

{"v":1,"traceId":"3f2a…","id":"a100…","parentId":"3f2a…","name":"chat gpt-4o-mini","kind":"client","startTime":1790848800005,"endTime":1790848800905,"attributes":{"gen_ai.usage.input_tokens":62,"…":"…"}}

startTime and endTime are milliseconds since the epoch; attributes are the span attributes above, unshortened. A span that never ended (the process died) is not in the file. If the directory cannot be written, the exporter warns once and the run goes on. listTraces({ dir, limit }) and readTrace(idOrPrefix, { dir }) read the files from code.

The files hold what the spans hold. By default that is no message or tool content: the deprecated input, prompt, args and result attributes are recorded only with redactContent: false (see Deprecated attribute names), and captureContent: true adds the gen_ai.* content. Keep .lousho/ out of version control; projects made by lousho init ignore it.

lousho dev and lousho chat write these files when you pass --traces (into .lousho/traces) or --traces=<dir>, for a spec file, an agent directory or a module, so a spec-file agent needs no code to be traced. It is opt-in on purpose: the files persist span timings, token counts and costs, and prompt and tool content only if the agent sets redactContent: false or captureContent: true, so nothing is written to disk unless you ask. The spec file format has no traces setting.

The built server

The server lousho build --target=node-server or --target=docker generates exports traces with no code when the standard OTEL_EXPORTER_OTLP_ENDPOINT (or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) variable is set where it runs. It registers an OpenTelemetry tracer provider that sends spans over OTLP/HTTP, and gives the agent createOtelTraceExporter(), so every /chat turn arrives as the invoke_agent / chat / execute_tool tree above. An exporter the agent's config already has (an agent directory's or a module's) keeps running: both receive every span.

The OpenTelemetry SDK is not bundled. Install these packages next to dist/server.js, or add them to the build's package.json dependencies before you build the docker image:

npm install @opentelemetry/sdk-trace-node@^2 @opentelemetry/resources@^2 @opentelemetry/exporter-trace-otlp-http
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 OTEL_SERVICE_NAME=support-agent node dist/server.js

The server imports them at start-up only when the variable is set. When one is missing it logs which packages to install and serves without exporting traces.
service.name is OTEL_SERVICE_NAME when set, otherwise the agent's name. OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_HEADERS (for an API key) and the batch settings (OTEL_BSP_SCHEDULE_DELAY, ...) work as the OTel SDK documents them.
On SIGINT or SIGTERM the server sends the spans it still holds before it exits.
With neither variable set, nothing is imported and the server runs as before.

Message content follows the agent's own settings (Message and argument content is opt-in).

Backends

Any backend that ingests OpenTelemetry traces (OTLP) receives these spans. Backends and tools that understand the GenAI semantic conventions can render them as LLM, agent and tool calls with token usage; others show them as ordinary spans with attributes. Which conventions a given backend supports, and which version of them, varies; check its documentation.

Navigation Menu

Repository navigation

observability.md

observability.md

Tracing and observability

Quick start

Advanced: the executor API

Span tree

Attributes

Agent run: `invoke_agent {agent name}`

Summing token usage over spans

Model call: `chat {model}` (CLIENT)

Tool call: `execute_tool {tool}`

Flows

Errors

Metrics

Message and argument content is opt-in

Advanced: the executor API

Deprecated attribute names

Local traces

The built server

Backends

FilesExpand file tree

observability.md

Latest commit

History

observability.md

File metadata and controls

Tracing and observability

Quick start

Advanced: the executor API

Span tree

Attributes

Agent run: invoke_agent {agent name}

Summing token usage over spans

Model call: chat {model} (CLIENT)

Tool call: execute_tool {tool}

Flows

Errors

Metrics

Message and argument content is opt-in

Advanced: the executor API

Deprecated attribute names

Local traces

The built server

Backends

Agent run: `invoke_agent {agent name}`

Model call: `chat {model}` (CLIENT)

Tool call: `execute_tool {tool}`