Observability

For maintainers. Using T3 Code? See docs/user.

T3 Code has one server-side observability model:

pretty logs go to stdout for humans
completed spans go to a local NDJSON trace file
traces, metrics, and logs can also be exported over OTLP to a real backend like Grafana LGTM

The local trace file is the persisted source of truth for normal local launches. Those launches do not write a separate server log file, but SSH-managed launches also persist the remote process's stdout/stderr at ~/.t3/ssh-launch/<state>/server.log.

Where To Find Things

Logs

Every log the server writes goes to stdout for humans:

destination: stdout
format: Logger.consolePretty()
normal local persistence: none
SSH-managed launch persistence: ~/.t3/ssh-launch/<state>/server.log

When OTLP logs are configured the same records are also exported as OTLP log records, batched and carrying the trace and span id of whatever was running, so a log line in the backend links back to the span that produced it.

If you want a log message to show up in the local trace file, emit it inside an active span with Effect.log.... Logger.tracerLogger will attach it as a span event.

Traces

Completed spans are written as NDJSON records to serverTracePath. The default depends on how the server starts: production and explicitly configured homes use <home>/userdata/logs/server.trace.ndjson (so ~/.t3/userdata/... by default, or /custom/path/userdata/... with --home-dir /custom/path), a linked worktree dev run uses <worktree>/.t3/userdata/logs/server.trace.ndjson, and an implicit dev run outside a linked worktree uses ~/.t3/dev/logs/server.trace.ndjson.

Important fields common to both record types:

type: effect-span or otlp-span
name: span name
traceId, spanId, parentSpanId: correlation
durationMs: elapsed time
attributes: structured context
events: embedded logs and custom events

effect-span records also contain exit with Success, Failure, or Interrupted. otlp-span records instead carry OTLP resource, scope, and optional status fields.

The TraceRecord, EffectTraceRecord, and OtlpTraceRecord schemas live in packages/shared/src/observability.ts.

DPoP proof failures include the safe environment.dpop.failure_code span attribute. A time_window failure means that a signed proof was too old or too far in the future for the environment server's allowed window. It can point to a date or time problem on either device, but it can also result from a delayed request.

Metrics

Metrics are not written to a local file.

local persistence: none
remote export: OTLP only, when configured
current definitions: apps/server/src/observability/Metrics.ts

If OTLP is not configured, metrics still exist in-process, but you will not have a local artifact to inspect.

Related Artifacts

Provider event NDJSON files still exist for provider runtime streams. Those are separate from the main server trace file.

Run The Server In Instrumented Mode

There are two useful modes:

local-only: stdout + local server.trace.ndjson
full local observability: stdout + local trace file + OTLP export to Grafana/Tempo/Prometheus/Loki

The local trace file is always on. OTLP export is opt-in.

Option 1: Local Traces Only

You do not need any extra env vars. Just run the app normally and inspect server.trace.ndjson.

Examples:

npx t3

node --run dev

node --run dev:desktop

Option 2: Run With A Local LGTM Stack

1. Start Grafana LGTM

docker run --name lgtm \
  -p 3000:3000 \
  -p 4317:4317 \
  -p 4318:4318 \
  --rm -ti \
  grafana/otel-lgtm

Then open http://localhost:3000.

Default Grafana login:

username: admin
password: admin

2. Export OTLP env vars

export T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces
export T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics
export T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs
export T3CODE_OTLP_SERVICE_NAME=t3-local

Optional:

export T3CODE_TRACE_MIN_LEVEL=Info
export T3CODE_TRACE_TIMING_ENABLED=true

3. Launch the app from that same shell

CLI:

npx t3

Monorepo web/server dev:

node --run dev

Monorepo desktop dev:

node --run dev:desktop

Packaged desktop app:

Launch the actual app executable from the same shell so the desktop app and embedded backend inherit T3CODE_OTLP_*.

macOS app bundle example:

T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces \
T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics \
T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs \
T3CODE_OTLP_SERVICE_NAME=t3-desktop \
"/Applications/T3 Code.app/Contents/MacOS/T3 Code"

Direct binary example:

T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces \
T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics \
T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs \
T3CODE_OTLP_SERVICE_NAME=t3-desktop \
./path/to/your/desktop-app-binary

Do not rely on launching from Finder, Spotlight, the dock, or the Start menu after setting shell env vars. Those launches usually will not pick them up.

4. Fully restart after changing env

The backend reads observability config at process start. If you change OTLP env vars, stop the app completely and start it again.

Option 3: The Standard `OTEL_*` Variables

If your machine already exports the OpenTelemetry environment variables for everything else running on it, T3 Code joins in without being told twice. Nothing above is required:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=lab

The base endpoint is a base, not a full URL: traces go to <endpoint>/v1/traces, metrics to <endpoint>/v1/metrics, and log records to <endpoint>/v1/logs, exactly as the specification says. Set OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT when a signal needs a full URL of its own.

Ambient OTEL_* variables turn export on by themselves. A work collector in your shell profile means T3 Code exports to it, so use OTEL_SDK_DISABLED=true if that is not what you want.

Which Processes Export

The desktop app is two processes, and each is its own OpenTelemetry producer:

The server, under service name t3-server.
The Electron main process, under service name t3-desktop. It owns app startup, window and menu work, backend supervision, and updates, none of which the server can see. It reads the same sources in the same order as the server, so a machine that points one of them at a collector points both.

The web client reports as t3-web, so the three service names are t3-server, t3-desktop, and t3-web.

Service names are static and the environment cannot change them. OTEL_SERVICE_NAME and a service.name inside OTEL_RESOURCE_ATTRIBUTES are both refused, with a warning naming the one you set. This is a deliberate departure from what most OpenTelemetry SDKs do, and the reason is that a service name is not a preference: renaming one process merges two services in every dashboard built on them, and a shell profile that names the app it was written for should not be able to do that to T3 Code. Use OTEL_RESOURCE_ATTRIBUTES to tell instances apart, which is what it is for:

export OTEL_RESOURCE_ATTRIBUTES=service.instance.id=laptop-01,deployment.environment=lab

T3CODE_OTLP_SERVICE_NAME still renames the server, because it is T3 Code's own variable and nobody sets it across a fleet by accident. There is no equivalent for the main process.

On macOS, ambient variables reach the desktop app only when it is launched from a shell. Opening it from the Dock, Finder, or Spotlight inherits launchd's environment instead, which is why the instrumented walkthrough above launches from the same shell that exported the variables. Settings and T3CODE_OTLP_* are not affected, and the server the desktop app spawns inherits whatever the main process was given.

Precedence

For each signal, the first source that names its endpoint wins:

T3CODE_OTLP_*
the desktop bootstrap envelope
Settings, under observability
OTEL_*

Whichever source wins takes the whole signal, not just the URL. Traces sent to a T3CODE_OTLP_TRACES_URL endpoint keep T3 Code's own wire format, headers, batching, and export interval even when OTEL_* variables are set, because those variables describe the collector they named rather than this one. T3CODE_OTLP_EXPORT_INTERVAL_MS is the exception, and applies to every signal wherever it goes.

The three signals are resolved separately, so traces can come from one source and metrics or logs from another.

OTEL_SDK_DISABLED=true outranks all four and stops every export, including one configured through Settings.

What Is Read

Variable	Effect
`OTEL_SDK_DISABLED`	Stops all export
`OTEL_EXPORTER_OTLP_ENDPOINT`	Base URL for every signal
`OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT`	Full URL for one signal
`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_HEADERS`	Export headers, per signal overriding the shared ones
`OTEL_EXPORTER_OTLP_PROTOCOL`, `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_PROTOCOL`	`http/protobuf` (default) or `http/json`
`OTEL_{TRACES,METRICS,LOGS}_EXPORTER`	A list; the signal is exported when it contains `otlp`, which is the default
`OTEL_SERVICE_VERSION`, `OTEL_RESOURCE_ATTRIBUTES`	Resource identity attached to every span, metric, and log record
`OTEL_SERVICE_NAME`	Refused with a warning; service names are static
`OTEL_BSP_SCHEDULE_DELAY`, `OTEL_METRIC_EXPORT_INTERVAL`, `OTEL_BLRP_SCHEDULE_DELAY`	Export interval, one per signal
`OTEL_BSP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE`	Spans per batch, log records per batch
`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE`	`cumulative` or `delta`

The wire format defaults to http/protobuf when the endpoint came from OTEL_*, matching the specification, and stays http/json for a T3CODE_OTLP_* setup that never mentioned a protocol.

OTEL_EXPORTER_OTLP_PROTOCOL=grpc is refused rather than downgraded, because T3 Code has no gRPC transport and posting an HTTP body to a gRPC endpoint fails in a way that is harder to read than exporting nothing. The refusal is logged at startup and turns off only the signal that named gRPC, and only when that signal had no other endpoint to go to.

Header and resource-attribute values are percent decoded, so OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20abc sends the space and a base64 credential keeps its = padding.

Known Gaps

Not everything in the specification is implemented. These are the ones worth knowing about:

No gRPC. OTEL_EXPORTER_OTLP_PROTOCOL=grpc is refused rather than downgraded, because T3 Code has no gRPC transport and posting an HTTP body to a gRPC endpoint fails in a way that is harder to read than exporting nothing. The refusal is logged at startup and turns off only the signal that named gRPC, so OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=grpc leaves traces exporting.
No compression and no client TLS. OTEL_EXPORTER_OTLP_COMPRESSION, OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_CLIENT_KEY, and OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE are ignored. A collector that requires mutual TLS needs a proxy in front of it.
No export timeouts. OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_TIMEOUT, and OTEL_METRIC_EXPORT_TIMEOUT are per-request deadlines, and this exporter has no per-request knob, so they are ignored. Spending them on the shutdown flush instead would be the wrong meaning and would let a generous collector timeout hold the server open on every restart.
Browser traces are always JSON. The proxy that forwards traces from the client posts OTLP/HTTP JSON regardless of OTEL_EXPORTER_OTLP_PROTOCOL. Both are valid OTLP/HTTP and most collectors accept either, so this only matters against one that takes protobuf and nothing else. When the trace exporter resolves to http/protobuf, a startup warning names the split rather than letting the browser half disappear while the server half looks healthy.
No protocol name of T3 Code's own. OTEL_EXPORTER_OTLP_PROTOCOL describes the endpoint these variables named. A T3CODE_OTLP_* endpoint always uses http/json, which is what it has always used.
lowmemory temporality is not available. OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE accepts cumulative and delta. lowmemory logs a warning and falls back to cumulative.
OTEL_SERVICE_VERSION is not a specification variable. It is read as a convenience because the exporter library reads it too. OTEL_RESOURCE_ATTRIBUTES=service.version=... is the portable spelling.
service.name cannot be set from the environment. See Which Processes Export above. OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES=service.name=... are both refused with a warning, and the service.name key is dropped rather than passed through so the exporter never receives two of them.

Everything else not listed above is ignored, including OTEL_BSP_MAX_QUEUE_SIZE, OTEL_BLRP_MAX_QUEUE_SIZE, OTEL_BSP_EXPORT_TIMEOUT, OTEL_BLRP_EXPORT_TIMEOUT, sampler variables, propagator variables, and the attribute and span limit variables.

When A Value Cannot Be Used

A variable T3 Code cannot act on never stops it from starting. Two things can happen instead, and both are logged once at startup:

A warning, then the default. A misspelled protocol, an unavailable temporality, a timeout or batch size that is not a whole number, or a pair list that is not valid percent encoding is reported and ignored, and everything else keeps exporting. One bad value never costs you the other variables.
Export off. Only OTEL_EXPORTER_OTLP_PROTOCOL=grpc does this, because it names a transport T3 Code does not speak rather than a value it failed to parse.

An empty value means the same thing as an unset one, so OTEL_SERVICE_VERSION= reads as if the variable were not there at all. An empty OTEL_SERVICE_NAME is not an attempt to rename anything, so it is not warned about either. OTEL_SDK_DISABLED follows the specification's one rule for booleans: the case-insensitive string true is the only value that switches export off, and anything else, including yes and 1, leaves it on.

A OTEL_EXPORTER_OTLP_HEADERS or OTEL_RESOURCE_ATTRIBUTES value that fails to decode is discarded whole rather than partly. A half-parsed credential reaches the collector as the same authentication error a wrong one would, which reads like a bad token instead of a bad variable.

These variables configure a signal only when they also supplied its endpoint. A T3CODE_OTLP_* name, the desktop bootstrap envelope, or Settings winning the URL takes the whole signal with it, so an ambient OTEL_EXPORTER_OTLP_ENDPOINT cannot reach in and change the wire format, headers, or batching of an export it did not point anywhere. Traces, metrics, and logs are answered separately throughout, so OTEL_EXPORTER_OTLP_METRICS_PROTOCOL applies to metrics alone and leaves traces and logs as they were.

Once these variables are the ones configuring the exporter, the specification's own defaults apply: OTEL_BSP_SCHEDULE_DELAY 5s, OTEL_METRIC_EXPORT_INTERVAL 60s, OTEL_BLRP_SCHEDULE_DELAY 1s, and OTEL_BSP_MAX_EXPORT_BATCH_SIZE and OTEL_BLRP_MAX_EXPORT_BATCH_SIZE 512 each. A T3CODE_OTLP_* setup keeps the numbers T3 Code has always used.

How To Use Traces And Metrics To Debug The Server

Start With The Local Trace File

The trace file is the fastest way to inspect raw span data.

Resolve the path for the launch mode once. Production and explicitly configured homes store runtime state under the base directory's userdata folder:

TRACE_FILE="${T3CODE_HOME:-$HOME/.t3}/userdata/logs/server.trace.ndjson"

A dev server started from a linked worktree defaults to that worktree's local home:

TRACE_FILE="$WORKTREE/.t3/userdata/logs/server.trace.ndjson"

Only an implicit dev run outside a linked worktree uses the shared dev directory:

TRACE_FILE="$HOME/.t3/dev/logs/server.trace.ndjson"

Tail the selected file:

tail -f "$TRACE_FILE"

Show failed spans:

jq -c 'select(.type == "effect-span" and .exit._tag != "Success") | {
  name,
  durationMs,
  exit,
  attributes
}' "$TRACE_FILE"

Show slow spans:

jq -c 'select(.durationMs > 1000) | {
  name,
  durationMs,
  traceId,
  spanId
}' "$TRACE_FILE"

Inspect embedded log events:

jq -c 'select(any(.events[]?; .attributes["effect.logLevel"] != null)) | {
  name,
  durationMs,
  events: [
    .events[]
    | select(.attributes["effect.logLevel"] != null)
    | {
        message: .name,
        level: .attributes["effect.logLevel"]
      }
  ]
}' "$TRACE_FILE"

Follow one trace:

jq -r 'select(.traceId == "TRACE_ID_HERE") | [
  .name,
  .spanId,
  (.parentSpanId // "-"),
  .durationMs
] | @tsv' "$TRACE_FILE"

Filter orchestration commands:

jq -c 'select(.attributes["orchestration.command_type"] != null) | {
  name,
  durationMs,
  commandType: .attributes["orchestration.command_type"],
  aggregateKind: .attributes["orchestration.aggregate_kind"]
}' "$TRACE_FILE"

Filter git activity:

jq -c 'select(.attributes["git.operation"] != null) | {
  name,
  durationMs,
  operation: .attributes["git.operation"],
  cwd: .attributes["git.cwd"],
  hookEvents: [
    .events[]
    | select(.name == "git.hook.started" or .name == "git.hook.finished")
  ]
}' "$TRACE_FILE"

Use Tempo When You Need A Real Trace Viewer

Tempo is better than raw NDJSON when you want to:

search across many traces
inspect parent/child relationships visually
compare many slow traces
drill into one failing request without hand-joining by traceId

Recommended flow in Grafana:

Open Explore.
Pick the Tempo data source.
Set the time range to something recent like Last 15 minutes.
Start broad. Do not begin with a very narrow query.
Look for spans from your configured service name, then narrow by span name or attributes.

Good first searches:

service name such as t3-local, t3-dev, or t3-desktop
span names like sendTurn or a Git operation such as GitVcsDriver.statusDetails.status
Git spans whose git.operation attribute identifies the operation
orchestration spans with attributes like orchestration.command_type

Once you know traces are arriving, narrower TraceQL queries for names such as sendTurn or Git operation names become useful.

Use Metrics To See Systemic Problems

Traces are best for one request. Metrics are best for trends.

Good metric families to watch:

t3_rpc_request_duration
t3_orchestration_command_duration
t3_orchestration_command_ack_duration
t3_provider_turn_duration
t3_git_command_duration

Counters tell you volume and failure rate:

t3_rpc_requests_total
t3_orchestration_commands_total
t3_provider_turns_total
t3_git_commands_total

Use metrics when the question is:

"is this always slow?"
"did this get worse after a change?"
"which command type is failing most often?"

Use traces when the question is:

"what happened in this specific request?"
"which child span caused this one slow interaction?"
"what logs were emitted inside the failing flow?"

What The New Ack Metric Means

t3_orchestration_command_ack_duration measures:

start: command dispatch enters the orchestration engine
end: the first committed domain event for that command is published by the server

That is a server-side acknowledgment metric. It does not measure:

websocket transit to the browser
client receipt
React render time

If you need those later, add client-side instrumentation or a dedicated server fanout metric.

Common Workflows

"Why did this request fail?"

Start with the local NDJSON file.
Find effect-span records where exit._tag != "Success".
Group by traceId.
Inspect sibling spans and span events.
If needed, move to Tempo for the full trace tree.

"Why is the UI feeling slow?"

Search for slow top-level spans in the trace file or Tempo.
Check child spans for sqlite, git, provider, or terminal work.
Look at the matching duration metrics to see whether the slowness is systemic.

"Did this command take too long to acknowledge?"

Check t3_orchestration_command_ack_duration by commandType.
If it is high, inspect the corresponding orchestration trace.
Look at child spans for projection, sqlite, provider, or git work.

"Are git hooks causing latency?"

Filter git.operation spans.
Inspect git.hook.started and git.hook.finished events.
Compare hook timing to the enclosing git span duration.

"Why do I have spans locally but nothing in Grafana?"

Usually one of these is true:

T3CODE_OTLP_TRACES_URL was not set
the app was launched from a different environment than the one where you exported the vars
the app was not fully restarted after changing env
Grafana is looking at the wrong time range or service name

If the local NDJSON file is updating, local tracing is working. The problem is almost always OTLP export configuration or process startup.

How To Think About Adding Tracing To Future Code

Prefer Boundaries Over Tiny Helpers

Good span boundaries:

RPC methods
orchestration command handling
provider adapter calls
external process calls
persistence writes
queue handoffs

Avoid tracing every tiny helper. Most helpers should inherit the active span rather than create a new one.

Reuse `Effect.fn(...)` Where It Already Exists

The codebase already uses Effect.fn("name") heavily. That should usually be your first tracing boundary.

For ad hoc work:

import { Effect } from "effect";

const runThing = Effect.gen(function* () {
  yield* Effect.annotateCurrentSpan({
    "thing.id": "abc123",
    "thing.kind": "example",
  });

  yield* Effect.logInfo("starting thing");
  return yield* doWork();
}).pipe(Effect.withSpan("thing.run"));

Put High-Cardinality Detail On Spans

Use span annotations for IDs, paths, and other detailed context:

yield *
  Effect.annotateCurrentSpan({
    "provider.thread_id": input.threadId,
    "provider.request_id": input.requestId,
    "git.cwd": input.cwd,
  });

Keep Metric Labels Low Cardinality

Good metric labels:

operation kind
method name
provider kind
aggregate kind
outcome

Bad metric labels:

raw thread IDs
command IDs
file paths
cwd
full prompts
full model strings when a normalized family label would do

Detailed context belongs on spans, not metrics.

Use Logs As Span Events

Logs inside a span become part of the trace story:

yield * Effect.logInfo("starting provider turn");
yield * Effect.logDebug("waiting for approval response");

Those messages show up as span events because Logger.tracerLogger is installed.

Use The Pipeable Metrics API

withMetrics(...) is the default way to attach a counter and timer to an effect:

import { someCounter, someDuration, withMetrics } from "../observability/Metrics.ts";

const program = doWork().pipe(
  withMetrics({
    counter: someCounter,
    timer: someDuration,
    attributes: {
      operation: "work",
    },
  }),
);

Detailed API Reference

Runtime Wiring

The server observability layer is assembled in apps/server/src/observability/Layers/Observability.ts.

It provides:

pretty stdout logger
Logger.tracerLogger
local NDJSON tracer
optional OTLP trace exporter
optional OTLP metrics exporter
Effect trace-level and timing refs

The Electron main process assembles its own in apps/desktop/src/app/DesktopObservability.ts, with the same pieces plus an optional OTLP log exporter, and resolves its endpoints in apps/desktop/src/app/DesktopOtlpExport.ts. Both processes read the OTEL_* variables through packages/shared/src/otelEnvironment.ts, so neither can disagree with the other about what a variable means.

Env Vars

Local trace file:

T3CODE_TRACE_FILE: override trace file path
T3CODE_TRACE_MAX_BYTES: per-file rotation size, default 10485760
T3CODE_TRACE_MAX_FILES: rotated file count, default 10
T3CODE_TRACE_BATCH_WINDOW_MS: flush window, default 200
T3CODE_TRACE_MIN_LEVEL: minimum trace level, default Info
T3CODE_TRACE_TIMING_ENABLED: enable timing metadata, default true

OTLP export:

T3CODE_OTLP_TRACES_URL: OTLP trace endpoint
T3CODE_OTLP_METRICS_URL: OTLP metric endpoint
T3CODE_OTLP_LOGS_URL: OTLP log endpoint
T3CODE_OTLP_EXPORT_INTERVAL_MS: export interval, default 10000
T3CODE_OTLP_SERVICE_NAME: server service name, default t3-server. The Electron main process does not read it and is always t3-desktop.

If the OTLP URLs are unset, local tracing still works and metrics stay in-process only.

What Is Instrumented Today

Current high-value span and metric boundaries include:

Effect RPC websocket request spans from effect/rpc
RPC request metrics in apps/server/src/observability/RpcInstrumentation.ts
startup phases
orchestration command processing
orchestration command acknowledgment latency
provider session and turn operations
git command execution and git hook events
terminal session lifecycle
sqlite query execution

Current Constraints

logs outside spans are not persisted in the trace file; SSH-managed launch stdout/stderr is still captured in its launcher log
metrics are not snapshotted locally
the old serverLogPath still exists in config for compatibility, but the trace file is the primary structured persisted artifact

Uh oh!

FilesExpand file tree

observability.md

Latest commit

History