diff --git a/docs/reference/README.md b/docs/reference/README.md index 2765bf90..c3ed5dec 100644 --- a/docs/reference/README.md +++ b/docs/reference/README.md @@ -11,6 +11,7 @@ Reference is the technical description of the machinery and how to operate it. A - [Lore CLI command reference](lore-cli-commands.md) — every `lore` command, subcommand, argument, and flag, generated from `lore --markdown-help`. - [Lore CLI configuration reference](lore-cli-config.md) — every field in the per-repository `config.toml` and user-level `cli.toml`, with each field's type, default, and on-disk location. - [Lore Server configuration reference](lore-server-config.md) — every `loreserver` CLI flag, config-file layer, and settings field, including the AWS, DynamoDB, Consul, and hook plugin backends. +- [Lore Server metrics and traces reference](lore-server-metrics.md) — every OpenTelemetry metric, span, and attribute the server exposes, grouped by subsystem, with instrument types and units. ## Suggested starting points diff --git a/docs/reference/lore-server-metrics.md b/docs/reference/lore-server-metrics.md new file mode 100644 index 00000000..df9c2f73 --- /dev/null +++ b/docs/reference/lore-server-metrics.md @@ -0,0 +1,372 @@ +# Lore Server metrics and traces + +## Synopsis + +```text +. { =, ... } +``` + +Lore Server (`loreserver`) emits metrics, traces, and logs through OpenTelemetry instrumentation. This page catalogs the metrics and spans the server exposes: their names, instrument types, units, and attributes. For the settings that turn export on and tune it, see the [Telemetry settings](lore-server-config.md#telemetry-settings) section of the Lore Server configuration reference. + +Export happens over OTLP (gRPC) and only when `[telemetry.exporter]` is configured. Point the exporter at any OpenTelemetry-compatible collector or backend to scrape the metrics and spans below. + +## How metric names are built + +Most instruments are created through an internal `InstrumentProvider`, which prepends a per-subsystem **namespace** to the **instrument name** with a `.` separator. The exported metric name is `.`. For example, the S3 backend's namespace is `urc.aws.s3` and its latency instrument is `operation_duration`, so the exported metric is `urc.aws.s3.operation_duration`. + +Two namespace prefixes coexist in the current codebase: + +- `urc.*` is the original internal name for Lore and prefixes most instruments today. +- `lore.*` prefixes the newer v1 gRPC services. + +The helper constructors fix each instrument's type, unit, and buckets: + +| Constructor | Instrument | Unit | +| --- | --- | --- | +| `latency_histogram_ms` | `Histogram` | `milliseconds` | +| `size_histogram` | `Histogram` | `bytes` | +| `length_histogram` | `Histogram` | none (count) | +| `counter` | `Counter` | none | +| `gauge` | `Gauge` | none | + +> [!NOTE] +> Many `urc.*` metrics name low-level internals (the async runtime, the QUIC transport, storage compaction). Treat instrument names as observability surface, not a stability contract: the legacy `urc.*` prefixes and internal instruments can change between releases. The request metrics and the operation-duration metrics in the first sections below are the most stable and the most useful for alerting. + +## Shared attributes + +Two attribute keys recur across most instruments: + +| Attribute | Meaning | +| --- | --- | +| `context` | The operation being measured, for example `get`, `put`, `query`. A single namespace records one instrument across many `context` values. | +| `success` | `true` or `false`, set by the timing combinators from the operation's result. | + +Result-aware combinators also set `option=some`/`option=none` for optional results. Every exported metric additionally carries the configured `[telemetry.additional_labels]` and the resource attributes below. + +## Resource attributes + +Every exported metric, span, and log carries these resource attributes, set once at startup: + +| Attribute | Source | Default | +| --- | --- | --- | +| `service.name` | Environment variable `LORE_APP` | `lore` | +| `service.version` | Library version (`LORE_LIBRARY_VERSION`) | build version | +| `deployment.environment.name` | Environment variable `LORE_ENV` | unset | +| `instance` | Environment variable `PLATFORM_INSTANCE_ID` | unset | + +Plugin-contributed resource detectors and every `[telemetry.additional_labels]` entry are merged in as additional resource attributes. + +## Request metrics + +### HTTP server + +Recorded for every HTTP request by the HTTP metrics layer. Instrumentation scope `http`, using OpenTelemetry HTTP semantic-convention names. + +| Metric | Type | Unit | +| --- | --- | --- | +| `http.server.active_requests` | `UpDownCounter` | none | +| `http.server.request.duration` | `Histogram` | `s` | + +Attributes: + +| Attribute | On | Notes | +| --- | --- | --- | +| `http.request.method` | both | Request method. | +| `http.route` | both | Matched route, leading `/` stripped. Unmatched requests record `unmatched_path`. | +| `user_agent.name` | both | Normalized user-agent. Non-matching agents record ``; a missing header records ``. | +| `http.response.status_code` | duration | Response status, recorded on completion. | + +### gRPC server + +Recorded for every gRPC call by the gRPC metrics layer. Instrumentation scope `grpc`, using OpenTelemetry HTTP and RPC semantic-convention names. + +| Metric | Type | Unit | +| --- | --- | --- | +| `http.server.active_requests` | `UpDownCounter` | none | +| `http.server.request.duration` | `Histogram` | `s` | +| `rpc.server.duration` | `Histogram` | `s` | + +Attributes: `http.request.method`, `http.route`, `rpc.service`, `rpc.method`, `user_agent.name`, plus `rpc.grpc.status_code` and `http.response.status_code` on completion. + +> [!NOTE] +> Calls that return `Unimplemented` are deliberately not recorded, so probes for unsupported methods don't inflate the request metrics. + +## Operation-duration metrics + +A large set of subsystems share one pattern: a single `.operation_duration` latency histogram (`Histogram`, unit `milliseconds`), split by a `context` attribute naming the operation and a `success` attribute. Use `context` to break latency down per operation and `success` to isolate the error path. + +| Namespace | `context` values | +| --- | --- | +| `urc.store.immutable.local` | `obliterate`, `evict`, `compact` | +| `urc.store.immutable.aws` | `exist`, `get`, `put`, `exist_batch`, `obliterate`, `query`, `copy` | +| `urc.store.mutable.aws` | `list`, `load`, `store`, `compare_and_swap` | +| `urc.aws.s3` | `head_bucket`, `list_objects_v2`, `head_object`, `get_object`, `put_object`, `list_versions`, `delete_object` | +| `urc.aws.dynamodb` | `table_exists`, `get_item`, `batch_get_item`, `put_item`, `delete_item`, `transact_write_items`, `query_paginated`, `query_single` | +| `urc.quic.stream_handler` | `handle_message` | +| `urc.auth.jwk_service` | `get_keys` | +| `urc.authnz.urc_auth` | `lookup_user_permissions`, `check_user_permission` | +| `urc.authnz.rebac` | `create_resource`, `delete_resource` | +| `urc.replication.client` | `query`, `get`, `put`, `replication_put` | +| `urc.store.replicated` | `query`, `get`, `put`, `obliterate` (as `immutable.operation_duration`) | + +The gRPC service handlers use a parallel per-message latency histogram named `stream.message.handler.duration` (`Histogram`, `milliseconds`) instead of `operation_duration`: + +| Metric | `context` values | +| --- | --- | +| `urc.grpc.storage_service.stream.message.handler.duration` | `get`, `put`, `copy` | +| `urc.grpc.replication_service.stream.message.handler.duration` | `put` | + +## Storage metrics + +### Local immutable store + +Namespace `urc.store.immutable.local`. Alongside its `operation_duration` histogram (see above), the local store reports compaction sizing and a health snapshot. + +Compaction (recorded during compaction runs): + +| Metric | Type | +| --- | --- | +| `urc.store.immutable.local.compaction_target_size` | `Gauge` | +| `urc.store.immutable.local.compaction_total_size` | `Gauge` | +| `urc.store.immutable.local.compaction_group_target_size` | `Gauge` | +| `urc.store.immutable.local.compaction_group_evicted_size` | `Gauge` | +| `urc.store.immutable.local.compaction_group_final_total_size` | `Gauge` | +| `urc.store.immutable.local.compaction_final_size` | `Gauge` | +| `urc.store.immutable.local.compaction_group_evicted_count` | `Counter` | + +Health snapshot (sampled on the metrics interval; note the hyphen separators): + +| Metric | Type | Meaning | +| --- | --- | --- | +| `urc.store.immutable.local.fragment-count` | `Gauge` | Fragments held by the local store. | +| `urc.store.immutable.local.used-bytes` | `Gauge` | Bytes used on disk. | +| `urc.store.immutable.local.available` | `Gauge` | Bytes available. | + +### AWS DynamoDB + +Namespace `urc.aws.dynamodb`. In addition to `operation_duration`, the DynamoDB backend reports request-shape histograms. All carry the `table_name` attribute; `put_item` also carries `conditional`. + +| Metric | Type | Meaning | +| --- | --- | --- | +| `urc.aws.dynamodb.accumulation_operation_duration` | `Histogram` (ms) | Duration of a paginated accumulation. | +| `urc.aws.dynamodb.batch_operation_size` | `Histogram` | Items per batch operation. | +| `urc.aws.dynamodb.nested_api_calls.total` | `Histogram` | Underlying API calls per logical operation. | +| `urc.aws.dynamodb.unprocessed_retries.total` | `Histogram` | Retries for unprocessed items. | +| `urc.aws.dynamodb.query.accumulation.count` | `Histogram` | Items returned by an accumulating query. | +| `urc.aws.dynamodb.query.accumulation.scanned_count` | `Histogram` | Items scanned by an accumulating query. | + +### AWS S3, mutable store, immutable store + +Namespaces `urc.aws.s3`, `urc.store.mutable.aws`, and `urc.store.immutable.aws` each expose the shared `operation_duration` histogram documented above. + +### Composite store + +Namespace `urc.store.immutable.composite`. Carries a `replica_type` attribute. + +| Metric | Type | +| --- | --- | +| `urc.store.immutable.composite.get` / `.put` / `.query` / `.exist` / `.exist_batch` | `Counter` | +| `urc.store.immutable.composite.local.caching_total` | `Counter` | +| `urc.store.immutable.composite.get.inflight.receiver` | `Counter` | +| `urc.store.immutable.composite.topology.refresh.num_changes` | `Counter` | +| `urc.store.immutable.composite.topology.refresh.num_peer_errors` | `Counter` | +| `urc.store.immutable.composite.topology.refresh.num_peers` | `Gauge` | +| `urc.store.immutable.composite.topology.refresh.iteration.duration` | `Histogram` (ms) | + +### Replication + +Namespaces `urc.store.replicated` and `urc.replication.client` expose the shared `operation_duration` histograms above, plus: + +| Metric | Type | Meaning | +| --- | --- | --- | +| `urc.store.replicated.client.regenerate.duration` | `Histogram` (ms) | Client regeneration time. | +| `urc.replication.client.replication.message_dropped` | `Counter` | Dropped replication messages. | +| `urc.replication.client.replication.timeout` | `Counter` | Replication timeouts. | +| `urc.replication.client.replication.connect` / `.replication.disconnect` | `Counter` | Connect and disconnect events. | +| `urc.replication.client.replication.num_attempts` | `Histogram` | Attempts per replication. | + +## QUIC transport metrics + +The QUIC transport reports detailed connection health. These are low-level and the most likely to change between releases; they are most useful for transport debugging rather than long-term dashboards. + +### Connection monitor + +Namespace `urc.quic.client_monitor`, carrying a `context` attribute. + +| Metric | Type | Unit | +| --- | --- | --- | +| `urc.quic.client_monitor.connection.path.rtt` | `Histogram` | `milliseconds` | +| `urc.quic.client_monitor.connection.path.cwnd` | `Histogram` | `bytes` | + +Path gauges (`Gauge`): `.connection.path.congestion_events`, `.connection.path.lost_packets`, `.connection.path.lost_bytes`, `.connection.path.sent_packets`, `.connection.path.black_holes_detected`, `.connection.path.current_mtu`. + +UDP gauges (`Gauge`): `.connection.udp.tx.datagrams`, `.connection.udp.tx.bytes`, `.connection.udp.tx.ios`, `.connection.udp.rx.datagrams`, `.connection.udp.rx.bytes`, `.connection.udp.rx.ios`. + +Frame gauges (`Gauge`): `.connection.frame.tx.data_blocked`, `.connection.frame.tx.stream_data_blocked`, `.connection.frame.tx.streams_blocked_bidi`, `.connection.frame.tx.stream`, `.connection.frame.tx.reset_stream`, `.connection.frame.rx.data_blocked`, `.connection.frame.rx.stream_data_blocked`, `.connection.frame.rx.max_data`, `.connection.frame.rx.max_stream_data`. + +### Quinn connection + +Namespace `urc.quinn`. + +| Metric | Type | +| --- | --- | +| `urc.quinn.connection.data_blocked` | `Gauge` | +| `urc.quinn.connection.max_bidi_streams` | `Gauge` | +| `urc.quinn.connection.stream_data_blocked` | `Gauge` | +| `urc.quinn.connection.streams_blocked_bidi` | `Gauge` | +| `urc.quinn.connection.duration_seconds` | `Histogram` | + +### Stream handler and messages + +Namespace `urc.quic.stream_handler` reports, besides `operation_duration`: + +| Metric | Type | +| --- | --- | +| `urc.quic.stream_handler.stream.peak_pending_chunks` | `Histogram` | +| `urc.quic.stream_handler.stream.peak_pending_chunk_stall_duration` | `Histogram` | + +Storage message payload and content sizes (`Histogram`, unit `bytes`): + +| Metric | +| --- | +| `urc.quic.message.get.payload_size` | +| `urc.quic.message.get.content_size` | +| `urc.quic.message.put.payload_size` | +| `urc.quic.message.put.content_size` | + +## Service counters + +### Branch and repository counters + +Branch operations increment counters on both the legacy and v1 revision services: + +| Metric (legacy / v1 namespace) | Type | +| --- | --- | +| `urc.revision_service.num_branches_created` / `lore.revision.v1.revision_service.num_branches_created` | `Counter` | +| `urc.revision_service.num_branches_deleted` / `lore.revision.v1.revision_service.num_branches_deleted` | `Counter` | +| `urc.revision_service.num_branches_pushed` / `lore.revision.v1.revision_service.num_branches_pushed` | `Counter` | +| `urc.repository_service.num_repositories_created` / `lore.repository.v1.repository_service.num_repositories_created` | `Counter` | +| `urc.repository_service.num_repositories_deleted` / `lore.repository.v1.repository_service.num_repositories_deleted` | `Counter` | + +The revision services also report revision-list timing, split by `start_type` and `list_strategy` attributes: + +| Metric (per legacy / v1 namespace) | Type | +| --- | --- | +| `...revision_list.resolve_start.duration` | `Histogram` (ms) | +| `...revision_list.resolve_start.relative_age_seconds` | `Histogram` | +| `...revision_list.walk.duration` | `Histogram` (ms) | + +### Lock service + +Namespace `urc.lock_service`, split by `context` (`lock`, `status`, `unlock`, `admin_lock`). + +| Metric | Type | +| --- | --- | +| `urc.lock_service.locking.request.resources.length` | `Histogram` | +| `urc.lock_service.status.request.resources.length` | `Histogram` | + +### Task queue + +Namespace `urc.taskqueue`, carrying `name` and `context` attributes. + +| Metric | Type | +| --- | --- | +| `urc.taskqueue.submitted_tasks` | `Counter` | +| `urc.taskqueue.num_rate_limited` | `Counter` | +| `urc.taskqueue.num_task_result_send_failed` | `Counter` | +| `urc.taskqueue.num_task_work_send_failed` | `Counter` | +| `urc.taskqueue.active_tasks` | `Gauge` | +| `urc.taskqueue.pending_tasks` | `Gauge` | + +### Peer discovery + +Namespace `urc.hashicorp.service_peer_discovery`: + +| Metric | Type | +| --- | --- | +| `urc.hashicorp.service_peer_discovery.refresh_peers_loop.iteration.duration` | `Histogram` (ms) | + +### TLS certificate expiry + +Namespace `urc.certs`. Attributes: `cert_path`, `subject`, `serial`, `expiry_timestamp`. + +| Metric | Type | Unit | Description | +| --- | --- | --- | --- | +| `urc.certs.expiry_seconds` | `Gauge` | `seconds` | Seconds until the certificate expires (negative if expired). | + +Alert on this gauge crossing your renewal threshold to catch expiring certificates before connections fail. + +## Runtime and process metrics + +### Server liveness + +| Metric | Namespace | Type | Attributes | +| --- | --- | --- | --- | +| `urc.server.up` | `urc.server` | `Gauge` | `version`, `settings_hash` | + +### Task instrumentation + +Namespace `lore.runtime`, with `context_label`, `spawn_file`, and `spawn_line_number` attributes: + +| Metric | Type | +| --- | --- | +| `lore.runtime.tasks.spawned.total` | `Counter` | +| `lore.runtime.tasks.running.total` | `UpDownCounter` | + +### Async runtime bridge + +When the server is built with `tokio_unstable`, it bridges Tokio runtime and per-task metrics into OpenTelemetry under the `tokio_runtime` and `quicsrv_tokio_task` scopes (the latter carries a `quinn_server_name` attribute). These are raw runtime instruments (worker counts, poll and park counts, busy ratios, queue depths, poll and delay durations) intended for runtime-level performance debugging. They are the most implementation-specific instruments the server emits; consult the Tokio runtime-metrics documentation for the meaning of individual counters. + +## Traces + +Lore Server starts a new trace at each inbound request and records spans across request handling, storage, replication, and the QUIC transport. + +### Trace-root spans + +| Span | Surface | Notable fields | +| --- | --- | --- | +| `lore_http_tracing` | HTTP requests | `http.method`, `http.path`, `correlation_id`, `user_agent`, `repository_id`, `user_id` | +| `lore_tracing` | gRPC calls | `rpc.system`, `rpc.service`, `rpc.method`, `correlation_id`, `user_agent`, `repository_id`, `user_id` | +| `urc` | tower-http layer | `correlation_id`, `method`, `uri` | + +Child spans cover gRPC methods (per storage, replication, environment, admin, and notification service), per-fragment storage work, QUIC command dispatch (one span per wire command, such as `Get`, `Put`, `Query`, `Copy`), replication protocol operations, and branch-push work. Standard span field keys include `correlation_id`, `repository_id`, `partition_id`, `user_id`, `branch_id`, `revision`, `connection_id`, `transport`, `protocol`, and `opcode`. + +### Sampling + +Trace sampling is parent-based with a custom per-operation sampler. The base fraction is `[telemetry.traces].sample_rate` (default `0.05`); spans tagged as low-tier (high-volume) sample at `[telemetry.traces].sample_rate_low_tier` (default `0.001`). See [Metrics and traces](lore-server-config.md#metrics-and-traces) for the settings. + +> [!NOTE] +> The spans named `urc-quic` and `ReplicationClient::PutStreamImplementation` are filtered out of OTLP trace export by design, so they don't appear in your tracing backend even when sampled. + +## Examples + +### Enable OTLP export + +Add an exporter and metrics or traces tables to the server configuration, then point the endpoint at your collector. See the [Telemetry settings](lore-server-config.md#telemetry-settings) reference for every field. + +```toml +[telemetry.exporter] +endpoint = "http://collector:4317" +queue_size = 2048 +timeout = 5000 + +[telemetry.metrics] +sample_interval_millis = 10000 + +[telemetry.traces] +sample_rate = 0.05 +service_name = "lore-server" +``` + +### Request latency, split by route + +Query the `http.server.request.duration` histogram grouped by `http.route` and `http.response.status_code` to find slow or failing routes. + +### Storage error rate + +Filter any `operation_duration` metric (for example `urc.aws.s3.operation_duration`) by `success=false`, grouped by `context`, to see which storage operations are failing. + +## See also + +- [Lore Server configuration reference](lore-server-config.md) for the `[telemetry.*]` settings that turn export on and tune it. +- [OpenTelemetry documentation](https://opentelemetry.io/docs/) for OTLP collectors, exporters, and semantic conventions.