title	Integrations

Monitoring integrations

TrogonEventStore supports several methods to integrate with external monitoring and observability tools. Those include:

OpenTelemetry: export telemetry to an OpenTelemetry-compatible endpoint
Prometheus: collect metrics in Prometheus
Datadog: monitor and measure the cluster with Datadog
ElasticSearch: this section describes how to collect TrogonEventStore logs in ElasticSearch
Vector: collect metrics and logs to your APM tool using Vector

Prometheus

You can collect TrogonEventStore metrics to Prometheus and configure Grafana dashboards to monitor your deployment. TrogonEventStore exposes Prometheus metrics on /-/metrics. Refer to metrics documentation to learn more.

Older versions can be monitored by Prometheus using the community-supported exporter available in the GitHub repository.

OpenTelemetry Exporter

TrogonEventStore passively exposes metrics for scraping on the /-/metrics endpoint. It can also actively export logs, metrics, and traces using the OpenTelemetry Protocol (OTLP).

A number of APM providers natively support OTLP, so you might be able to send TrogonEventStore telemetry directly to your APM provider. Alternatively, you can export to the OpenTelemetry Collector, which can then fan out to a variety of backends. You can find out more about the OpenTelemetry collector.

Configuration

Sample JSON configuration:

{
  "EventStore": {
    "OpenTelemetry": {
      "Otlp": {
        "Endpoint": "http://localhost:4317",
        "Headers": ""
      },
      "Logs": {
        "Enabled": true
      },
      "Metrics": {
        "Otlp": {
          "Endpoint": "http://metrics-collector:4317"
        }
      },
      "Traces": {
        "Enabled": true,
        "Otlp": {
          "Endpoint": "http://traces-collector:4317"
        }
      }
    }
  }
}

The shared EventStore:OpenTelemetry:Otlp section provides defaults for every enabled OTLP signal. The EventStore:OpenTelemetry:Logs:Otlp, EventStore:OpenTelemetry:Metrics:Otlp, and EventStore:OpenTelemetry:Traces:Otlp sections can override only the settings that need to differ for each signal.

All OpenTelemetry environment variables are optional. Configure them only in deployment environments that should export telemetry to an OTLP collector.

The configuration can specify:

Name	Description
EventStore__OpenTelemetry__Otlp__Endpoint	Shared destination where the OTLP exporter will send telemetry
EventStore__OpenTelemetry__Otlp__Headers	Optional shared headers for the connection
EventStore__OpenTelemetry__Logs__Enabled	Enables OTLP log export
EventStore__OpenTelemetry__Logs__Otlp__Endpoint	Optional log-specific OTLP destination
EventStore__OpenTelemetry__Metrics__Enabled	Enables OTLP metric export from runtime configuration
EventStore__OpenTelemetry__Metrics__Otlp__Endpoint	Optional metric-specific OTLP destination
EventStore__OpenTelemetry__Traces__Enabled	Enables OTLP trace export from runtime configuration
EventStore__OpenTelemetry__Traces__Otlp__Endpoint	Optional trace-specific OTLP destination

Headers are key-value pairs separated by commas. For example:

"Headers": "api-key=value,other-config-value=value"

The interval is taken from the ExpectedScrapeIntervalSeconds value in metricsconfig.json in the server installation directory:

"ExpectedScrapeIntervalSeconds": 15

Troubleshooting

Symptom	Solution
Logs are not exported	Check that `EventStore__OpenTelemetry__Logs__Enabled` is set to `true`.
Metrics are not exported	Check that `EventStore__OpenTelemetry__Metrics__Enabled` is `true`, any `EventStore__OpenTelemetry__Metrics__Otlp__*` key is set, or `Otlp.Enabled` is `true` in `metricsconfig.json`.
Traces are not exported	Check that `EventStore__OpenTelemetry__Traces__Enabled` is `true` or any `EventStore__OpenTelemetry__Traces__Otlp__*` key is set.
Telemetry arrives at the wrong collector	Check whether a per-signal `Logs:Otlp`, `Metrics:Otlp`, or `Traces:Otlp` section is overriding the shared `EventStore:OpenTelemetry:Otlp` destination.

Datadog

The best way to integrate TrogonEventStore telemetry with Datadog today is by using the built-in OpenTelemetry export support.

You can use the community-supported integration to collect TrogonEventStore logs and metrics in Datadog.

Find out more details about the integration in Datadog documentation.

Vector

Vector is a lightweight and ultra-fast tool for building observability pipelines. (from Vector website)

You can use Vector for extracting metrics or logs from a self-managed TrogonEventStore server.

Installation

Follow the installation instructions provided by Vector to deploy the agent. You can deploy and run it on the same machine where you run TrogonEventStore server. If you run TrogonEventStore in Kubernetes, you can run Vector as a sidecar for each of the TrogonEventStore pods.

Configuration

Each Vector instance needs to be configured with sources and sinks. When configured properly, it will collect information from each source, apply the necessary transformation (if needed), and send the transformed information to the configured sink.

Vector provides many different sinks, you most probably will find your preferred monitoring platform among those sinks.

Collecting metrics

Vector's historical eventstoredb_metrics source depended on the legacy /stats HTTP endpoint. That endpoint is no longer part of the supported management surface, so new deployments should collect metrics from the Prometheus or OpenTelemetry endpoints instead and use the gRPC monitoring service for runtime diagnostics that need grouped stats or queue-level detail.

Collecting logs

To collect logs, you can use the file source and configure it to target TrogonEventStore log file. For log collection, Vector must run on the same machine as TrogonEventStore server as it collects the logs from files on the local file system.

[sources.eventstoredb_logs]
type = "file"
# If you changed the default log location, please update the filepath accordingly.
include = ["/var/log/eventstore"]
read_from = "end"

Example

In this example, Vector runs on the same machine as TrogonEventStore, collects logs, and then sends them to Datadog.

[sources.eventstoredb_logs]
type = "file"
include = ["/var/log/eventstore"]
read_from = "end"

[sinks.dd_logs]
type = "datadog_logs"
inputs = ["sources.eventstoredb_logs"]
default_api_key = "${DD_API_KEY}"
compression = "gzip"

ElasticSearch

Elastic Stack is one of the most popular tools for ingesting and analyzing logs and statistics:

Elasticsearch was built for advanced filtering and text analysis.
Filebeat allow tailing files efficiently.
Logstash enables log transformations and processing pipelines.
Kibana is a dashboard and visualization UI for Elasticsearch data.

TrogonEventStore exposes structured information through its logs and statistics, allowing straightforward integration with mentioned tooling.

Logstash

Logstash is the plugin based data processing component of the Elastic Stack which sends incoming data to Elasticsearch. It's excellent for building a text-based processing pipeline. It can also gather logs from files (although Elastic recommends now Filebeat for that, see more in the following paragraphs). Logstash needs to either be installed on the TrogonEventStore node or have access to logs storage. The processing pipeline can be configured through the configuration file (e.g. logstash.conf). This file contains the three essential building blocks:

input - source of logs, e.g. log files, system output, Filebeat.
filter - processing pipeline, e.g. to modify, enrich, tag log data,
output - place where we'd like to put transformed logs. Typically that contains Elasticsearch configuration.

See the sample Logstash 8.2 configuration file. It shows how to take the TrogonEventStore log files, split them based on the log type (regular and stats) and output them to separate indices to Elasticsearch:

#######################################################
#  TrogonEventStore logs file input
#######################################################
input {
  file {
    path => "/var/log/eventstore/*/log*.json"
    start_position => "beginning"
    codec => json
  }
}

#######################################################
#  Filter out stats from regular logs
#  add respecting field with log type
#######################################################
filter {
  # check if log path includes "log-stats"
  # so pattern for stats
  if [log][file][path] =~ "log-stats" {
    mutate {
      add_field => {
        "log_type" => "stats"
      }
    }
  }
  else {
    mutate {
      add_field => {
        "log_type" => "logs"
      }
    }
  }
}

#######################################################
#  Send logs to Elastic
#  Create separate indexes for stats and regular logs
#  using field defined in the filter transformation
#######################################################
output {
  elasticsearch {
    hosts => [ "elasticsearch:9200" ]
    index => 'eventstoredb-%{[log_type]}'
  }
}

You can play with such configuration through the sample docker-compose.

Filebeat

Logstash was an initial attempt by Elastic to provide a log harvester tool. However, it appeared to have performance limitations. Elastic came up with the Beats family, which allows gathering data from various specialized sources (files, metrics, network data, etc.). Elastic recommends Filebeat as the log collection and shipment tool off the host servers. Filebeat uses a backpressure-sensitive protocol when sending data to Logstash or Elasticsearch to account for higher volumes of data.

Filebeat can pipe logs directly to Elasticsearch and set up a Kibana data view.

Filebeat needs to either be installed on the TrogonEventStore node or have access to logs storage. The processing pipeline can be configured through the configuration file (e.g. filebeat.yml). This file contains the three essential building blocks:

input - configuration for file source, e.g. if stored in JSON format.
output - place where we'd like to put transformed logs, e.g. Elasticsearch, Logstash,
setup - additional setup and simple transformations (e.g. Elasticsearch indices template, Kibana data view).

See the sample Filebeat 8.2 configuration file. It shows how to take the TrogonEventStore log files, output them to Elasticsearch prefixing index with eventstoredb and create a Kibana data view:

#######################################################
#  TrogonEventStore logs file input
#######################################################
filebeat.inputs:
  - type: log
    paths:
      - /var/log/eventstore/*/log*.json
    json.keys_under_root: true
    json.add_error_key: true

#######################################################
#  ElasticSearch direct output
#######################################################
output.elasticsearch:
  index: "eventstoredb-%{[agent.version]}"
  hosts: ["elasticsearch:9200"]

#######################################################
#  ElasticSearch dashboard configuration
#  (index pattern and data view)
#######################################################
setup.dashboards:
  enabled: true
  index: "eventstoredb-*"

setup.template:
  name: "eventstoredb"
  pattern: "eventstoredb-%{[agent.version]}"

#######################################################
#  Kibana dashboard configuration
#######################################################
setup.kibana:
  host: "kibana:5601"

You can play with such configuration through the sample docker-compose.

Filebeat with Logstash

Even though Filebeat can pipe logs directly to Elasticsearch and do a basic Kibana setup, you'd like to have more control and expand the processing pipeline. That's why for production, it's recommended to use both. Multiple Filebeat instances (e.g. from different TrogonEventStore clusters) can collect logs and pipe them to Logstash, which will play an aggregator role. Filebeat can output logs to Logstash, and Logstash can receive and process these logs with the Beats input. Logstash can transform and route logs to Elasticsearch instance(s).

In that configuration, Filebeat should be installed on the TrogonEventStore node (or have access to file logs) and define Logstash as output. See the sample Filebeat 8.2 configuration file.

#######################################################
#  TrogonEventStore logs file input
#######################################################
filebeat.inputs:
  - type: log
    paths:
      - /var/log/eventstore/*/log*.json
    json.keys_under_root: true
    json.add_error_key: true

#######################################################
#  Logstash output to transform and prepare logs
#######################################################
output.logstash:
  hosts: ["logstash:5044"]

Then the sample Logstash 8.2 configuration file will look like the below. It shows how to take the TrogonEventStore logs from Filebeat, split them based on the log type (regular and stats) and output them to separate indices to Elasticsearch:

#######################################################
#  Filebeat input 
#######################################################
input {
  beats {
    port => 5044
  }
}

#######################################################
#  Filter out stats from regular logs
#  add respecting field with log type
#######################################################
filter {
  # check if log path includes "log-stats"
  # so pattern for stats
  if [log][file][path] =~ "log-stats" {
    mutate {
      add_field => {
        "log_type" => "stats"
      }
    }
  }
  else {
    mutate {
      add_field => {
        "log_type" => "logs"
      }
    }
  }
}

#######################################################
#  Send logs to Elastic
#  Create separate indexes for stats and regular logs
#  using field defined in the filter transformation
#######################################################
output {
  elasticsearch {
    hosts => [ "elasticsearch:9200" ]
    index => 'eventstoredb-%{[log_type]}'
  }
}

You can play with such configuration through the sample docker-compose.

FilesExpand file tree

integrations.md

Latest commit

History

integrations.md

File metadata and controls

Monitoring integrations

Prometheus

OpenTelemetry Exporter

Configuration

Troubleshooting

Datadog

Vector

Installation

Configuration

Collecting metrics

Collecting logs

Example

ElasticSearch

Logstash

Filebeat

Filebeat with Logstash