Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,6 @@ ipch/
.DS_Store

src/EventStore/EventStore.Common/Properties/AssemblyVersion.cs
src/EventStore/EventStore.ClientAPI/Properties/AssemblyVersion.cs

*.o
*.ii
*.s
Expand Down
3 changes: 1 addition & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ FROM mcr.microsoft.com/dotnet/sdk:10.0-${CONTAINER_RUNTIME} AS test
WORKDIR /build
COPY --from=build ./build/published-tests ./published-tests
COPY --from=build ./build/ci ./ci
COPY --from=build ./build/src/EventStore.Core.Tests/Services/Transport/Tcp/test_certificates/ca/ca.crt /usr/local/share/ca-certificates/ca_eventstore_test.crt
COPY ./scripts/test.sh /build/test.sh
RUN mkdir ./test-results
RUN chmod +x /build/test.sh
Expand Down Expand Up @@ -89,7 +88,7 @@ ReplicationIp: 0.0.0.0" >> /etc/eventstore/eventstore.conf

VOLUME /var/lib/eventstore /var/log/eventstore

EXPOSE 1112/tcp 1113/tcp 2113/tcp
EXPOSE 1112/tcp 2113/tcp

HEALTHCHECK --interval=5s --timeout=5s --retries=24 \
CMD curl --fail --insecure https://localhost:2113/-/liveness || curl --fail http://localhost:2113/-/liveness || exit 1
Expand Down
3 changes: 3 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ services:
- shared.env
environment:
- EVENTSTORE_GOSSIP_SEED=172.30.240.12:2113,172.30.240.13:2113
- EVENTSTORE_NODE_IP=172.30.240.11
Comment thread
cursor[bot] marked this conversation as resolved.
- EVENTSTORE_REPLICATION_IP=172.30.240.11
- EVENTSTORE_CERTIFICATE_FILE=/etc/eventstore/certs/node/node.crt
- EVENTSTORE_CERTIFICATE_PRIVATE_KEY_FILE=/etc/eventstore/certs/node/node.key
Expand Down Expand Up @@ -44,6 +45,7 @@ services:
- shared.env
environment:
- EVENTSTORE_GOSSIP_SEED=172.30.240.11:2113,172.30.240.13:2113
- EVENTSTORE_NODE_IP=172.30.240.12
- EVENTSTORE_REPLICATION_IP=172.30.240.12
- EVENTSTORE_CERTIFICATE_FILE=/etc/eventstore/certs/node/node.crt
- EVENTSTORE_CERTIFICATE_PRIVATE_KEY_FILE=/etc/eventstore/certs/node/node.key
Expand Down Expand Up @@ -71,6 +73,7 @@ services:
- shared.env
environment:
- EVENTSTORE_GOSSIP_SEED=172.30.240.11:2113,172.30.240.12:2113
- EVENTSTORE_NODE_IP=172.30.240.13
- EVENTSTORE_REPLICATION_IP=172.30.240.13
- EVENTSTORE_CERTIFICATE_FILE=/etc/eventstore/certs/node/node.crt
- EVENTSTORE_CERTIFICATE_PRIVATE_KEY_FILE=/etc/eventstore/certs/node/node.key
Expand Down
12 changes: 8 additions & 4 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,13 @@ surfaces for running a node or cluster.

TrogonEventStore keeps the database node focused on the durable event log:

- Application event access is gRPC-first.
- HTTP is reserved for the Admin UI, health probes, metrics, and other
infrastructure-level concerns.
- Database client APIs, cluster coordination, and follower-to-leader forwarding
use gRPC over the node HTTP(S) endpoint. Database replication uses gRPC over
a dedicated replication HTTP(S) endpoint.
- Regular HTTP routes are reserved for the Admin UI, health probes, metrics,
and other infrastructure-level concerns.
- The server does not open a separate legacy EventStore TCP protocol listener
or support its TCP transport configuration.
- The project is FOSS-only. The documentation does not describe unsupported
proprietary server features.
- Rich read models, user-defined query engines, connector runtimes, and
Expand All @@ -37,7 +41,7 @@ For a production node, review:

## Protocols and clients

The supported application protocol is gRPC. Existing TrogonEventStore-compatible
The supported database protocol is gRPC. Existing TrogonEventStore-compatible
gRPC clients can be useful while the TrogonDB client libraries continue to
evolve, but the server documentation should be treated as authoritative for this
repository.
Expand Down
5 changes: 3 additions & 2 deletions docs/admin-ui.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The TrogonEventStore Admin UI is available at `http://SERVER_IP:2113/ui` and hel

The dashboard opens at `/ui` and combines the daily operational view in one place:

- _Cluster status_: live gossip membership, node state, checkpoints, TCP and HTTP endpoints, replica status, and a copy-friendly snapshot.
- _Cluster status_: live gossip membership, node state, checkpoints, HTTP(S) endpoints, replica status, and a copy-friendly snapshot.
- _Queue pressure_: live queue length, throughput, processing time, and currently processed messages.
- _Node probes_: inline Ping, Node info, and Gossip checks rendered inside the UI.

Expand All @@ -24,7 +24,8 @@ The _Observability_ page focuses on runtime diagnostics:

- queue groups and individual queue rows
- current and last processed messages
- TCP connection statistics
- active connections on the node and replication HTTP/gRPC endpoints, including client identity, protocol, security, traffic rates, totals, and pending bytes
- live gRPC replication sessions, byte totals, pending bytes, and send queue depth
- snapshot output for copy-paste debugging

## Configuration
Expand Down
11 changes: 11 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,17 @@ and sharding work. Features that need their own compute model, query model, or
serving model should be separate components that consume the database through
subscriptions or reads.

## Network protocol boundary

gRPC carries database client APIs, cluster coordination, and follower-to-leader
forwarding over the node HTTP(S) endpoint. Ordinary HTTP routes on that endpoint
serve the Admin UI, health probes, metrics, and other operator workflows.
Database replication uses gRPC over a dedicated replication HTTP(S) endpoint so
operators can isolate high-volume replication traffic from clients.

Both endpoints use the same TLS and node identity configuration. The node has no
separate legacy EventStore TCP protocol listener or configuration surface.

## Projection execution

Projection execution is future external component work by default.
Expand Down
18 changes: 13 additions & 5 deletions docs/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,15 +55,22 @@ The multi-address DNS name cluster discovery only works for clusters that use ce

### Internal communication

When setting up a cluster, the nodes must be able to reach each other over both the HTTP channel, and the internal TCP channel. You should ensure that these ports are open on firewalls on the machines and between the machines.
Cluster nodes use gRPC over each node's dedicated replication HTTP(S) endpoint
for database replication. Gossip, elections, and follower-to-leader request
forwarding use the node HTTP(S) endpoint. Ensure every node can reach both
advertised endpoints on every other node.

Learn more about [internal TCP configuration](networking.md#replication-protocol) and [HTTP configuration](networking.md#http-configuration) to set up the cluster properly.
Learn more about the [node](networking.md#http-configuration) and
[replication](networking.md#internal-cluster-traffic) endpoints before
configuring cluster firewall or network policy rules.

## Cluster with DNS

When you tell TrogonEventStore to use DNS for its gossip, the server will resolve the DNS name to a list of IP addresses and connect to each of those addresses to find other nodes. This method is very flexible because you can change the list of nodes on your DNS server without changing the cluster configuration. The DNS method is also useful in automated deployment scenarios when you control both the cluster deployment and the DNS server from your infrastructure-as-code scripts.

To use DNS discovery, you need to set the `ClusterDns` option to the DNS name that allows making an HTTP call to it. When the server starts, it will attempt to make a gRPC call using the `https://<cluster-dns>:<gossip-port>` URL (`http` if the cluster is insecure).
To use DNS discovery, set the `ClusterDns` option to a DNS name that resolves to
the cluster nodes. When the server starts, it attempts a gRPC call over
`https://<cluster-dns>:<gossip-port>` (`http` if the cluster is insecure).

When using a certificate signed by a publicly trusted CA, you'd normally use the wildcard certificate. Ensure that the cluster DNS name fits the wildcard, otherwise the request will fail on SSL check.

Expand Down Expand Up @@ -107,7 +114,8 @@ The setting accepts a comma-separated list of IP addresses or host names with th

TrogonEventStore uses a quorum-based replication model. When working normally, a cluster has one node known as a leader, and the remaining nodes are followers. The leader node is responsible for coordinating writes while it is the leader. Cluster nodes use a consensus algorithm to determine which node should be the leader and which should be followers. TrogonEventStore bases the decision as to which node should be the leader on a number of factors.

For a cluster node to have this information available to them, the nodes gossip with other nodes in the cluster. Gossip runs over HTTP interfaces of cluster nodes.
For a cluster node to have this information available to them, the nodes gossip
with other nodes in the cluster over the node HTTP(S) endpoint.

The gossip protocol configuration can be changed using the settings listed below. Pay attention to the settings related to time, like intervals and timeouts, when running in a cloud environment.

Expand Down Expand Up @@ -229,7 +237,7 @@ candidate.

### Follower

A cluster assigns the follower role based on an election process. A cluster uses one or more nodes with the follower role to form the quorum, or the majority of nodes necessary to confirm that the write is persisted.
A cluster assigns the follower role based on an election process. A cluster uses one or more nodes with the follower role to form the quorum, or the majority of nodes necessary to confirm that the write is persisted. When a follower accepts a request that must run on the leader, it forwards the request to the leader over gRPC on the leader's HTTP(S) endpoint.

### Read-only replica

Expand Down
29 changes: 17 additions & 12 deletions docs/diagnostics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ TrogonEventStore provides several ways to diagnose and troubleshoot issues.
- [Metrics](metrics.md): collect standard metrics using Prometheus or OpenTelemetry.
- [Monitoring and alerting](monitoring.md): turn health, metrics, and logs into operational signals.
- [Stats](#statistics): runtime statistics exposed through the monitoring gRPC service.
- [Connection statistics](#connection-statistics): active HTTP and gRPC connections on both listeners.

You can also use external tools to measure the performance of TrogonEventStore and monitor the cluster health. Learn more on the [Integrations](./integrations.md) page.

Expand All @@ -19,6 +20,21 @@ cluster-wide state.
`Monitoring.Stats` collects fresh node-local statistics and memoizes the result for up to one second. It does
not read persisted statistics events.

## Connection statistics

Use the `Monitoring.ConnectionStats` RPC to inspect active connections on the
node and replication HTTP(S) listeners. Each result identifies the local and
remote endpoints, connection ID, observed client name, protocol, application,
TLS state, connection time, total bytes, and pending bytes.

Use `Monitoring.ReplicationStats` when the diagnostic question is specifically
about live database replication sessions. Its results include the subscription
and connection IDs, peer endpoint, byte totals, pending bytes, and send queue
depth.

Both RPCs report node-local snapshots. Query each cluster member when diagnosing
a cluster-wide connection or replication problem.

When statistics persistence is enabled, each node writes events to a reserved `$stats-<host:port>` stream. For
example, a single local node writes to `$stats-127.0.0.1:2113`.

Expand Down Expand Up @@ -49,17 +65,6 @@ type `$statsCollected`.
"proc-gc-largeHeapSize": 0,
"proc-gc-timeInGc": 0.0,
"proc-gc-totalBytesInHeaps": 0,
"proc-tcp-connections": 0,
"proc-tcp-receivingSpeed": 0.0,
"proc-tcp-sendingSpeed": 0.0,
"proc-tcp-inSend": 0,
"proc-tcp-measureTime": "00:00:19.0534210",
"proc-tcp-pendingReceived": 0,
"proc-tcp-pendingSend": 0,
"proc-tcp-receivedBytesSinceLastRun": 0,
"proc-tcp-receivedBytesTotal": 0,
"proc-tcp-sentBytesSinceLastRun": 0,
"proc-tcp-sentBytesTotal": 0,
"es-checksum": 1613144,
"es-checksumNonFlushed": 1613144,
"sys-drive-/System/Volumes/Data-availableBytes": 545628151808,
Expand Down Expand Up @@ -104,7 +109,7 @@ type `$statsCollected`.
"es-queue-MonitoringQueue-lengthLifetimePeak": 0,
"es-queue-MonitoringQueue-totalItemsProcessed": 14,
"es-queue-MonitoringQueue-inProgressMessage": "<none>",
"es-queue-MonitoringQueue-lastProcessedMessage": "GetFreshTcpConnectionStats",
"es-queue-MonitoringQueue-lastProcessedMessage": "GetFreshStats",
"es-queue-PersistentSubscriptions-queueName": "PersistentSubscriptions",
"es-queue-PersistentSubscriptions-groupName": "",
"es-queue-PersistentSubscriptions-avgItemsPerSecond": 1,
Expand Down
3 changes: 3 additions & 0 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,9 @@ Before running a durable node or cluster:
- Store data, index, and logs on durable volumes.
- Expose `/-/liveness`, `/-/readiness`, and `/-/metrics` to the platform.
- Use gRPC clients for application reads and writes.
- Expose the node HTTP(S) endpoint to clients and operators. Allow peer nodes to
reach the dedicated replication HTTP(S) endpoint on the private cluster
network. No legacy EventStore TCP protocol listener is required or supported.

## Linux service notes

Expand Down
Loading
Loading