From 9345b25ac56fb1a913262e17e13864af832faac3 Mon Sep 17 00:00:00 2001 From: Alex Luong Date: Fri, 14 Aug 2026 14:40:59 +0700 Subject: [PATCH] docs: document event and delivery log storage and retention The event-delivery-log guide was a TODO stub, and CLICKHOUSE_LOG_RETENTION_TTL_DAYS was undocumented despite being the only retention mechanism Outpost implements. Covers what the log stores, when an event enters it, how the PostgreSQL and ClickHouse backends differ as the log grows, ClickHouse TTL, and self-managed retention for PostgreSQL. Adds the retention variable to the configuration reference. Refs hookdeck/outpost#1027 Co-Authored-By: Claude Opus 5 (1M context) --- docs/content/self-hosting/configuration.mdoc | 1 + .../guides/event-delivery-log.mdoc | 69 ++++++++++++++++++- 2 files changed, 68 insertions(+), 2 deletions(-) diff --git a/docs/content/self-hosting/configuration.mdoc b/docs/content/self-hosting/configuration.mdoc index 93b322bf3..b4ddda0ef 100644 --- a/docs/content/self-hosting/configuration.mdoc +++ b/docs/content/self-hosting/configuration.mdoc @@ -63,6 +63,7 @@ Choose one for event log persistence: |----------|-------------| | `POSTGRES_URL` | PostgreSQL connection URL | | `CLICKHOUSE_ADDR` | ClickHouse address (e.g., `localhost:9000`) | +| `CLICKHOUSE_LOG_RETENTION_TTL_DAYS` | Days to retain event and delivery logs in ClickHouse (default: `0`, retained indefinitely). No PostgreSQL equivalent; see the [Event & Delivery Log](/docs/outpost/self-hosting/guides/event-delivery-log) guide. | ## Delivery diff --git a/docs/content/self-hosting/guides/event-delivery-log.mdoc b/docs/content/self-hosting/guides/event-delivery-log.mdoc index 2fc3d7568..b2d251697 100644 --- a/docs/content/self-hosting/guides/event-delivery-log.mdoc +++ b/docs/content/self-hosting/guides/event-delivery-log.mdoc @@ -3,6 +3,71 @@ title: Event & Delivery Log description: "Overview of event and delivery log storage, querying, and retention behavior for self-hosted Outpost." --- -TODO +The event and delivery log is Outpost's record of what happened to each event: the event as it was received, and every delivery attempt made for it. It's what backs the events API, the tenant user portal, and manual retries. -Need to talk about data retention and retention policies. +This is separate from [application logging](/docs/outpost/self-hosting/guides/logging), which covers the stdout and audit logs your services produce. + +## What's stored + +Outpost records two kinds of thing: + +- **Events** — the payload and metadata as published, along with the tenant, topic, and time. +- **Delivery attempts** — one record per attempt against a destination, with the outcome, the destination's response, and the attempt number. + +Configuration data lives elsewhere. Tenants and destinations are stored in Redis, so removing log data never affects a tenant's configured destinations. + +### When an event is written + +An event is written to the log together with its first delivery attempt, rather than at the moment it's published. + +Outpost matches each published event against the tenant's destinations. An event with no matches produces no delivery attempt and isn't written to the log. + +## Choosing a backend + +Outpost stores logs in one of the following. Set the corresponding variable to select it, and see the [Configuration Reference](/docs/outpost/self-hosting/configuration) for the full set of connection variables. + +- **PostgreSQL** — `POSTGRES_URL` +- **ClickHouse** — `CLICKHOUSE_ADDR` + +PostgreSQL is the simpler choice if you already run it and don't expect to accumulate a large log. Note that it has no built-in retention, so you manage that yourself. + +ClickHouse holds its query performance as the log grows. That matters most for the [Metrics API](/docs/outpost/features/metrics), which aggregates across the same event and delivery data to produce charts, and where those aggregations get expensive on PostgreSQL once the log is large. It's what Hookdeck operates. + +How much log you accumulate depends on your event volume, payload sizes, and retry behavior, so benchmark against your own data. As a rough guide, PostgreSQL tends to be comfortable into the low millions of rows per table, or per partition if you partition the tables yourself. + +Switching backends later means migrating your existing log data, so it's worth deciding before you accumulate history. + +## Retention + +Retention determines how long event history stays queryable through the API and the tenant user portal, and how long an event remains available for manual retry. It's a product decision as much as a storage one: a seven day retention means your tenants can't retry anything older than a week. + +### ClickHouse + +Set `CLICKHOUSE_LOG_RETENTION_TTL_DAYS` to the number of days to keep. `0`, the default, retains logs indefinitely. + +| Variable | Default | Description | +|----------|---------|-------------| +| `CLICKHOUSE_LOG_RETENTION_TTL_DAYS` | `0` | Days to retain event and delivery logs. `0` retains indefinitely. | + +Outpost applies the value as a ClickHouse TTL on startup, so a change takes effect on the next restart and applies to existing data as well as new logs. + +### PostgreSQL + +Outpost has no built-in retention for PostgreSQL. Without an external retention process, the log grows indefinitely. + +The straightforward approach is a scheduled batched delete, oldest first, removing attempts before events: + +```sql +DELETE FROM attempts WHERE time < now() - interval '30 days'; +DELETE FROM events WHERE time < now() - interval '30 days'; +``` + +Run these in bounded batches rather than as a single statement, on whatever scheduler you already operate, such as a Kubernetes `CronJob`, a systemd timer, or [pg_cron](https://github.com/citusdata/pg_cron) if the extension is available to you. Both tables have a primary key leading with `time`, so these are index range scans rather than full table scans. You'll still take the dead tuple and vacuum cost that any large delete incurs. + +The `events` and `attempts` tables are declared as range partitioned on `time`, but Outpost creates only DEFAULT partitions and doesn't manage the partition lifecycle. Dropping expired partitions is cheaper than deleting rows at scale, so if you're running PostgreSQL at volume, managing partitions yourself with a tool like [pg_partman](https://github.com/pgpartman/pg_partman) is worth considering. + +## Querying + +Event and delivery history is available through the events endpoints of the [Outpost API](/docs/outpost/api), and to your tenants through the [tenant user portal](/docs/outpost/features/tenant-user-portal). The [Metrics API](/docs/outpost/features/metrics) aggregates over the same data. + +All three are backed by whichever log store you configured, so query performance and retention follow from that choice.