Skip to content

docs: make Helm the primary install method on ClickStack OTel collector page #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mintlify wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: make Helm the primary install method on ClickStack OTel collector page #115

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
227 changes: 217 additions & 10 deletions products/clickstack/ingesting-data/collector.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,56 @@ Users deploying OTel collectors in the agent role will typically use the [defaul

We [recommend using the official ClickStack distribution of the collector](/products/clickstack/deployment/hyperdx-only#otel-collector) for the gateway role when sending to Managed ClickStack, where possible. If you choose to bring your own, ensure it includes the [ClickHouse exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter).

The collector can be deployed either via Helm (recommended for Kubernetes) or via Docker. The official [ClickStack Helm chart](https://github.com/ClickHouse/ClickStack-helm-charts) embeds the upstream [OpenTelemetry collector Helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts) as a subchart with the ClickStack distribution image preconfigured—see the [ClickStack Helm deployment guide](/products/clickstack/deployment/helm) if you want to install the full stack including HyperDX. For a standalone collector deployment, the upstream chart can be used directly with the ClickStack image, as shown below.

<Tabs>

<Tab title="Helm">

Add the upstream OpenTelemetry Helm repository:

```shell
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
```

Create a `values.yaml` configuring the ClickStack image and Managed ClickStack credentials:

```yaml
# values.yaml
mode: deployment

image:
repository: docker.clickhouse.com/clickhouse/clickstack-otel-collector
tag: "2.19.0"

ports:
otlp:
enabled: true
otlp-http:
enabled: true

extraEnvs:
- name: CLICKHOUSE_ENDPOINT
value: "https://your-instance.clickhouse.cloud:8443"
- name: CLICKHOUSE_USER
value: "default"
- name: CLICKHOUSE_PASSWORD
value: "<password>"
```

Install the chart:

```shell
helm install clickstack-otel-collector open-telemetry/opentelemetry-collector -f values.yaml
```

For production deployments, we recommend storing `CLICKHOUSE_PASSWORD` in a Kubernetes secret and referencing it via `extraEnvsFrom` instead of inlining the value.

</Tab>

<Tab title="Docker">

To deploy the ClickStack distribution of the OTel connector in a standalone mode, run the following docker command:

```shell
Expand All @@ -52,7 +102,11 @@ docker run -e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} -e CLICKHOUSE_USER=defa
ClickStack images are now published as `clickhouse/clickstack-*` (previously `docker.hyperdx.io/hyperdx/*`).
</Info>

Note that we can overwrite the target ClickHouse instance with environment variables for `CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME`, and `CLICKHOUSE_PASSWORD`. The `CLICKHOUSE_ENDPOINT` should be the full ClickHouse Cloud HTTP endpoint, including the protocol and port—for example, `https://99rr6dm6v3.us-central1.gcp.clickhouse.cloud:8443`.
</Tab>

</Tabs>

The target ClickHouse instance is configured via the environment variables `CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME`, and `CLICKHOUSE_PASSWORD`. The `CLICKHOUSE_ENDPOINT` should be the full ClickHouse Cloud HTTP endpoint, including the protocol and port—for example, `https://99rr6dm6v3.us-central1.gcp.clickhouse.cloud:8443`.

For details on retrieving your Managed ClickStack credentials, see [here](/products/cloud/guides/sql-console/connection-details).

Expand All @@ -66,9 +120,34 @@ You should use a user with the [appropriate credentials](/products/clickstack/in

#### Configuring Managed ClickStack instance {#configuring-managed-clickstack}

All docker images, which include the OpenTelemetry collector, can be configured to use a Managed ClickStack instance via the environment variables `CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME` and `CLICKHOUSE_PASSWORD`:
The OpenTelemetry collector can be configured to use a Managed ClickStack instance via the environment variables `CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME` and `CLICKHOUSE_PASSWORD`. How these are set depends on your deployment method:

<Tabs>

<Tab title="Helm">

For example the all-in-one image:
Override the relevant entries under `extraEnvs` in your `values.yaml`, then upgrade the release:

```yaml
# values.yaml
extraEnvs:
- name: CLICKHOUSE_ENDPOINT
value: "<HTTPS_ENDPOINT>"
- name: CLICKHOUSE_USER
value: "<CLICKHOUSE_USER>"
- name: CLICKHOUSE_PASSWORD
value: "<CLICKHOUSE_PASSWORD>"
```

```shell
helm upgrade clickstack-otel-collector open-telemetry/opentelemetry-collector -f values.yaml
```

</Tab>

<Tab title="Docker">

All docker images which include the OpenTelemetry collector can be configured via environment variables. For example the all-in-one image:

```shell
export CLICKHOUSE_ENDPOINT=<HTTPS ENDPOINT>
Expand All @@ -80,6 +159,10 @@ export CLICKHOUSE_PASSWORD=<CLICKHOUSE_PASSWORD>
docker run -e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} -p 8080:8080 -p 4317:4317 -p 4318:4318 clickhouse/clickstack-otel-collector:latest
```

</Tab>

</Tabs>

<ExtendingConfig/>

#### Docker Compose {#docker-compose-otel-managed}
Expand Down Expand Up @@ -113,6 +196,62 @@ With Docker Compose, modify the collector configuration using the same environme

If you're managing your own OpenTelemetry collector in a standalone deployment - such as when using the HyperDX-only distribution - we [recommend still using the official ClickStack distribution of the collector](/products/clickstack/deployment/hyperdx-only#otel-collector) for the gateway role where possible, but if you choose to bring your own, ensure it includes the [ClickHouse exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter).

The collector can be deployed either via Helm (recommended for Kubernetes) or via Docker. The official [ClickStack Helm chart](https://github.com/ClickHouse/ClickStack-helm-charts) embeds the upstream [OpenTelemetry collector Helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts) as a subchart and auto-wires the OpAMP endpoint, ClickStack image, and HyperDX API key via a shared `clickstack-config` ConfigMap and `clickstack-secret` Secret—see the [ClickStack Helm deployment guide](/products/clickstack/deployment/helm) if you want to install the full stack including HyperDX. For a standalone collector deployment that connects to an existing HyperDX, the upstream chart can be used directly with the ClickStack image, as shown below.

<Tabs>

<Tab title="Helm">

Add the upstream OpenTelemetry Helm repository:

```shell
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
```

Create a `values.yaml` configuring the ClickStack image, ClickHouse credentials, and the OpAMP endpoint of your HyperDX deployment:

```yaml
# values.yaml
mode: deployment

image:
repository: docker.clickhouse.com/clickhouse/clickstack-otel-collector
tag: "2.19.0"

ports:
otlp:
enabled: true
otlp-http:
enabled: true

extraEnvs:
- name: CLICKHOUSE_ENDPOINT
value: "tcp://clickhouse.your-namespace.svc.cluster.local:9000?dial_timeout=10s"
- name: CLICKHOUSE_USER
value: "otelcollector"
- name: CLICKHOUSE_PASSWORD
value: "<password>"
- name: OPAMP_SERVER_URL
value: "http://hyperdx.your-namespace.svc.cluster.local:4320"
- name: HYPERDX_API_KEY
value: "<your-ingestion-api-key>"
```

Install the chart:

```shell
helm install clickstack-otel-collector open-telemetry/opentelemetry-collector -f values.yaml
```

The `OPAMP_SERVER_URL` should resolve to your HyperDX service. When HyperDX and the collector run in the same cluster, use the in-cluster service DNS name (e.g. `http://hyperdx.your-namespace.svc.cluster.local:4320`). HyperDX exposes the OpAMP API at `/v1/opamp` on port `4320` by default.

For production deployments, we recommend storing `CLICKHOUSE_PASSWORD` and `HYPERDX_API_KEY` in a Kubernetes secret and referencing them via `extraEnvsFrom` instead of inlining the values.

</Tab>

<Tab title="Docker">

To deploy the ClickStack distribution of the OTel connector in a standalone mode, run the following docker command:

```shell
Expand All @@ -125,10 +264,6 @@ docker run -e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} -e CLICKHOUSE_ENDPOINT=${CLIC
ClickStack images are now published as `clickhouse/clickstack-*` (previously `docker.hyperdx.io/hyperdx/*`).
</Info>

Note that we can overwrite the target ClickHouse instance with environment variables for `CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME`, and `CLICKHOUSE_PASSWORD`. The `CLICKHOUSE_ENDPOINT` should be the full ClickHouse HTTP endpoint, including the protocol and port—for example, `http://localhost:8123`.

**These environment variables can be used with any of the docker distributions which include the connector.**

The `OPAMP_SERVER_URL` should point to your HyperDX deployment - for example, `http://localhost:4320`. HyperDX exposes an OpAMP (Open Agent Management Protocol) server at `/v1/opamp` on port `4320` by default. Make sure to expose this port from the container running HyperDX (e.g., using `-p 4320:4320`).

<Info>
Expand All @@ -137,6 +272,14 @@ The `OPAMP_SERVER_URL` should point to your HyperDX deployment - for example, `h
For the collector to connect to the OpAMP port it must be exposed by the HyperDX container e.g. `-p 4320:4320`. For local testing, OSX users can then set `OPAMP_SERVER_URL=http://host.docker.internal:4320`. Linux users can start the collector container with `--network=host`.
</Info>

</Tab>

</Tabs>

The target ClickHouse instance is configured via the environment variables `CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME`, and `CLICKHOUSE_PASSWORD`. The `CLICKHOUSE_ENDPOINT` should be the full ClickHouse HTTP endpoint, including the protocol and port—for example, `http://localhost:8123`.

**These environment variables can be used with any of the docker distributions which include the connector.**

<Info>
**Production user**

Expand All @@ -147,9 +290,36 @@ You should use a user with the [appropriate credentials](/products/clickstack/in

#### Configuring ClickHouse instance {#configuring-clickhouse-instance}

All docker images, which include the OpenTelemetry collector, can be configured to use a clickhouse instance via the environment variables `OPAMP_SERVER_URL` ,`CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME` and `CLICKHOUSE_PASSWORD`:
The OpenTelemetry collector can be configured to use a ClickHouse instance via the environment variables `OPAMP_SERVER_URL`, `CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME` and `CLICKHOUSE_PASSWORD`. How these are set depends on your deployment method:

<Tabs>

<Tab title="Helm">

Override the relevant entries under `extraEnvs` in your `values.yaml`, then upgrade the release:

```yaml
# values.yaml
extraEnvs:
- name: OPAMP_SERVER_URL
value: "<OPAMP_SERVER_URL>"
- name: CLICKHOUSE_ENDPOINT
value: "<HTTPS_ENDPOINT>"
- name: CLICKHOUSE_USER
value: "<CLICKHOUSE_USER>"
- name: CLICKHOUSE_PASSWORD
value: "<CLICKHOUSE_PASSWORD>"
```

```shell
helm upgrade clickstack-otel-collector open-telemetry/opentelemetry-collector -f values.yaml
```

</Tab>

<Tab title="Docker">

For example the all-in-one image:
All docker images which include the OpenTelemetry collector can be configured via environment variables. For example the all-in-one image:

```shell
export OPAMP_SERVER_URL=<OPAMP_SERVER_URL>
Expand All @@ -162,6 +332,10 @@ export CLICKHOUSE_PASSWORD=<CLICKHOUSE_PASSWORD>
docker run -e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} -e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} -p 8080:8080 -p 4317:4317 -p 4318:4318 clickhouse/clickstack-otel-collector:latest
```

</Tab>

</Tabs>

<ExtendingConfig/>

#### Docker Compose {#docker-compose-otel}
Expand Down Expand Up @@ -200,7 +374,36 @@ With Docker Compose, modify the collector configuration using the same environme

By default, the ClickStack OpenTelemetry collector isn't secured when deployed outside of the Open Source distributions and doesn't require authentication on its OTLP ports.

To secure ingestion, specify an authentication token when deploying the collector using the `OTLP_AUTH_TOKEN` environment variable. For example:
To secure ingestion, specify an authentication token when deploying the collector using the `OTLP_AUTH_TOKEN` environment variable. How this is set depends on your deployment method:

<Tabs>

<Tab title="Helm">

Add `OTLP_AUTH_TOKEN` to `extraEnvs` in your `values.yaml`, then upgrade the release:

```yaml
# values.yaml
extraEnvs:
- name: OTLP_AUTH_TOKEN
value: "a_very_secure_string"
- name: CLICKHOUSE_ENDPOINT
value: "<HTTPS_ENDPOINT>"
- name: CLICKHOUSE_USER
value: "<CLICKHOUSE_USER>"
- name: CLICKHOUSE_PASSWORD
value: "<CLICKHOUSE_PASSWORD>"
```

```shell
helm upgrade clickstack-otel-collector open-telemetry/opentelemetry-collector -f values.yaml
```

For production deployments, we recommend storing `OTLP_AUTH_TOKEN` and `CLICKHOUSE_PASSWORD` in a Kubernetes secret and referencing them via `extraEnvsFrom`.

</Tab>

<Tab title="Docker">

```sh
export CLICKHOUSE_ENDPOINT=<HTTPS_ENDPOINT>
Expand All @@ -218,6 +421,10 @@ docker run \
clickhouse/clickstack-otel-collector:latest
```

</Tab>

</Tabs>

We additionally recommend:

- Configuring the collector to communicate with ClickHouse over HTTPS.
Expand Down