Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,20 @@ expose.
ClickHouse user default database. It also calls out the `REMOTE` grant for
clustered preflight and the S3 IAM permissions needed by collect and delete.

- Documented the collection scope for an installation-wide cleanup. An object
layout with a `<installation>/<replica>/` level tempted an operator to
collect one replica's prefix, which misses the other replicas' orphans. The
guides now say to collect the parent prefix into one inventory table and
rely on the clustered comparison against every replica's
`system.remote_data_paths` — but only when that parent prefix holds one
`S3DISKNAME`'s objects, since the orphan check matches by disk name and
would misclassify another disk's objects under the same prefix as orphaned.

- Documented that static mode does not read the standard `AWS_ACCESS_KEY_ID`
and `AWS_SECRET_ACCESS_KEY` variables, with a mapping to the `S3GC_*`
Secret keys, including the optional session token in the copy-paste Secret
creation commands.

## [0.7.0] - 2026-08-24

A safety release, and its headline is uncomfortable: the dangerous
Expand Down
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,13 @@ Deleting an object is irreversible. `s3gc` preserves the following controls:
- The Kubernetes renderer requires a digest-pinned image and keeps credentials
outside the manifest.

For an installation-wide cleanup, collect the parent prefix that holds every
replica's objects and set the cluster name, so the comparison covers all
replicas. A single-replica prefix misses the other replicas' orphans. Only
collect a parent prefix that holds one `S3DISKNAME`'s objects — the orphan
check matches by disk name, so another disk's objects under the same prefix
would look orphaned.

Use a unique collection-table prefix for each bucket and prefix. Never commit
credentials, rendered manifests, target-cluster details, or run output.

Expand Down Expand Up @@ -82,6 +89,9 @@ Choose one S3 authentication mode with `S3GC_S3AUTH`:
| `aws` | boto3 chain; optional `S3GC_S3PROFILE` | AWS SSO or a named workstation profile |
| `iam` | MinIO workload-identity provider | EKS IRSA, EC2 instance profiles, or ECS task roles |

Static mode does not read `AWS_ACCESS_KEY_ID` or `AWS_SECRET_ACCESS_KEY`. Pass
those values as `S3GC_S3ACCESSKEY` and `S3GC_S3SECRETKEY`.

`S3GC_S3PROFILE` selects `aws`; the tool rejects contradictory settings. An
`aws` preview still needs `s3:ListBucket` for the configured bucket and prefix.

Expand Down
45 changes: 39 additions & 6 deletions deploy/kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,21 @@ Set the values in `s3gc.env`, then check these five items before rendering:
4. Choose a unique, database-qualified `COLLECTTABLEPREFIX`, such as `s3gc.s3gc_<run>_`. Keep it after a partial delete so the replacement Job can use the deletion checkpoints. A bare prefix uses the ClickHouse user's current database, which can differ from `default` and cause a grant failure.
5. Set `USEAGE_HOURS` to 24 or more. Raise it for slow merges or long mutations; production phases cannot lower it.

For an installation-wide cleanup, set `S3PATH` to the parent prefix that
contains every replica's objects, such as `clickhouse/<installation>/`, not a
single `<installation>/<replica>/` prefix — but only when that parent prefix
holds objects for the one `S3DISKNAME` being cleaned up. The orphan check
matches ClickHouse's `system.remote_data_paths` by `disk_name`, so an object
that belongs to a different disk stored under the same parent prefix would
never match and would look orphaned. When disks share a parent prefix, collect
a disk-scoped prefix instead, or give each disk its own S3 path layout.
Collect the chosen prefix once into one inventory table on the pinned host.
With `CLUSTERNAME` set, dry-run and delete compare
the inventory with `system.remote_data_paths` on all replicas, so an object
that any replica still references is never a candidate. A single-replica
prefix misses the other replicas' orphans. Collect per-replica prefixes only
when you intend to, and give each one its own `COLLECTTABLEPREFIX`.

Use an immutable multi-architecture image digest in `IMAGE`. CI prints the
exact value after publishing. Do not use an image tag.

Expand Down Expand Up @@ -106,13 +121,24 @@ its own S3 client and the Secret or workload identity instead.

For `S3AUTH=static`, create the Secret from a private, shell-style credential
file. Keep this file outside the repository, restrict it to its owner, and
never print or commit it. Its values must be exported as follows:
never print or commit it. s3gc does not read `AWS_ACCESS_KEY_ID` or
`AWS_SECRET_ACCESS_KEY` in static mode. Store those values under the s3gc
names instead:

| AWS variable | Secret key |
| --- | --- |
| `AWS_ACCESS_KEY_ID` | `S3GC_S3ACCESSKEY` |
| `AWS_SECRET_ACCESS_KEY` | `S3GC_S3SECRETKEY` |
| `AWS_SESSION_TOKEN` | `S3GC_S3SESSIONTOKEN` (optional) |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Carry the mapped session token into the Secret

For temporary AWS credentials, following this new mapping still produces an unusable Secret: the subsequent export example, kubectl create secret command, and unset command omit S3GC_S3SESSIONTOKEN. Even if the private file defines the mapped token, the literal-only command silently drops it, so static authentication fails. Add the optional token to the copy-paste workflow.

Useful? React with 👍 / 👎.


The file must export these values:

```bash
export S3GC_CHUSER=s3gc
export S3GC_CHPASS='<ClickHouse password>'
export S3GC_S3ACCESSKEY='<S3 access key>'
export S3GC_S3SECRETKEY='<S3 secret key>'
# export S3GC_S3SESSIONTOKEN='<S3 session token>' # only for temporary credentials
```

Create the Secret. This command creates it once and refuses to overwrite an
Expand All @@ -123,13 +149,20 @@ set -a
. /private/path/s3gc-secrets.env
set +a

kubectl -n <namespace> create secret generic <credentials-secret-name> \
--from-literal=S3GC_CHUSER="$S3GC_CHUSER" \
--from-literal=S3GC_CHPASS="$S3GC_CHPASS" \
--from-literal=S3GC_S3ACCESSKEY="$S3GC_S3ACCESSKEY" \
literals=(
--from-literal=S3GC_CHUSER="$S3GC_CHUSER"
--from-literal=S3GC_CHPASS="$S3GC_CHPASS"
--from-literal=S3GC_S3ACCESSKEY="$S3GC_S3ACCESSKEY"
--from-literal=S3GC_S3SECRETKEY="$S3GC_S3SECRETKEY"
)
if [ -n "${S3GC_S3SESSIONTOKEN:-}" ]; then
literals+=(--from-literal=S3GC_S3SESSIONTOKEN="$S3GC_S3SESSIONTOKEN")
fi

kubectl -n <namespace> create secret generic <credentials-secret-name> \
"${literals[@]}"

unset S3GC_CHUSER S3GC_CHPASS S3GC_S3ACCESSKEY S3GC_S3SECRETKEY
unset S3GC_CHUSER S3GC_CHPASS S3GC_S3ACCESSKEY S3GC_S3SECRETKEY S3GC_S3SESSIONTOKEN
```

The S3 principal needs `s3:ListBucket` on the scoped bucket/prefix for
Expand Down
Loading