From 44c7d0332322c52152e1e0c77317ebd5f696d8ca Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Wed, 16 Sep 2026 12:39:44 +0200 Subject: [PATCH] Improved documentation --- CHANGELOG.md | 8 +- README.md | 365 ++++++----------------- deploy/kubernetes/README.md | 255 +++++++--------- deploy/kubernetes/example.env | 12 +- skills/altinity-clickhouse-s3gc/SKILL.md | 15 +- 5 files changed, 230 insertions(+), 425 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b4b688c..c818dee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,13 @@ expose. ## [Unreleased] -Nothing yet. +### Documentation + +- Reworked the Kubernetes guidance into a phase-based production runbook and a + compact configuration reference. It now states the workload-identity + contract while preserving the Job's least-privilege ServiceAccount-token + setting, and the packaged agent skill uses the same approval and age-window + safeguards. ## [0.7.0] - 2026-08-24 diff --git a/README.md b/README.md index 960d736..11bc30a 100644 --- a/README.md +++ b/README.md @@ -1,46 +1,40 @@ # s3gc -`s3gc` finds and removes orphaned objects from ClickHouse S3 disks and other -S3-compatible storage. An object is a candidate only when it exists under the -configured bucket/prefix but is absent from ClickHouse -`system.remote_data_paths` for the configured disk. +`s3gc` finds orphaned objects on a ClickHouse S3 disk or compatible object +store. It collects an inventory into ClickHouse, compares it with +`system.remote_data_paths`, and reports objects that ClickHouse does not +reference. -## How it works +For customer and production work, use the Kubernetes runbook: -1. Collect object names, sizes, and timestamps into an auxiliary ClickHouse - table. -2. Anti-join that inventory with `system.remote_data_paths` (or all replicas of - a configured cluster). -3. Report candidates in dry-run mode, or delete them in batches and record - confirmed deletion checkpoints in the auxiliary table. - -The command-line script supports these actions directly. For Kubernetes, the -repository supplies a one-shot Job runner that separates collection, review, -and deletion. +```text +collect → dry-run → explicit human approval → delete → verify +``` -## Safety +Read the [Kubernetes Job runner](deploy/kubernetes/README.md) before you run a +cleanup. It is the authoritative production procedure. -Deleting an object is irreversible. Always run and review a dry-run before -deletion, and scope the configured bucket and prefix as narrowly as possible. +## Safety boundary -- Use a unique collection-table prefix for each cleanup. -- For clustered ClickHouse, use the cluster name and expected replica count. -- A failed delete Job does not automatically retry. Successfully deleted - batches remain checkpointed, so a replacement delete Job can resume safely. -- Never put credentials, customer manifests, or target-cluster details in Git. +Deleting an object is irreversible. `s3gc` preserves the following controls: -## Requirements +- A delete Job needs the `DELETE_ORPHANS` confirmation token. +- A clustered delete checks the configured cluster and expected replica count + before it calls S3. +- A failed delete Job does not retry automatically. Confirmed batches remain + checkpointed, so a replacement Job can resume with the same collection-table + prefix. +- `USEAGE` must be at least 24 hours outside development automation. This age + window protects a part that S3 has received before ClickHouse registers it. +- The Kubernetes renderer requires a digest-pinned image and keeps credentials + outside the manifest. -- Python 3.11 for local development; the container image also uses Python 3.11. -- Network access to ClickHouse and the target S3-compatible endpoint. -- A ClickHouse user that can read `system.remote_data_paths` and manage the - auxiliary table. -- S3 permissions appropriate to the action: list for collection, plus delete - for deletion. +Use a unique collection-table prefix for each bucket and prefix. Never commit +credentials, rendered manifests, target-cluster details, or run output. -## Quick start +## Local setup and read-only preview -Create a local environment and inspect the available options: +Install the pinned dependencies and inspect the available options: ```bash python3.11 -m venv .venv @@ -48,308 +42,127 @@ python3.11 -m venv .venv .venv/bin/python s3gc.py --help ``` -Configuration can be supplied as command-line arguments or `S3GC_*` -environment variables. Set the ClickHouse connection, S3 endpoint/bucket/prefix, -region, disk name, and either static S3 keys or workload identity. Keep secrets -in your approved secret manager or environment, not in command history. - -Run a dry-run first: - -```bash -.venv/bin/python s3gc.py --verbose --dry-run -``` - -For a production or customer cleanup, use the Kubernetes procedure below rather -than a one-line delete command. - -## AI-agent skill - -This repository includes a reusable operational skill at -[`skills/altinity-clickhouse-s3gc`](skills/altinity-clickhouse-s3gc/). It guides -agents through the same guarded production workflow documented here, including -per-replica ClickHouse routing, reviewed dry-run results, explicit human delete -approval, and final verification. - -Install it by symlinking or copying the directory into your agent's skills -directory. For example, for Codex: - -```bash -mkdir -p ~/.codex/skills -ln -s /path/to/s3gc/skills/altinity-clickhouse-s3gc \ - ~/.codex/skills/altinity-clickhouse-s3gc -``` - -For Claude Code, substitute `~/.claude/skills`. Invoke it with -`$altinity-clickhouse-s3gc` or ask the agent to plan a reviewed `s3gc` cleanup. -The skill does not permit an agent to supply the deletion confirmation; an -authorized human must approve the reviewed dry-run result. - -## Direct script examples - -The following is a non-secret target configuration. Replace every -`` value and do not commit this environment to Git: +Set configuration with flags or `S3GC_*` environment variables. A minimum +non-secret setup looks like this: ```bash -export S3GC_CHHOST='' +export S3GC_CHHOST='' export S3GC_CHPORT=8123 export S3GC_CHUSER='' export S3GC_S3IP='s3.eu-central-1.amazonaws.com' export S3GC_S3PORT=443 export S3GC_S3BUCKET='' -export S3GC_S3PATH='/' +export S3GC_S3PATH='/' export S3GC_S3REGION='eu-central-1' export S3GC_S3SECURE_FLAG=true export S3GC_S3DISKNAME=s3 export S3GC_CLUSTERNAME='' export S3GC_EXPECTED_REPLICAS=2 export S3GC_COLLECTTABLEPREFIX='s3gc_example_' -export S3GC_AGE=24 -export S3GC_USEAGE=24 # minimum 24; raising is fine, lowering is refused +export S3GC_USEAGE=24 ``` -### S3 authentication modes - -Select one with `S3GC_S3AUTH` (or `--s3auth`): - -| Mode | Credentials | Needs boto3 | Typical use | -|---|---|---|---| -| `static` (default) | `S3GC_S3ACCESSKEY` + `S3GC_S3SECRETKEY`, optionally `S3GC_S3SESSIONTOKEN` | no | long-lived keys, or explicit temporary credentials | -| `aws` | boto3 credential chain, optionally `S3GC_S3PROFILE` | **yes** | AWS SSO / named profiles on a workstation | -| `iam` | MinIO workload identity provider | no | EKS IRSA, EC2 instance profile, ECS task role | - -`S3GC_S3PROFILE` implies `aws`. Contradictory combinations are rejected rather -than silently resolved. - -#### Static credentials - -Inject static credentials from a secret manager or interactive shell rather -than saving them in a file: - -```bash -export S3GC_CHPASS='' -export S3GC_S3ACCESSKEY='' -export S3GC_S3SECRETKEY='' -``` - -Every `S3GC_*` boolean accepts `true/false`, `yes/no`, `on/off`, `1/0`, or an -empty value for false. Unset also means false. - -#### AWS SSO or a named profile - -Authenticate with the AWS CLI first, then let `s3gc` resolve temporary -credentials through the boto3 chain: +Inject passwords and keys from a secret manager or your shell. Do not save them +in a file or command history. Then run a preview: ```bash -aws sso login --profile my-sso-profile - -export S3GC_S3AUTH=aws -export S3GC_S3PROFILE=my-sso-profile -export S3GC_S3IP=s3.amazonaws.com -export S3GC_S3PORT=443 -export S3GC_S3REGION=us-east-1 -export S3GC_S3SECURE_FLAG=true -.venv/bin/python ./s3gc.py --verbose --dry-run -``` - -`S3GC_S3ACCESSKEY` and `S3GC_S3SECRETKEY` are unused in `aws` mode, and setting -them is an error rather than a silent override. The resolved credentials must -allow `s3:ListBucket` on the bucket for that prefix **even for `--dry-run`** — -collection lists objects. On failure `s3gc` prints the required permission and -the commands to verify it: - -```bash -aws sts get-caller-identity --profile my-sso-profile -aws s3api list-objects-v2 --bucket --prefix --max-keys 1 --profile my-sso-profile -``` - -#### Workload identity (EKS/IRSA, EC2, ECS) - -```bash -export S3GC_CHPASS='' -export S3GC_S3AUTH=iam +.venv/bin/python s3gc.py --verbose --dry-run ``` -`iam` uses MinIO's AWS IAM credential provider and refreshes temporary -credentials from EKS IRSA/workload identity, an EC2 instance profile, or an ECS -task role. Prefer it over `aws` inside Kubernetes: it keeps `boto3` out of the -request path and avoids credentials expiring during a long collect or delete. +Use direct deletion only for controlled development work. The Kubernetes +runbook separates collection, review, approval, deletion, and verification. -It does not read AWS CLI profiles, `aws sso login` state, `~/.aws/config`, or -`AWS_PROFILE`; use `aws` mode for that workstation workflow. +## Authentication and object stores -#### GCS and other stores without batch delete +Choose one S3 authentication mode with `S3GC_S3AUTH`: -GCS has no batch `DeleteObjects`. `s3gc` detects a `storage.googleapis.com` -endpoint and falls back to per-object deletion automatically, warning that it is -slower; `--use-remove-objects=false` sets it explicitly. Note the disk name is -usually `gcs`, not `s3`, and GCS needs **HMAC/interop** keys: +| Mode | Credentials | Use it for | +| --- | --- | --- | +| `static` | `S3GC_S3ACCESSKEY` and `S3GC_S3SECRETKEY`; optional session token | explicit credentials from a secret manager | +| `aws` | boto3 chain; optional `S3GC_S3PROFILE` | AWS SSO or a named workstation profile | +| `iam` | MinIO workload-identity provider | EKS IRSA, EC2 instance profiles, or ECS task roles | -```bash -export S3GC_S3ACCESSKEY='GOOG1...' -export S3GC_S3SECRETKEY='...' -export S3GC_S3IP=storage.googleapis.com -export S3GC_S3PORT=443 -export S3GC_S3SECURE_FLAG=true -export S3GC_S3DISKNAME=gcs -.venv/bin/python ./s3gc.py --verbose --use-remove-objects=false -``` +`S3GC_S3PROFILE` selects `aws`; the tool rejects contradictory settings. An +`aws` preview still needs `s3:ListBucket` for the configured bucket and prefix. -### Safe split workflow +GCS uses HMAC interoperability keys and usually names the object disk `gcs`. +It does not support S3 batch deletion, so `s3gc` falls back to slower +per-object deletion. Set `S3GC_S3DISKNAME=gcs`; do not use a `*_cache` disk as +the reference scope. -Collection makes an auxiliary table; the second command reads it and reports -candidates without deleting objects: +## Kubernetes deployment -```bash -.venv/bin/python s3gc.py --collectonly --keepdata -.venv/bin/python s3gc.py --usecollected --dry-run -``` +The Kubernetes runner is in [deploy/kubernetes](deploy/kubernetes/). It uses a +one-shot Job, a non-secret environment file, and a Kubernetes Secret or +workload identity for credentials. -### Collect has no resume — shard large buckets +Start with these links: -A crashed or interrupted `--collectonly` restarts its listing from the -beginning; there is no checkpoint. On a multi-million-object bucket that can -cost hours, and long runs are exactly where a rotating password or a dropped -connection tends to strike. +1. [Run the production workflow](deploy/kubernetes/README.md#runbook) +2. [Configure the Job](deploy/kubernetes/README.md#configuration-reference) +3. [Copy the non-secret example](deploy/kubernetes/example.env) -Shard the listing by prefix and re-run only the shards that failed. This is safe -to repeat: the auxiliary table is a `ReplacingMergeTree` keyed on `objpath`, so -re-listing a shard is idempotent. - -```bash -# buckets laid out as /<3-char hash>/ -for shard in 0 1 2 3 4 5 6 7 8 9 a b c d e f g h i j k l m n o p q r s t u v w x y z; do - S3GC_S3PATH="/${shard}" ./s3gc.py --collectonly --keepdata || \ - echo "shard ${shard} FAILED — re-run just this one" -done -``` - -The same variables can be passed as flags (for example, -`--ch-host` or `--s3-bucket`). Run `.venv/bin/python s3gc.py --help` for the -complete flag and environment-variable reference. Avoid direct deletion for -customer or production work; use the reviewed Kubernetes workflow instead. - -## Container image - -Released images are **public** at `ghcr.io/altinity/s3gc`, so Kubernetes needs no -`imagePullSecret`. Always reference them **by digest**, never by tag — tags get -re-pushed and stop reproducing what you tested: +Released images are public at `ghcr.io/altinity/s3gc`. Always use a digest: ```bash docker pull ghcr.io/altinity/s3gc@sha256: ``` -CI prints the exact `IMAGE=` line in its job summary; paste that into your -`.env`. `render.py` refuses anything not digest-pinned. - -Build locally for a quick check: - -```bash -docker build -f docker/Dockerfile -t s3gc:local . -``` - -To publish by hand, **both architectures are mandatory** — ClickHouse node pools -are frequently arm64, and an amd64-only image will not schedule there: - -```bash -docker buildx build --platform linux/amd64,linux/arm64 \ - -f docker/Dockerfile -t ghcr.io/altinity/s3gc: --push . -docker buildx imagetools inspect ghcr.io/altinity/s3gc: # expect amd64 AND arm64 -``` - -> Buildx builder containers cache `/etc/resolv.conf` at creation time. A builder -> left running across a network or VPN change fails with -> `lookup registry-1.docker.io: i/o timeout` while the host resolves fine. -> Recreate the builder, or create one with `--driver-opt network=host`. - -The CI workflow runs tests on every pull request and publishes from pushes to -`master` and version tags, authenticating to GHCR with the automatic -`GITHUB_TOKEN`. - -## Kubernetes - -The Kubernetes runner lives in [`deploy/kubernetes/`](deploy/kubernetes/). It -uses a digest-pinned image and external Kubernetes Secrets; it does not create -or store credentials in the repository. - -For customer and production work, follow: - -```text -collect → dry-run → approved delete → verify -``` - -The concise operator procedure, Secret requirements, and renderer configuration -are in [deploy/kubernetes/README.md](deploy/kubernetes/README.md). A guarded -`dev-automation` phase is available only for non-production testing; it runs -collect, dry-run, and delete in one Job and still requires an explicit delete -confirmation. +CI publishes the exact `IMAGE=` value in its job summary. The renderer rejects +mutable image tags. ## Durable run history -Each run appends structured events to `_log` in -ClickHouse, beside the auxiliary table: phase start, collect progress, -per-sample start, one row per confirmed delete batch, and a closing total, each -carrying the scope the run was pointed at. The table is never truncated. +By default, each run writes structured events to +`_log` in ClickHouse. The table records phase +starts, collect progress, deletion checkpoints, totals, warnings, and errors. +It outlives the Job and rotated pod logs. -This exists because pod logs are ephemeral — the kubelet rotates them and a -deleted Job takes them with it — so a completed cleanup would otherwise leave no -evidence of what it removed. ClickHouse is the sink because the connection and -grants already exist; writing the log into the bucket would make the *next* -collect see it as an orphan and delete it. +Query one run on the same replica-pinned ClickHouse host: ```sql SELECT event_time, phase, event, objects, bytes, message -FROM _log -WHERE run_id = '' +FROM ._log +WHERE run_id = '' ORDER BY event_time; ``` -`--runid` labels the run (Kubernetes Jobs use the Job name automatically); -`--runlog false` turns the table off. It needs the same `CREATE TABLE` grant as -the auxiliary table, and a missing grant degrades to stdout only rather than -failing the run. +Set `RUNLOG=false` only when stdout is an adequate record. A run-log failure +falls back to stdout and does not stop a cleanup. -## Testing +## AI-agent skill -Run all isolated unit tests: +[`skills/altinity-clickhouse-s3gc`](skills/altinity-clickhouse-s3gc/) gives an +agent the same safety boundaries as the Kubernetes runbook. Install it by +symlinking or copying the directory into the agent's skills directory. For +Codex: ```bash -.venv/bin/python -m pytest -v +mkdir -p ~/.codex/skills +ln -s /path/to/s3gc/skills/altinity-clickhouse-s3gc \ + ~/.codex/skills/altinity-clickhouse-s3gc ``` -Run only the development-automation tests: +The skill never supplies the delete confirmation. An authorized human must +approve the reviewed dry-run result. + +## Development checks + +Run offline tests: ```bash -.venv/bin/python -m pytest -v -k dev_automation +.venv/bin/python -m pytest -v -m "not dev_cluster" ``` -Validate that the example Kubernetes configuration renders without creating a -cluster resource: +Render and validate the example Job without contacting a cluster: ```bash -python3 deploy/kubernetes/render.py deploy/kubernetes/example.env > /tmp/s3gc-job.yaml +.venv/bin/python deploy/kubernetes/render.py deploy/kubernetes/example.env > /tmp/s3gc-job.yaml docker run --rm --entrypoint /kubeconform -v /tmp:/tmp:ro \ ghcr.io/yannh/kubeconform@sha256:85dbef6b4b312b99133decc9c6fc9495e9fc5f92293d4ff3b7e1b30f5611823c \ -strict -summary /tmp/s3gc-job.yaml ``` -The unit suite does not contact ClickHouse, S3, or Kubernetes. The reserved -`dev_cluster` pytest marker is excluded from CI; any future tests using it must -be selected explicitly with `.venv/bin/python -m pytest -m dev_cluster` after -reviewing their fixture scope. The collect/dry-run/delete exercise is manual -because it can intentionally delete development objects. - -## Repository layout - -- `s3gc.py` — collection, anti-join, and deletion logic. -- `docker/` — Python 3.11 container image and Kubernetes entrypoint. -- `deploy/kubernetes/` — plain Job template, renderer, example configuration, - and operator guide. -- `tests/` — pytest safety, renderer, and entrypoint tests. - -## History and roadmap - -See [`CHANGELOG.md`](CHANGELOG.md) for the full history, including why each -change was made and the evidence behind defects found in production use. - -Planned: concurrency and asynchronous collection/deletion; a `--collectafter` -checkpoint so an interrupted collect can resume instead of re-listing. +See [CLAUDE.md](CLAUDE.md) for contributor requirements, +[CHANGELOG.md](CHANGELOG.md) for operational history, and [TODO.md](TODO.md) +for deferred work. diff --git a/deploy/kubernetes/README.md b/deploy/kubernetes/README.md index 0c5c697..4901a27 100644 --- a/deploy/kubernetes/README.md +++ b/deploy/kubernetes/README.md @@ -1,83 +1,24 @@ # Kubernetes Job runner -Run `s3gc` as a one-shot Job in the ClickHouse namespace. For customer and -production work, always run: +Run `s3gc` as a one-shot Job in the ClickHouse namespace. This guide is the +production procedure. -```text -collect → dry-run → approved delete → verify -``` +## Prerequisite -The Job reaches ClickHouse through an in-cluster Service; no laptop tunnel is -needed. - -> **`CHHOST` must be a PER-REPLICA Service, never the load-balanced one.** -> The auxiliary table is a node-local `ReplacingMergeTree`, not a Replicated -> table. A load-balanced Service round-robins, so `collect` can write the table -> on one replica while `dry-run`/`delete` land on another and find nothing. -> Use `chi---0-0` (replica 0), not `clickhouse-`, and use -> the **same** host for every phase of a cleanup. - -## Before the first Job - -- Use an immutable, multi-architecture image digest: - `ghcr.io/altinity/s3gc@sha256:`. CI prints the exact `IMAGE=` line in - its job summary. Pin by digest, never by tag — tags get re-pushed. -- The image must be multi-arch. ClickHouse node pools are often arm64 (one - customer cluster is 5x arm64 + 1x amd64), and an amd64-only image simply will - not schedule there. -- Create or reuse a namespace-local ServiceAccount. -- Create a runtime Secret named by `CREDENTIALS_SECRET`: - - `S3AUTH=static`: `S3GC_CHUSER`, `S3GC_CHPASS`, `S3GC_S3ACCESSKEY`, - `S3GC_S3SECRETKEY`, plus `S3GC_S3SESSIONTOKEN` for temporary credentials. - - `S3AUTH=iam`: only `S3GC_CHUSER` and `S3GC_CHPASS`, with an - identity-enabled ServiceAccount. Note the template sets - `automountServiceAccountToken: false`, so `iam` also needs a ServiceAccount - that actually projects a token. -- Confirm the ClickHouse Service name, cluster macro, expected replica count, - S3 bucket/prefix, and disk name. Use a unique `COLLECTTABLEPREFIX` per - bucket/prefix cleanup. - -### Choosing `S3AUTH` - -| `S3AUTH` | Credentials | Needs boto3 | Use when | -|---|---|---|---| -| `iam` (default here) | MinIO workload identity — IRSA, IMDS, ECS task role | no | the normal Kubernetes case | -| `static` | `S3GC_S3ACCESSKEY`/`S3GC_S3SECRETKEY` (+ optional `S3GC_S3SESSIONTOKEN`) from the Secret | no | no workload identity available | -| `aws` | boto3 chain, optionally `S3PROFILE` | **yes** | rarely in-cluster; this is a workstation SSO path | - -`S3PROFILE` requires `S3AUTH=aws` and the renderer rejects other combinations. - -Prefer `iam`: it hands MinIO the credential provider, so temporary credentials -refresh during a long collect or delete instead of expiring mid-run. - -### Minimum ClickHouse grants - -Create or provision a dedicated ClickHouse user for `s3gc`, then grant it: +Use Python 3.11 from the repository root. Create the project virtual +environment before you render a Job: -```sql -GRANT SELECT ON system.* TO s3gc; -- remote_data_paths, one, disks, tables -GRANT SELECT, INSERT, CREATE TABLE ON .* TO s3gc; -- auxiliary + run-log tables +```bash +python3.11 -m venv .venv +.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt ``` -### Values that vary per cluster, and bite when wrong +`render.py` uses only the Python standard library. The runbook still uses +`.venv/bin/python` so every command runs with the supported Python version. -- **`S3PATH` may legitimately be empty** — some buckets keep blobs at the root. - A wrong prefix silently lists nothing and reports a clean bucket. -- **The disk is not always called `s3`** — GCS-backed clusters commonly use - `gcs`. `S3DISKNAME` sets both the anti-join scope and the aux table name, so - the wrong value makes *every* blob look orphaned. -- **A `*_cache` disk is a filesystem cache over the same blobs**, not a second - reference scope; scope the anti-join to the underlying object disk. -- **`SAMPLES` must match the value used at collect time** — the aux table is - `PARTITION BY CRC32(objpath) % SAMPLES`, and a mismatch loses partition - pruning (measured: ~2 min vs ~26 min per sample). s3gc now warns on mismatch. -- **GCS has no batch delete** — s3gc detects a `storage.googleapis.com` endpoint - and falls back to one request per object, which is markedly slower. +## Runbook -Never commit credentials, rendered customer manifests, or customer `.env` -files to this repository. - -## Local run directory +### 1. Prepare a private run directory Keep generated files outside this repository: @@ -87,6 +28,9 @@ mkdir -p "$S3GC_RUN_DIR" cp deploy/kubernetes/example.env "$S3GC_RUN_DIR/s3gc.env" ``` +The directory will contain the non-secret configuration and one rendered +manifest for each phase: + ```text $S3GC_RUN_DIR/ ├── s3gc.env @@ -96,68 +40,109 @@ $S3GC_RUN_DIR/ └── verify.yaml ``` -Fill `s3gc.env` from `example.env`. For production, start with -`SAMPLES=4`, `USEAGE_HOURS=24`, `ORDER_BY_OBJPATH=false`, and a 12-hour -deadline. +Do not commit this directory. It identifies a target environment even when it +contains no credentials. + +### 2. Confirm the cleanup scope + +Set the values in `s3gc.env`, then check these five items before rendering: + +1. Set `CHHOST` to one per-replica ClickHouse Service and use the same host for every phase. The inventory table is node-local; a load-balanced Service can send a later phase to a replica without that table. +2. Set the exact bucket, prefix, and underlying object-disk name. `S3PATH` may be empty. GCS commonly uses `gcs`; never use a `*_cache` disk. +3. Set the actual `CLUSTERNAME` and `EXPECTED_REPLICAS` for clustered cleanup. Delete fails closed if this preflight does not match. +4. Choose a unique `COLLECTTABLEPREFIX`. Keep it after a partial delete so the replacement Job can use the deletion checkpoints. +5. Set `USEAGE_HOURS` to 24 or more. Raise it for slow merges or long mutations; production phases cannot lower it. + +Use an immutable multi-architecture image digest in `IMAGE`. CI prints the +exact value after publishing. Do not use an image tag. + +Collection has no resume checkpoint. For a large bucket, collect separate +prefix shards and rerun only a failed shard. Re-listing a shard is safe because +the auxiliary table replaces rows by object path. + +### 3. Provide credentials and identity + +Create or reuse the Kubernetes Secret named by `CREDENTIALS_SECRET`: + +| `S3AUTH` | Secret values | +| --- | --- | +| `static` | `S3GC_CHUSER`, `S3GC_CHPASS`, `S3GC_S3ACCESSKEY`, `S3GC_S3SECRETKEY`, and an optional `S3GC_S3SESSIONTOKEN` | +| `iam` | `S3GC_CHUSER` and `S3GC_CHPASS` | +| `aws` | `S3GC_CHUSER`, `S3GC_CHPASS`, plus a credential source available to boto3 | + +Set `SERVICE_ACCOUNT` to an existing namespace-local ServiceAccount that your +cluster configures for the chosen workload-identity mode. The Job keeps +`automountServiceAccountToken: false` because it does not call the Kubernetes +API. For `iam`, the platform must inject or otherwise provide the workload +identity credentials. Existing ClickHouse or ClickHouse-backup ServiceAccounts +are suitable when they meet that cluster policy. + +Create or provision a dedicated ClickHouse user with the required grants: -> **`USEAGE_HOURS` has a hard floor of 24 and the renderer enforces it.** -> ClickHouse uploads a part's blobs to S3 and registers them in -> `system.remote_data_paths` a moment later. In that window a live blob looks -> orphaned, and nothing re-checks it before the delete — so this window is the -> only thing protecting a part that is still being written. Raise it if a -> cluster has slow merges or long mutations; you cannot lower it. The one -> exception is `PHASE=dev-automation`, which seeds and deletes its own fixtures -> and is already documented as non-production. +```sql +GRANT SELECT ON system.* TO s3gc; +GRANT SELECT, INSERT, CREATE TABLE ON .* TO s3gc; +``` -Before the first *full* delete against a newly published image or a cluster you -have not deleted from before, run one bounded delete with `USETOTAL` set to a -few thousand. It exercises the whole path — anti-join, S3 deletion, tombstone -write-back — in minutes, and a failure costs you that instead of a multi-hour -run that dies partway with objects already removed. Clear `USETOTAL` for the -real run. +The second grant covers the auxiliary inventory and durable run-log tables. -## Run each phase +### 4. Run each phase -For each phase, update only `PHASE`, `JOB_NAME`, and (for delete) -`DELETE_CONFIRMATION` in `s3gc.env`, then render and apply: +For each phase, edit only the values shown below, render the Job, server-side +validate it, apply it, and follow its logs: ```bash -python3 deploy/kubernetes/render.py "$S3GC_RUN_DIR/s3gc.env" \ +.venv/bin/python deploy/kubernetes/render.py "$S3GC_RUN_DIR/s3gc.env" \ > "$S3GC_RUN_DIR/.yaml" kubectl apply --dry-run=server -f "$S3GC_RUN_DIR/.yaml" kubectl apply -f "$S3GC_RUN_DIR/.yaml" kubectl -n logs -f job/ ``` -| Phase | Required values | Result | -|---|---|---| -| `collect` | `PHASE=collect` | Lists S3 objects into the auxiliary ClickHouse table. No deletion. | -| `dry-run` | `PHASE=dry-run` | Reports candidates and total size. Review this result. | -| `delete` | `PHASE=delete`, `DELETE_CONFIRMATION=DELETE_ORPHANS` | Checks cluster/replicas, deletes candidates, and checkpoints confirmed progress. | -| `verify` | `PHASE=dry-run` | Must report zero candidates. | +| Phase | Change in `s3gc.env` | Required outcome | +| --- | --- | --- | +| Collect | Set `PHASE=collect` and a new `JOB_NAME`. | Inventory S3 objects. No deletion occurs. | +| Review | Set `PHASE=dry-run` and a new `JOB_NAME`. | Review the candidate count, bytes, bucket, and prefix. | +| Delete | After written human approval, set `PHASE=delete`, a new `JOB_NAME`, and `DELETE_CONFIRMATION=DELETE_ORPHANS`. | Delete and checkpoint confirmed batches. | +| Verify | Set `PHASE=dry-run`, a new `JOB_NAME`, and clear `DELETE_CONFIRMATION`. | Report zero candidates. | -### Development automation only +Do not run delete from an unreviewed dry-run. Before the first full delete for +a new image or cluster, set `USETOTAL` to a few thousand objects for one +bounded delete. Clear it before the full run. -`PHASE=dev-automation` runs `collect → dry-run → delete` in one Job. It always -starts with a fresh auxiliary table and requires -`DELETE_CONFIRMATION=DELETE_ORPHANS`, `CLUSTERNAME`, and `EXPECTED_REPLICAS`. -Any failed stage stops the Job and later stages do not run; successful delete -batches remain checkpointed. Do not use this phase for customer or production -work because it removes the manual dry-run approval gate. +### 5. Recover or verify -## Durable run history +If a delete Job fails, create a replacement delete Job with a new `JOB_NAME` +and the same `COLLECTTABLEPREFIX`. Do not re-collect unless the scope changed. +`backoffLimit: 0` prevents automatic retries. + +After a successful delete, run the verification dry-run. Do not declare the +cleanup complete until it reports zero candidates. + +## Configuration reference + +`render.py` reads a non-secret `KEY=VALUE` file. It rejects missing required +values, unpinned images, invalid phases, invalid booleans, and unsafe age +windows. Start with [example.env](example.env). + +| Group | Required values | Notes | +| --- | --- | --- | +| Job | `JOB_NAME`, `NAMESPACE`, `IMAGE`, `IMAGE_PULL_SECRET`, `SERVICE_ACCOUNT`, `CREDENTIALS_SECRET` | `JOB_NAME` is a DNS label and becomes the durable run ID. Leave `IMAGE_PULL_SECRET` empty for the public image. | +| ClickHouse | `CHHOST`, `CHPORT`, `CLUSTERNAME`, `EXPECTED_REPLICAS`, `COLLECTTABLEPREFIX` | Pin `CHHOST` to one replica for the entire cleanup. | +| Object store | `S3IP`, `S3PORT`, `S3BUCKET`, `S3PATH`, `S3REGION`, `S3SECURE_FLAG`, `S3DISKNAME`, `S3AUTH`, `S3PROFILE` | `S3PROFILE` requires `S3AUTH=aws`. | +| Limits | `SAMPLES`, `DELETE_BATCH_SIZE`, `USEAGE_HOURS`, `ACTIVE_DEADLINE_SECONDS`, `TTL_SECONDS_AFTER_FINISHED`, `MEMORY_REQUEST`, `MEMORY_LIMIT` | Keep `SAMPLES` unchanged after collect for partition pruning. | +| Behavior | `PHASE`, `DELETE_CONFIRMATION`, `ORDER_BY_OBJPATH`, `VERBOSE` | The renderer requires the confirmation token for delete and development automation. | + +`USETOTAL` is optional and limits one use phase. `RUNLOG` defaults to `true`; +set it to `false` only when stdout is an adequate operational record. -Pod logs are **not** a record. The kubelet rotates container output, so -`kubectl logs` cannot return the start of a long run, and -`ttlSecondsAfterFinished` deletes the Job and its pods along with everything -they printed. `kubectl logs -f` is for watching, not for evidence. +## Durable run history -So each run also appends to `_log` in -ClickHouse, beside the auxiliary table. That table is **never truncated**, and -`S3GC_RUNID` is set to the Job name, so a row traces back to the Job that wrote -it long after the pod is gone. +Pod logs are for monitoring, not durable evidence. The kubelet rotates them, +and the Job TTL removes the pod. By default, `s3gc` writes run events to +`_log` in ClickHouse. -What a delete actually did, on the same replica-pinned `CHHOST`: +Use the same replica-pinned host to inspect a run: ```sql SELECT event_time, phase, event, objects, bytes, message @@ -166,34 +151,22 @@ WHERE run_id = '' ORDER BY event_time; ``` -How far a *failed* delete got before it died, which is what tells you whether to -start a replacement delete Job: - -```sql -SELECT max(objects) AS deleted, max(bytes) AS reclaimed -FROM ._log -WHERE run_id = '' AND event = 'checkpoint'; -``` +For a failed delete, inspect the latest `checkpoint` event before starting its +replacement Job. A run-log write failure falls back to stdout and does not stop +the cleanup. -Every phase of a cleanup, newest first: +## Development automation only -```sql -SELECT run_id, min(event_time) AS started, max(event_time) AS ended, - anyIf(message, event = 'error') AS error -FROM ._log -GROUP BY run_id ORDER BY started DESC; -``` +`PHASE=dev-automation` runs collect, dry-run, and delete in one Job. It starts +with a fresh inventory and requires `DELETE_CONFIRMATION=DELETE_ORPHANS`, +`CLUSTERNAME`, and `EXPECTED_REPLICAS`. -Set `RUNLOG=false` to opt out. A missing `CREATE TABLE` grant degrades to -stdout only with one warning rather than failing the run, and a run-log write -that fails mid-delete disables the log instead of stopping the deletion. +Use it only for isolated development fixtures. It bypasses the manual review +and approval gate, and it alone may use an age window below 24 hours. -## Safety +## Preserved safeguards -- Delete checks the local cluster macro and expected replica count before S3 - calls. -- Confirmed deletions are tombstoned in the auxiliary table. If a delete Job - fails, create a new delete Job name with the same table prefix; do **not** - re-collect. -- No Job retries automatically (`backoffLimit: 0`). -- Do not run delete until the customer explicitly approves the dry-run result. +- The manifest runs as a non-root user with a read-only root filesystem and no Linux capabilities. +- Credentials come from `CREDENTIALS_SECRET` or workload identity, never the image or environment file. +- The renderer requires a digest-pinned image and validates delete confirmation before the Job reaches the cluster. +- Delete requires the cluster/replica preflight, records checkpoints, and never retries automatically. diff --git a/deploy/kubernetes/example.env b/deploy/kubernetes/example.env index 373e95b..f7d6b0f 100644 --- a/deploy/kubernetes/example.env +++ b/deploy/kubernetes/example.env @@ -1,5 +1,8 @@ -# Copy this file outside the repository, fill non-secret target values, and render it. -# Credentials must be supplied separately by the named Kubernetes Secret. +# Copy this file to a private run directory. Fill only non-secret target values. +# Supply credentials through the named Kubernetes Secret or workload identity. +# See deploy/kubernetes/README.md for the phase-by-phase production runbook. +# +# Job identity and lifecycle JOB_NAME=s3gc-example-dry-run NAMESPACE=clickhouse # Always digest-pinned; CI prints the exact line in its job summary. @@ -14,12 +17,14 @@ DELETE_CONFIRMATION= CREDENTIALS_SECRET=s3gc-credentials SERVICE_ACCOUNT=s3gc +# ClickHouse scope. CHHOST must be one per-replica Service for every phase. CHHOST=clickhouse.example.svc.cluster.local CHPORT=8123 CLUSTERNAME=example-cluster EXPECTED_REPLICAS=2 COLLECTTABLEPREFIX=s3gc_example_ +# Object-store scope. S3PATH may be empty for a bucket-root layout. S3IP=s3.example.com S3PORT=443 S3BUCKET=example-bucket @@ -33,6 +38,7 @@ S3AUTH=iam # Only for S3AUTH=aws. The session token, if any, belongs in the Secret. S3PROFILE= +# Processing limits. Keep SAMPLES unchanged after collection. SAMPLES=4 DELETE_BATCH_SIZE=1000 # Minimum 24, and the renderer refuses less for every phase except @@ -47,6 +53,8 @@ USEAGE_HOURS=24 USETOTAL= # Leave false for Kubernetes Jobs; global ordering is unnecessary for deletion. ORDER_BY_OBJPATH=false + +# Pod limits and observability. ACTIVE_DEADLINE_SECONDS=14400 TTL_SECONDS_AFTER_FINISHED=604800 MEMORY_REQUEST=1Gi diff --git a/skills/altinity-clickhouse-s3gc/SKILL.md b/skills/altinity-clickhouse-s3gc/SKILL.md index 6889982..eff6650 100644 --- a/skills/altinity-clickhouse-s3gc/SKILL.md +++ b/skills/altinity-clickhouse-s3gc/SKILL.md @@ -12,7 +12,7 @@ and removing approved orphaned objects. It is not a generic S3 cleanup procedure ## Non-negotiable rules - Deletion is irreversible. For customer and production work, use the Kubernetes Job workflow: - **collect → dry-run → explicit customer approval → delete → verify**. + **collect → dry-run → explicit human approval → delete → verify**. - Scope the bucket and prefix as narrowly as possible. A wrong prefix can produce a misleading clean result or expand deletion scope. - Use the underlying object disk (for example `s3` or `gcs`) for `S3DISKNAME`, never its @@ -25,6 +25,8 @@ and removing approved orphaned objects. It is not a generic S3 cleanup procedure - Use a unique `COLLECTTABLEPREFIX` for each bucket/prefix cleanup. Preserve it after a partial delete: confirmed deletions are checkpointed, and a replacement delete Job resumes from that state without re-collecting. +- Keep `USEAGE_HOURS` at 24 or higher for production phases. This protects objects uploaded to S3 + before ClickHouse has registered their paths. - An agent must not supply `DELETE_ORPHANS`, create a delete manifest with that token, or bypass the reviewed dry-run gate. A human must review the immediate dry-run output and explicitly authorize deletion. @@ -44,13 +46,15 @@ for an in-cluster AWS cleanup; use static credentials only from a runtime Secret Read [the Kubernetes operator guide](../../deploy/kubernetes/README.md) and start from [`deploy/kubernetes/example.env`](../../deploy/kubernetes/example.env) copied to a private run directory outside the repository. Use an immutable multi-architecture image digest, not a tag. +Select an existing namespace-local ServiceAccount configured for the cluster's intended workload +identity mode; the runner does not need Kubernetes API credentials. ### 2. Collect and review Render and run a `collect` Job, then a `dry-run` Job against the same `CHHOST` and collection -table. Collect does not delete; dry-run reports the candidate count and size. Capture the rendered -configuration location and the non-secret Job/log evidence in the active support record without -copying credentials or customer artifacts into Git. +table. Collect does not delete; dry-run reports the candidate count and size. Review the exact +bucket and prefix too. Capture the rendered configuration location and the non-secret Job/log +evidence in the active support record without copying credentials or customer artifacts into Git. Treat an empty or missing collected inventory as a failure to investigate, not proof that the bucket is clean. Verify the prefix, disk name, and replica-pinned service first. For large buckets, @@ -65,7 +69,8 @@ authorized customer/operator. Do not proceed without explicit approval. After the human supplies the delete confirmation in the private runtime configuration, render a new delete Job using the same collection-table prefix. Monitor it to completion. A failed delete Job has no automatic retry; start a replacement delete Job with the same prefix so the confirmed -deletion checkpoints are preserved. Do not re-collect unless the scope itself changed. +deletion checkpoints are preserved. Do not re-collect unless the scope itself changed. Keep the +durable run log enabled unless stdout is the approved operational record. ### 4. Verify and hand off