Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 16 additions & 8 deletions deploy/personal-node/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Run a personal TinyCloud node

This ordinary-server profile runs a single node with local SQLite, local blocks, and Caddy HTTPS. It tracks the version TinyCloud itself runs in production: **1.20.6**, pinned by immutable image digest in [`compose.yaml`](compose.yaml). Caddy **2.10.2** is pinned by its multi-platform image digest there too. The prior published ordinary image 1.20.5 is used only to rehearse an upgrade and restore-based rollback. Do not use `latest` or 1.20.7: the latter was rolled back in production (TC-1176). New node-image changes remain unavailable to this published-image guide until a separately approved release ships.
This ordinary-server profile runs a single node with local SQLite, local blocks, and Caddy HTTPS. It pins the published **1.22.0** node image by immutable digest in [`compose.yaml`](compose.yaml): the multi-platform index pushed by the `docker.yml` run for the `v1.22.0` tag. Caddy **2.10.2** is pinned by its multi-platform image digest there too. The previous published image, 1.20.6, is used only to rehearse an upgrade and restore-based rollback. Pin by digest; do not use a mutable tag such as `latest`.

This is **not** a TEE: the VPS operator and its host can access the filesystem and key. No hardware attestation, dstack KMS isolation, or cross-host account migration is claimed. This guide does not configure registry discovery or hosted-to-owned-node replication (TC-1280/TC-1283). TLS authenticates your domain but does not attest the machine.

Expand All @@ -21,12 +21,12 @@ export NODE_HOME="$HOME/personal-tinycloud"
bash deploy/personal-node/node.sh init node.example.com
bash deploy/personal-node/node.sh check-missing-key
bash deploy/personal-node/node.sh start
bash deploy/personal-node/node.sh check-version 1.20.6
bash deploy/personal-node/node.sh check-version 1.22.0
```

Replace `node.example.com` with the domain you own. `init` writes a unique 48-byte random Static root to a `0600` private `node.env`; it is never echoed. The profile mounts a keyless TOML over the bundled image config, so even the older 1.20.6 image **fails** if the installation root is absent. `check-missing-key` must run before `start`; it requires the node's exact `Missing secret` diagnostic and rejects unrelated failures without printing their logs. Caddy obtains and renews a public certificate; do not bypass its trust verification.
Replace `node.example.com` with the domain you own. `init` writes a unique 48-byte random Static root to a `0600` private `node.env`; it is never echoed. The profile mounts a keyless TOML that replaces the config bundled in the image and fails closed when the installation root is absent. `check-missing-key` must run before `start`; it requires the node's exact `Missing secret` diagnostic and rejects unrelated failures without printing their logs. The upgrade rehearsal starts from the 1.20.6 image; CI runs the bare image check (`check-bare-image-missing-key`) on 1.22.0, starting the image with no mount, key or environment and requiring the same `Missing secret` diagnostic. Caddy obtains and renews a public certificate; do not bypass its trust verification.

The 1.20.6 config loader accepts both the legacy single-underscore nesting (`TINYCLOUD_KEYS_TYPE`, `TINYCLOUD_STORAGE_DATADIR`) and canonical double-underscore nesting (`TINYCLOUD_KEYS__TYPE`, `TINYCLOUD_STORAGE__DATADIR`). It merges canonical values last when both are set. This profile uses the canonical form; the existing dstack Compose retains its working single-underscore key-type setting.
The 1.20.6 and 1.22.0 config loaders accept both the legacy single-underscore nesting (`TINYCLOUD_KEYS_TYPE`, `TINYCLOUD_STORAGE_DATADIR`) and canonical double-underscore nesting (`TINYCLOUD_KEYS__TYPE`, `TINYCLOUD_STORAGE__DATADIR`). They merge canonical values last when both are set. This profile uses the canonical form; the existing dstack Compose retains its working single-underscore key-type setting.

To prove owner-held EOA operations, place your own wallet's 32-byte hexadecimal private key (without `0x`) in a separate `0600` file outside `NODE_HOME`, then run:

Expand Down Expand Up @@ -55,24 +55,32 @@ bash deploy/personal-node/node.sh backup /secure/path/node-backup-2026-10-16 "$B
export NODE_HOME="$HOME/personal-tinycloud-restored"
bash deploy/personal-node/node.sh restore /secure/path/node-backup-2026-10-16 "$BACKUP_PASSPHRASE_FILE"
bash deploy/personal-node/node.sh start
bash deploy/personal-node/node.sh check-version 1.20.6
bash deploy/personal-node/node.sh check-version 1.22.0
bash deploy/personal-node/node.sh verify /secure/path/owner.key
```

Use a fresh host/directory, not two live nodes serving the same identity and domain at once. Stop the old node before DNS cutover; wait for the new host's Caddy certificate, then verify before admitting clients. If verification fails, stop the destination and revert DNS to the old host. `verify` compares the public node DID saved before backup and the signed KV/SQL data; a different root fails rather than silently rotating. The backup is a full offline snapshot, **not** continuous replication. TLS ACME state is not in the backup, and a restore needs a reachable domain and valid certificate again.

## Upgrade and rollback rehearsal

CI executes [`ci.sh`](ci.sh) on a clean runner: a fresh 1.20.6 installation with missing-key denial, an unrelated-failure negative control, signed data and restart; then 1.20.5 → 1.20.6, with data checks before and after restart, fresh-volume restores, and rollback by restoring the **pre-upgrade** snapshot into another fresh instance on 1.20.5. An in-place downgrade of a migrated database is not supported. A normal new installation starts directly on 1.20.6; do not downgrade it merely to rehearse.
CI executes [`ci.sh`](ci.sh) on a clean runner: a fresh 1.22.0 installation with missing-key denial, an unrelated-failure negative control, the bare 1.22.0 image's missing-key failure, signed data and restart; then 1.20.6 → 1.22.0, with data checks before and after restart, fresh-volume restores, and rollback by restoring the **pre-upgrade** snapshot into another fresh instance on 1.20.6. An in-place downgrade of a migrated database is not supported. A normal new installation starts directly on 1.22.0; do not install 1.20.6 except to rehearse this upgrade.

```bash
# Run only on a disposable isolated runner with permission to create Docker
# containers. This is exactly the end-to-end job used by GitHub Actions.
bash deploy/personal-node/ci.sh
```

Before a future upgrade, take and verify an offline backup as above, stop the node, change the pinned image only to a reviewed released digest, start and verify identity and signed data. If verification fails, stop the candidate and restore that pre-upgrade backup to a fresh instance of the previous image; never blindly start the older binary against migrated data. The next-release upgrade step **requires the next approved production node release** and a new published-image CI run. TC-1176 prohibits promotion of that release until Sam separately clears the hold. Changes to the shipped default key in TC-1200 likewise require that next image; the mounted keyless profile is the safe 1.20.6 workaround, not proof that its bare image fails closed.
### What 1.22.0 changes for an upgrade from 1.20.6

- **Database migrations.** The upgrade applies seven migrations that 1.20.6 does not ship: `m20261014_000000_delegation_cid_alias`, `m20261015_000000_delegation_cursor_index`, `m20261017_000000_epoch_order_space_index` and `m20261018_000000_epoch_head` (1.21.0), `m20261020_000000_current_kv_key_c_index` (1.21.3, an index on PostgreSQL), `m20261021_000000_sql_store_manifest` (1.21.8, two empty tables) and `m20261022_000000_encryption_network_key` (1.21.9, a table and a nullable column, not yet used). 1.22.0 itself adds none. An older image refuses to start against a database that has recorded a migration it does not ship.
- **Rollback needs the restore.** Because of those migrations, rolling back from 1.22.0 means restoring the pre-upgrade backup into a fresh instance of the 1.20.6 image, as `ci.sh` does for its last instance. Take and verify that backup before the upgrade; data written after it is not in it.
- **Named keys and delegated SQL caveats are disabled.** `[global.encryption_network] named_keys_v1` and `[global.sql] delegated_caveats_enforced` default to off, and this profile does not set them. With them off, named-key routes are not mounted, a `tinycloud.sql/*` delegation carrying `tables`, `columns` or `readOnly` caveats is refused as an unsupported caveat, and `/version` does not list `encryption-named-keys-v1` or `sql-delegated-caveats-v1`. `check-version` asserts that neither feature is listed. Enabling either is outside this guide and is not exercised by CI.

### A later upgrade

Take and verify an offline backup as above, stop the node, change the pinned image only to a reviewed released digest, start and verify identity and signed data. If verification fails, stop the candidate and restore that pre-upgrade backup to a fresh instance of the previous image; never blindly start the older binary against migrated data. Moving the pin to a later release also means updating the previous/current pair in [`node.sh`](node.sh) and [`ci.sh`](ci.sh) and running the published-image CI job for that pair.

For a Caddy update, choose a reviewed release and resolve its published multi-platform index digest (for example, from the [Docker Hub tag metadata](https://hub.docker.com/v2/repositories/library/caddy/tags/2.10.2)). Replace only the `caddy@sha256:…` reference in `compose.yaml`, run the published-image CI job and verify HTTPS certificate issuance and renewal on a staging domain before upgrading a live proxy. Retain the Caddy `/data` and `/config` volumes; do not switch to a mutable tag.

The [personal-node workflow](../../.github/workflows/personal-node.yml) is the acceptance evidence only when its published-image job is green. Record the successful run URL and exact image versions/digests in TC-1282; no CI result here authorizes a production deployment or a stronger sovereignty claim.
The [personal-node workflow](../../.github/workflows/personal-node.yml) is the acceptance evidence only when its `published-images` job is green; its `dstack-profile` and `dstack-config-source` jobs cover the separate dstack Compose profile and the shipped key configuration. Record the successful run URL and exact image versions/digests in TC-1282; no CI result here authorizes a production deployment or a stronger sovereignty claim.
18 changes: 10 additions & 8 deletions deploy/personal-node/ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,18 @@ if "$here/node.sh" check-missing-key; then
fi
printf '%s\n' "$saved_image" > "$NODE_HOME/image"
"$here/node.sh" start
"$here/node.sh" check-version 1.20.6
"$here/node.sh" check-version 1.22.0
"$here/node.sh" check-bare-image-missing-key
"$here/node.sh" seed "$owner"
"$here/node.sh" restart
"$here/node.sh" verify "$owner"
"$here/node.sh" stop

export NODE_HOME="$work/tc1282-a"
"$here/node.sh" init localhost test 1.20.5
"$here/node.sh" init localhost test 1.20.6
"$here/node.sh" check-missing-key
"$here/node.sh" start
"$here/node.sh" check-version 1.20.5
"$here/node.sh" check-version 1.20.6
"$here/node.sh" seed "$owner"
"$here/node.sh" restart
"$here/node.sh" verify "$owner"
Expand All @@ -53,27 +54,28 @@ export NODE_HOME="$work/tc1282-a"
export NODE_HOME="$work/tc1282-b"
"$here/node.sh" restore "$work/pre-upgrade" "$passphrase"
"$here/node.sh" start
"$here/node.sh" check-version 1.20.5
"$here/node.sh" check-version 1.20.6
"$here/node.sh" verify "$owner"
"$here/node.sh" upgrade
"$here/node.sh" check-version 1.20.6
"$here/node.sh" check-version 1.22.0
"$here/node.sh" verify "$owner"
"$here/node.sh" backup "$work/post-upgrade" "$passphrase"
"$here/node.sh" stop

export NODE_HOME="$work/tc1282-c"
"$here/node.sh" restore "$work/post-upgrade" "$passphrase"
"$here/node.sh" start
"$here/node.sh" check-version 1.20.6
"$here/node.sh" check-version 1.22.0
"$here/node.sh" verify "$owner"
"$here/node.sh" stop

# Rollback is a restore from the pre-upgrade snapshot, not an in-place
# downgrade of a migrated database.
# downgrade of a migrated database: the upgrade applies database migrations
# that 1.20.6 does not know.
export NODE_HOME="$work/tc1282-d"
"$here/node.sh" restore "$work/pre-upgrade" "$passphrase"
"$here/node.sh" start
"$here/node.sh" check-version 1.20.5
"$here/node.sh" check-version 1.20.6
"$here/node.sh" verify "$owner"
"$here/node.sh" stop
printf '%s\n' 'Personal-node lifecycle passed: TLS, signed KV/SQL, restart, fresh restore, upgrade, rollback restore'
7 changes: 4 additions & 3 deletions deploy/personal-node/compose.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: personal-node
services:
node:
image: ${NODE_IMAGE:-ghcr.io/tinycloudlabs/tinycloud-node@sha256:41667d3d133aaa12290a2f3a744afacd838988327552d159c60f04466e66f86f}
image: ${NODE_IMAGE:-ghcr.io/tinycloudlabs/tinycloud-node@sha256:2f3c65d85aa94610456bf09cd54713bab5260097bb3d56858025bc6eadb5a4a0}
restart: unless-stopped
env_file:
- ${NODE_KEY_ENV_FILE:?Set the private node key file}
Expand All @@ -10,8 +10,9 @@ services:
ROCKET_ADDRESS: 0.0.0.0
volumes:
- ${NODE_DATA_DIR:?Set the node data directory}:/data
# 1.20.6 still embeds a sample key in its default TOML. This clean
# config masks the image copy, even when the operator key is absent.
# This config replaces the one bundled in the image and fails closed
# when the operator key is absent. The upgrade rehearsal starts from
# the 1.20.6 image; CI runs the bare image check on 1.22.0.
- ./tinycloud.toml:/tinycloud.toml:ro
security_opt:
- no-new-privileges:true
Expand Down
51 changes: 34 additions & 17 deletions deploy/personal-node/node.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@ umask 077

here="$(cd "$(dirname "$0")" && pwd)"
home="$(realpath -m "${NODE_HOME:-./personal-node}")"
image_1220='ghcr.io/tinycloudlabs/tinycloud-node@sha256:2f3c65d85aa94610456bf09cd54713bab5260097bb3d56858025bc6eadb5a4a0'
# The previous published image, used only to rehearse the upgrade.
image_1206='ghcr.io/tinycloudlabs/tinycloud-node@sha256:41667d3d133aaa12290a2f3a744afacd838988327552d159c60f04466e66f86f'
image_1205='ghcr.io/tinycloudlabs/tinycloud-node@sha256:eba68094f922b8d377d4439bfa14e7d12b875de0155f03810cf6b5d4cb3b1ab7'

fail() { printf '%s\n' "$*" >&2; exit 1; }
require_home() { [[ -f "$home/node.env" && -f "$home/image" && -f "$home/domain" ]] || fail 'Initialize or restore the node first'; }
Expand Down Expand Up @@ -64,18 +65,29 @@ identity() {
[[ -s "$home/current-identity" ]] || fail 'Node identity is unavailable'
}

# Accept only exit 1 plus the node's own complete ` Missing secret` line.
# Other diagnostics are never echoed.
require_missing_secret() {
local status="$1" file="$2" line missing=0
while IFS= read -r line; do
if [[ "$line" == ' Missing secret' ]]; then missing=1; fi
done < "$file"
[[ "$status" == 1 && "$missing" == 1 ]] ||
fail 'Missing-key check did not report the exact missing-secret error'
}

case "${1:-}" in
init)
[[ $# -ge 2 ]] || fail 'Usage: node.sh init DOMAIN [test] [1.20.5]'
[[ $# -ge 2 ]] || fail 'Usage: node.sh init DOMAIN [test] [1.20.6]'
[[ ! -e "$home" ]] || fail 'Choose a fresh NODE_HOME'
[[ "$2" =~ ^[a-zA-Z0-9.-]+$ ]] || fail 'Invalid domain'
[[ "${3:-}" == '' || "${3:-}" == test ]] || fail 'TLS mode must be test or omitted'
[[ "${4:-}" == '' || "${4:-}" == 1.20.5 ]] || fail 'Only 1.20.5 is supported as the previous image'
[[ "${4:-}" == '' || "${4:-}" == 1.20.6 ]] || fail 'Only 1.20.6 is supported as the previous image'
mkdir -p "$home/data"
printf '%s\n' "$2" > "$home/domain"
printf '%s\n' "${3:-production}" > "$home/tls-mode"
printf '%s\n' "$image_1206" > "$home/image"
if [[ "${4:-}" == 1.20.5 ]]; then printf '%s\n' "$image_1205" > "$home/image"; fi
printf '%s\n' "$image_1220" > "$home/image"
if [[ "${4:-}" == 1.20.6 ]]; then printf '%s\n' "$image_1206" > "$home/image"; fi
printf 'TINYCLOUD_KEYS__TYPE=Static\nTINYCLOUD_KEYS__SECRET=%s\n' \
"$(openssl rand -base64 48 | tr '+/' '-_' | tr -d '=\n')" > "$home/node.env"
sudo chown -R 1000:1000 "$home/data"
Expand All @@ -88,14 +100,19 @@ case "${1:-}" in
printf 'TINYCLOUD_KEYS__TYPE=Static\n' > "$blank"
status=0
NODE_KEY_ENV_FILE="$blank" timeout 30 "$0" compose-check >"$diagnostic" 2>&1 || status=$?
missing=0
while IFS= read -r line; do
if [[ "$line" == ' Missing secret' ]]; then missing=1; fi
done < "$diagnostic"
[[ "$status" == 1 && "$missing" == 1 ]] ||
fail 'Missing-key check did not report the exact missing-secret error'
require_missing_secret "$status" "$diagnostic"
printf '%s\n' 'Missing-key startup failed as required'
;;
check-bare-image-missing-key)
# The published image started as shipped: no mounted config, no key, no
# environment. Its own default configuration must refuse to start.
diagnostic="$(mktemp)"
trap 'rm -f "$diagnostic"' EXIT
status=0
timeout 60 docker run --rm "$image_1220" >"$diagnostic" 2>&1 || status=$?
require_missing_secret "$status" "$diagnostic"
printf '%s\n' 'Bare image startup without a key failed as required'
;;
compose-check)
compose run --rm --no-deps node
;;
Expand All @@ -111,11 +128,11 @@ case "${1:-}" in
compose stop
;;
check-version)
[[ $# == 2 ]] || fail 'Usage: node.sh check-version 1.20.5|1.20.6'
[[ $# == 2 ]] || fail 'Usage: node.sh check-version 1.20.6|1.22.0'
health
curl -fsS "${ca_args[@]}" "$(url)/version" |
jq -e --arg version "$2" '.version == $version and .quota_url == null and .inTEE == false' >/dev/null ||
fail 'Unexpected node version, hosted quota, or TEE mode'
jq -e --arg version "$2" '.version == $version and .quota_url == null and .inTEE == false and ((.features | index("encryption-named-keys-v1")) == null) and ((.features | index("sql-delegated-caveats-v1")) == null)' >/dev/null ||
fail 'Unexpected node version, hosted quota, TEE mode, or enabled named-key/delegated-SQL feature'
;;
verify|seed)
[[ $# == 2 && -f "$2" ]] || fail "Usage: node.sh $1 OWNER_KEY_FILE"
Expand Down Expand Up @@ -157,15 +174,15 @@ case "${1:-}" in
;;
upgrade)
require_home
[[ "$(<"$home/image")" == "$image_1205" ]] || fail 'Upgrade expects the previous published image'
[[ "$(<"$home/image")" == "$image_1206" ]] || fail 'Upgrade expects the previous published image'
compose stop node
printf '%s\n' "$image_1206" > "$home/image"
printf '%s\n' "$image_1220" > "$home/image"
compose up -d node
health
;;
destroy-test)
[[ "$(<"$home/tls-mode")" == test && "$(basename "$home")" == tc1282-* ]] || fail 'Only tc1282 test instances may be removed'
compose down -v
;;
*) fail 'Usage: node.sh {init|check-missing-key|start|check-version|seed|verify|restart|stop|backup|restore|upgrade|destroy-test}' ;;
*) fail 'Usage: node.sh {init|check-missing-key|check-bare-image-missing-key|start|check-version|seed|verify|restart|stop|backup|restore|upgrade|destroy-test}' ;;
esac
Loading