fix: serve resolves provider and Auto node identities again (TC-1442)

[samgbafa]

TC-1442 S0a'. Stacked on #355 (base fix/tc-1442-rewire-node-cli, head bb04778); the two should ship together.

Problem

aa6b161 (#145) replaced key_provider::resolve_identity_state(.., IdentityPurpose::Serve) in app with resolve_keys. Two of its arms bail:

• Keys::Provider: "provider key mode must be resolved through the node control key-provider path". node service install writes [global.keys] type = "provider" (service.rs, write_bootstrap_config_if_absent), so tinycloud serve --config <that file> exits 1 and the installed unit (Restart=on-failure) crash-loops.
• Keys::Auto (the default) with no dstack socket, or with TINYCLOUD_TEE_MODE=off: "No key source configured" / "TEE mode disabled ...". At 8722442 these generated and stored the node identity.

ControlPlaneHandle::set_identity_snapshot also lost its only caller, so the control plane kept the Probe snapshot taken at spawn and reported identityReady: false for a freshly generated identity.

Change

• resolve_keys returns Result<Option<StaticSecret>>. None replaces the Provider bail and the two Auto bails. The Static and dstack arms are unchanged apart from the Some wrapping.

• On None, app_with_control calls key_provider::resolve_identity_state(Some(&keys), datadir, IdentityPurpose::Serve), the same call 8722442 made. It generates and stores the identity on first start, then hands identity_snapshot to the control plane when one is attached.

• Auto fails closed where a TEE is indicated. Auto creates a local identity when TINYCLOUD_TEE_MODE is not off and the dstack paths returned no key: no dstack socket, or a build without dstack. In that case resolve_keys now refuses (Failed to start tinycloud-node, exit 1, no identity written) if any of these is present:

  • TINYCLOUD_TEE_MODE=dstack (case-insensitive). A dstack build already asks the socket for this value and fails if it is missing; a build without dstack used to fall through and mint.
  • tee.mode = "Dstack" (TINYCLOUD_TEE__MODE=Dstack)
  • TINYCLOUD_KEYS_TYPE / TINYCLOUD_KEYS__TYPE = Dstack, for Auto reached through an override. Variable names match case-insensitively, as figment's TINYCLOUD_ prefix does, so tinycloud_keys_type=Dstack counts too.
  • DSTACK_SIMULATOR_ENDPOINT set
  • a dstack socket present in a build without dstack support

  Example: refusing to create a new node identity: tee.mode = "Dstack" indicates a dstack TEE, but no dstack key is available (socket /var/run/dstack.sock). Start the node with its dstack socket, or remove tee.mode = "Dstack" if this host is not a TEE. Auto creates a local identity only when none of these is present.

• Provider keys refuse under the same check, whether or not an identity already exists: refusing keys.type = provider: TINYCLOUD_KEYS_TYPE=Dstack indicates a dstack TEE, where the node key must come from dstack. Use keys.type = Dstack, or remove TINYCLOUD_KEYS_TYPE=Dstack if this host is not a TEE. (exit 1, no identity written). TINYCLOUD_TEE_MODE=off does not exempt provider keys. A provider node with no marker starts as before.

• Production's signal is TINYCLOUD_KEYS_TYPE=Dstack plus the mounted /var/run/dstack.sock (every dstack compose and ops/rollback/*; none sets TINYCLOUD_TEE_MODE). That selects Keys::Dstack, never Auto, so production does not reach this branch. A build without dstack rejects type = Dstack at config load.

• TINYCLOUD_TEE_MODE=off keeps today's meaning. It is checked first, skips socket detection and every indication above, and creates or loads a local identity. On its own it does not load, because figment also maps it to tee.mode = "off", which is not a TeeMode variant. That is unchanged, and it is why scripts/tc959-docker.mjs also sets TINYCLOUD_TEE__MODE=Off.

• Creating an identity logs once, at WARN: created a new node identity at <location> (<did>). The location is the encrypted-file path, or the keychain item on macOS, and no key material is logged. This replaces the old INFO generated node identity secret on first run. A restart that loads the identity logs nothing. The node DID is the public value served at /.well-known/tinycloud/node-keys. Only Provider and Auto nodes create identities, and production uses dstack keys, so this line does not appear in production logs.

• tests/node_cli.rs no_arguments_keep_the_server_start_path depended on Auto failing. With Auto restored, a bare tinycloud in an empty HOME generates an identity and serves. The test now makes the server start fail on a share-email config error, which the server start path reports before it opens a key or binds. The assertions are unchanged.

Nothing else changes. No shipped deploy config uses Provider or Auto: the dstack composes, ops/rollback/*, scripts/ci/fixtures/tc959-n0-compose.yaml and docker.yml:784 set Dstack; tinycloud.toml, deploy/personal-node/*, deploy/share-email/* and test/docker-compose.yml set Static.

Tests (tests/startup_identity.rs, built binary)

Test	What it runs	On bb04778	On this branch
installed_provider_config_serves_a_stable_identity (Linux)	node service install (stub systemctl, temp HOME/XDG, encrypted-file provider), then serve --config <installed file>. Asserts node-keys answers, the log has exactly one created a new node identity at <installed data>/keys/identity.key.enc (<did>), node status --json has identityReady: true and the same nodeDid, and a restart serves the same DID with no creation line.	FAIL: "provider key mode must be resolved ..."	pass
installed_config_without_a_key_section_serves_a_stable_identity (Linux)	Same, with the [global.keys] section removed (Auto)	FAIL: "No key source configured"	pass
bare_start_without_a_tee_creates_one_identity_and_reloads_it (Linux)	Bare tinycloud, no config, no TEE indication: serves, writes data/keys/identity.key.enc, logs creation once; restart serves the same DID with no creation line	FAIL (5b9ebf2 code): no creation line	pass
auto_refuses_to_create_an_identity_where_a_tee_is_indicated (Linux)	Bare tinycloud with Auto keys under each indication above, plus production's plain TINYCLOUD_KEYS_TYPE=Dstack, no socket: exits non-zero, no data/keys, no creation line, the refusal names the indication	FAIL (5b9ebf2 code): kept running (minted and served)	pass
provider_refuses_where_a_tee_is_indicated (Linux)	Bare tinycloud with TINYCLOUD_KEYS__TYPE=provider plus each of TINYCLOUD_KEYS_TYPE=Dstack, TINYCLOUD_TEE_MODE=Dstack, TINYCLOUD_TEE__MODE=Dstack, DSTACK_SIMULATOR_ENDPOINT, no socket: exits non-zero, no data/keys, no creation line, the refusal names the marker	FAIL (570cb45 code): kept running (minted and served)	pass
shipped_config_with_a_static_secret_keeps_its_did	Repo tinycloud.toml + TINYCLOUD_KEYS__SECRET fixture, bare tinycloud and serve --config	pass	pass, same pinned DID
dstack::production_dstack_config_keeps_its_did (--features dstack)	Repo tinycloud.toml + TINYCLOUD_KEYS_TYPE=Dstack (production's env), bare tinycloud, stub guest agent on DSTACK_SIMULATOR_ENDPOINT	pass	pass, same pinned DID; every GetKey asked for tinycloud/keys/primary

The pinned DIDs were read from the base build, not computed by the new code. Each test also checks that its pin is the fixture key's node_did(). All fixture keys are throwaway constants.

Mutation check: with the set_identity_snapshot call removed, the provider test fails on identityReady: false, nodeDid: null.

CI's default job does not enable dstack, so the dstack DID test runs only with cargo test -p tinycloud-node --features dstack --test startup_identity. I ran that locally.

Local runs (with CI unset): cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p tinycloud-node --all-targets --features dstack -- -D warnings, cargo test -p tinycloud-node (all pass; the Postgres tests skip because no URL is set), and cargo test -p tinycloud-node --features dstack --lib --test startup_identity --test node_cli (all pass).

Interaction with TC-887 (#325, #326, #327)

• feat(TC-887): at-rest key escrow crate and ceremony CLI #325 (escrow crate and CLI) and feat(TC-887): at-rest keyring, key store and migration #326 (keyring in tinycloud-core) do not touch startup key resolution.
• feat(TC-887): node wiring for at-rest keys and escrow recovery #327 edits app_with_control after this change's lines: refuse_unknown_tee in the dstack get_info error arm, AtRestRuntime::new, at_rest::setup(.., key_setup.as_bytes(), ..), and .manage(at_rest_runtime). A trial merge (git merge-tree) with each PR head adds no conflicts beyond the ones those PRs already have against the current tree (tinycloud-core/src/migrations/mod.rs, and config.rs for feat(TC-887): node wiring for at-rest keys and escrow recovery #327). lib.rs merges cleanly.
• A provider-stored at-rest root is out of scope for feat(TC-887): node wiring for at-rest keys and escrow recovery #327. At-rest stays opt-in, and off, on Provider and Auto nodes.
• Order: land this with fix: re-wire the tinycloud CLI dispatch (TC-1442) #355, before feat(TC-887): node wiring for at-rest keys and escrow recovery #327. No rebase of feat(TC-887): at-rest key escrow crate and ceremony CLI #325 or feat(TC-887): at-rest keyring, key store and migration #326 is needed for this change. feat(TC-887): node wiring for at-rest keys and escrow recovery #327 needs its existing rebase onto main anyway; this change adds no textual work to that rebase.

Not changed

serve builds its tokio runtime in cli.rs block_on, so ROCKET_WORKERS and ROCKET_MAX_BLOCKING are ignored on that path, and serve reads --config (or the profile path), not TINYCLOUD_CONFIG_FILE. For service-installed nodes this makes no difference today. The generated unit sets no Environment= and always passes --config, and with those variables unset the tokio defaults (one worker per CPU, 512 blocking threads) match Rocket's. It would matter only if an operator added those variables to the unit. The production container runs with no arguments, which goes through rocket::execute and honours the Dockerfile's ROCKET_MAX_BLOCKING=1024. Not changed here.

[samgbafa]

APPROVE: see TC-1442 (reviewed exact head 570cb45)

[samgbafa]

APPROVE: see TC-1442 (reviewed exact head 08dc5a1)

[samgbafa]

APPROVE: see TC-1442 (main-merge only; patch-id unchanged vs 08dc5a1; head 50099b1)

Detail: merged #355's updated head 5d22a77 (which contains origin/main 9573f0e) into the approved head 08dc5a1 with a merge commit, no conflict. The PR diff against #355's head is unchanged: patch-id --stable 8a3cf020ca444dc0969c4b344404994c66d1f63a before and after, and the added/removed lines are byte-identical per file (lib.rs, node_control/key_provider.rs, routes/node_keys.rs, tests/node_cli.rs, tests/startup_identity.rs). Waiver per the release plan. A clean textual merge does not prove behaviour on the new main (main changed lib.rs by ~580 lines), so CI must be green at this head, and the startup_identity tests must have run, before merge.