feat(stargate): hot reload transport TLS material

[mikeyrcamp]

Why

Stargate, stargate-k8s-router, and Pylon read TLS material only during process startup. Kubernetes can update mounted Secret and ConfigMap files after certificate renewal or CA rotation, but the running processes continue using their in-memory TLS configuration.

Today:

• Server identities do not change after a mounted certificate and private key are renewed.
• Client trust stores do not change after a mounted CA or trust bundle is updated.
• Routine renewal and CA rotation therefore require coordinated workload restarts.
• Removing a trust root does not affect new or established connections until the participating client process restarts.
• No first-party reload status, readiness signal, or metric exists because no reload is attempted.

This creates avoidable disruption during ordinary renewal and delays enforcement of trust changes during emergency CA revocation.

The current startup-only reads are in:

• src/libraries/rust/stargate/crates/stargate/src/main/startup.rs
• src/libraries/rust/stargate/crates/stargate-k8s-router/src/main.rs
• src/libraries/rust/stargate/crates/pylon/src/startup.rs

Transport and material matrix

Hot reload must cover each TLS role rather than only a named binary or file:

Component	Connection role	Material to reload
Stargate	Server accepting reverse tunnels	Server certificate and private key
Stargate	Client dialing direct backends or relays	CA and trust bundle
Pylon	Server accepting direct tunnels	Server certificate and private key
Pylon	Client dialing reverse-tunnel endpoints	CA and trust bundle
stargate-k8s-router	Server accepting inbound tunnels	Server certificate and private key
stargate-k8s-router	Client dialing upstream Raw QUIC or WebTransport endpoints	CA and trust bundle

If one configured file currently supplies more than one role, each resulting runtime configuration must still reload safely and atomically.

Required behavior

• Detect Kubernetes Secret and ConfigMap projected-volume updates, including atomic symlink replacement.
• Read a complete replacement generation, build the candidate rustls and Quinn configuration off the active path, validate it, and atomically swap the complete identity or trust store.
• Apply a valid replacement to new TLS connections within 30 seconds after the complete projected update is visible inside the container. This bound excludes kubelet projection delay.
• Keep the last-known-good configuration when a replacement is missing, incomplete, inconsistent, oversized, or invalid.
• Debounce duplicate filesystem events and retry transient reads without busy-looping.
• Do not require a pod restart for ordinary leaf renewal or planned overlapping CA rotation.

Existing connections

• Routine server leaf renewal and trust expansion may allow established QUIC connections to drain normally.
• Trust contraction, including removal of a CA, must close connections using the affected client trust configuration and reconnect under the replacement trust store within the reload bound.
• Documentation must distinguish planned overlapping rotation from emergency trust revocation and describe the expected connection impact for each.

Security requirements

• Verify that a server certificate and private key match before activation.
• Reject an empty certificate chain, an empty trust bundle, or a bundle containing no usable trust anchors.
• Reject malformed, expired, or not-yet-valid replacement identities and retain the last-known-good identity.
• Never fall back to quic_insecure, a generated self-signed identity, an empty trust store, or another weaker mode because reload failed.
• Preserve explicit development-only insecure behavior when it was selected at process startup, but do not enable it dynamically.
• Keep wrong-root and wrong-SNI connections fail-closed before, during, and after reload.
• Do not log PEM contents, private keys, tokens, Secret values, or other credential material.
• Bound file sizes and watcher work so repeated events or malformed files cannot cause unbounded memory, CPU, or log growth.

Readiness and observability

• If secure mode is configured and no valid initial TLS material can be loaded, fail startup or remain not ready.
• A failed later reload must leave the process ready only while its last-known-good material remains usable. The failure must be visible through logs and metrics.
• Active identity expiry without a valid replacement must make the affected component not ready.
• Emit structured logs for successful and failed reloads with component, material type, result, and non-sensitive certificate metadata.
• Add pre-initialized, bounded reload success and failure counters by component and material type.
• Add a bounded certificate-expiry gauge if useful.
• Never use certificate serial numbers, fingerprints, paths, hostnames, or generation identifiers as metric labels. Serial numbers or fingerprints may appear in structured logs when useful for verification.

Acceptance criteria

1. Pylon direct mode presents a renewed server identity to new connections without a pod restart.
2. Stargate reverse mode presents a renewed server identity to new connections without a pod restart.
3. Stargate direct or relay clients use an updated trust bundle for new connections without a pod restart.
4. Pylon reverse-tunnel clients use an updated trust bundle for new connections without a pod restart.
5. stargate-k8s-router reloads both its inbound identity and its upstream trust for Raw QUIC and WebTransport.
6. Each valid replacement is active within the documented 30-second in-container reload bound.
7. An overlapping old-plus-new trust bundle supports planned CA rotation without insecure fallback.
8. Removing the old CA prevents new connections under the old root and closes affected established connections before reconnecting under the new trust store.
9. An invalid, incomplete, mismatched, empty, expired, or not-yet-valid replacement is rejected and the last-known-good configuration remains active.
10. Wrong-root and wrong-SNI connections fail closed throughout identity and trust rotation.
11. Tests exercise Kubernetes Secret and ConfigMap projected-volume semantics, including atomic directory symlink replacement, duplicate events, transient partial reads, and one-generation certificate and key snapshots.
12. Tests cover reload event storms and enforce bounded retry, memory, CPU, and log behavior.
13. Readiness and pre-initialized metrics distinguish initial load failure, successful reload, rejected reload, active material nearing expiry, and active material expiry.
14. Documentation describes the supported rotation order, emergency revocation procedure, behavior of existing connections, failure recovery, and verification commands.
15. The request-router chart does not require manual workload restarts for ordinary leaf renewal or planned overlapping CA rotation once every participating component supports reload.

Related

Relates to #502.

Follow-up to #585.