diff --git a/CHANGELOG.md b/CHANGELOG.md index d4326c3d..b3b7f976 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ - Policy/v3 registration checks the owner and existence of the policy's encryption networks. Every network the policy names (`contentSource.encryptionNetwork` and each `encryption` ceiling entry) must be owned by the owner of the content source's space, compared as principals (a `did:pkh` matches regardless of EIP-55 letter case; the chain id must match), and must exist on this node in state `active`. Every `kv` ceiling entry must name a space of that same owner. Otherwise registration returns `403 content-source-network-invalid` and writes nothing. A new registration whose `kvResource` or any `kv` ceiling entry uses the legacy `tinycloud://` form gets the same refusal; no client emits it. Share and CLI registrations carry one `kv` entry, equal to the source, and name the space owner's `:default` network, which the SDK creates or finds at sign-in, so they are unaffected. Stored registrations are not refused; they are classified with the same ownership rule (see the next entry). Refusals are counted as `tinycloud_policy_source_network_refused_total{reason}`, with reasons `unparsable`, `legacy_resource`, `foreign_owner`, `foreign_space`, `missing` and `inactive`. Details: TC-1423. - Policy/v3 registration records which of the owner's delegations granted what the policy roots carry, with their ancestor closure, in two new tables (migration `m20261019_000000_policy_v3_owner_authority`; its `down` drops both). Delegations from the owner of the capability's space or encryption network are tried before the general window of recent delegations to the owner that touch that space or network; a delegation whose ancestry includes one issued by the policy owner is never recorded; a grant that outlives the root is preferred. Policies registered before the migration are classified in the background on first use (steady-state reads add no query), or by an opt-in sweep (`policy_v3.owner_binding_backfill.enabled`, off by default). A stored policy that names an encryption network or a `kv` space that does not belong to the owner of its content source is classified `unknown` with reason `foreign_network`. The legacy cutover defaults to the migration's applied-at time, is logged at boot, and `policy_v3.legacy_cutover` overrides it after a rollback and re-upgrade. New `POST /policy/v3/authorized` lists the policies whose recorded authority includes a delegation, authorized like `/delegation/status`. See `docs/policy-owner-authority.md`. Details: TC-1423. - Policy/v3 registration refuses a root that expires later than registration + 31 days (+5 minutes) (`403 policy-root-lifetime-exceeded`), in both modes; released SDK roots pass. New `policy_v3.owner_authority_expiry`: `record` (default) accepts a root that outlives the owner authority and counts it, `enforce` refuses it (`403 policy-root-outlives-owner-authority`). Registrations made before the legacy cutover are not capped; roots over 31 days among them are counted. `GET /info` lists `policy-owner-authority-v1` and one of `policy-owner-authority-record` / `policy-owner-authority-enforce`. New counters `tinycloud_policy_owner_authority_expiry_exceeded_total{authority_ttl_bucket}`, `tinycloud_policy_root_lifetime_refused_total`, `tinycloud_policy_legacy_root_lifetime_exceeded_total`, `tinycloud_policy_owner_binding_backfill_total{outcome}`; counting queries in `docs/policy-owner-authority.md`. Details: TC-1423. +- Policy/v3 uses check the recorded owner authority: once a recorded delegation or any ancestor is revoked (also under an alias CID), admitted sessions stop reading and decrypting, and re-share admission, signed-KV and hook-ticket mints, challenge, mint and S0 admission are refused (`403 policy-owner-authority-revoked`), and so is delivery (`403 delivery-authorization-invalid`); a non-legacy `bound` authority past its expiry gives `policy-owner-authority-expired`; a policy without authority gives `policy-owner-authority-unknown`, unless it is legacy and `policy_v3.legacy_unknown_allow` (default off, logged) serves it with root-only checks. The mint's registration-time rescan runs only for such escaped policies. An expired root returns `root-expired`; other roots that are not live return `root-not-live`. Hook tickets carry a policy session's authority closure (at most 16 IDs, else `403 hook-ticket-authority-too-large`), and open hook streams re-check their authority before delivering (at most every `hooks.authority_recheck_seconds`, 5) and while idle (`hooks.idle_authority_recheck_seconds`, 30), closing on a revocation or three consecutive failed checks spaced an interval apart (events are withheld meanwhile); concurrent first uses of an unclassified policy share one classification; `delegation_chains_unrevoked` takes two queries at any depth. New counters `tinycloud_policy_owner_authority_denied_total{reason}` and `tinycloud_policy_owner_authority_unknown_allowed_total{reason}`. Details: TC-1423. ## [1.22.0] - 2026-10-23 diff --git a/docs/policy-owner-authority.md b/docs/policy-owner-authority.md index 68957693..93d38325 100644 --- a/docs/policy-owner-authority.md +++ b/docs/policy-owner-authority.md @@ -168,17 +168,16 @@ unless `classifier_version` changes. ## Policies registered earlier -A policy with no binding row is classified in the background on first use, -off the request: the selection runs as of its registration time (delegations -issued after it are not considered; UCANs carry no issue time and stay -eligible), then one transaction inserts the binding row first and its closure -after it. A concurrent writer that stored a row first wins. One task runs per -policy, and at most three in the whole process; a use that finds no free slot -starts none (a later use does). If the write fails (lock timeout, SQLite busy) or the result -is `starved`, the row stays pending and the request path doesn't start another -attempt for 60 seconds. Policies whose stored classification is final are -remembered in process (bounded), so steady-state reads add no query. An -`unknown` row written by an older `classifier_version` is classified again. +A policy with no final binding row is classified on first use, because the +use is checked against the result: the selection runs as of its registration +time (delegations issued after it are not considered; UCANs carry no issue +time and stay eligible), the request is served from that result, and the +write runs in the background: one transaction inserts the binding row first +and its closure after it, and a concurrent writer that stored a row first +wins. One write runs per policy, and at most three in the whole process. A +result that isn't final (its write failed or found no free slot, or it is +`starved`) is reused for 60 seconds before it is computed again. An `unknown` +row written by an older `classifier_version` is classified again. The opt-in sweep classifies pending policies off the request path: @@ -246,3 +245,67 @@ Pages hold at most 100 policies in policy-CID order; `nextCursor` is set when a page is full and passed back as `cursor`. There is no total count. Policies not yet classified (legacy policies not used since the upgrade, with the sweep off) are not listed. Database errors return `503 policy-listing-unavailable`. + +## Checking at use + +Every use of a policy checks its recorded owner authority, after the root +checks: reads and decrypts by admitted sessions (and their re-shared +children), re-share admission, signed-KV and hook-ticket mints, challenge, +mint (again on the mint's locked transaction), S0 admission, and delivery +authorization. + +| state | at use | +|---|---| +| `bound`, `bound-uncapped` | denied if any recorded delegation or ancestor is revoked, directly or under an alias CID (`403 policy-owner-authority-revoked`); a `bound` authority past its expiry is denied (`403 policy-owner-authority-expired`), except for legacy (`backfill`) rows, which are never expiry-checked | +| `root-authority` | nothing more | +| `unknown` | denied (`403 policy-owner-authority-unknown`) | + +The mint does not rescan the owner's delegations as of registration for +policies with a recorded authority. + +What to check is cached in process once a classification is final (bounded); +liveness is not cached. A steady-state use costs one indexed revocation query +for a recorded closure and none for root authority. A policy without a final +classification is classified in the request, served from that result, and +stored in the background; a result that isn't final is reused for 60 seconds. +Concurrent first uses of one policy share a single classification. + +A revocation takes effect on commit; a request already past the check may +finish. Signed-KV tickets minted before a revocation last at most 60 seconds. + +Other codes: an expired root gives `root-expired` (other non-live status +gives `root-not-live`); S0 admission returns the owner-authority code itself; +delivery keeps `delivery-authorization-invalid`. No code carries a CID or DID. +`policy-owner-authority-expired` is not reached by stored rows: a `bound` +root expires no later than its authority, and the root and session expiry +checks run first (`policy-session-expired` on reads, `root-expired` on +challenge and mint). It stays as a defence in depth. + +### Legacy escape + +```toml +[policy_v3] +legacy_unknown_allow = false # TINYCLOUD_POLICY_V3__LEGACY_UNKNOWN_ALLOW +``` + +When true, legacy (`backfill`) policies classified `unknown` are served with +the root-only checks (and the mint's registration-time rescan); when false they +are denied. The node logs a warning at boot, and per policy at most once an +hour when it serves one; uses are counted in +`tinycloud_policy_owner_authority_unknown_allowed_total{reason}`. Denials are +counted in `tinycloud_policy_owner_authority_denied_total{reason=revoked|expired|unknown}`. + +### Hook streams + +A hook ticket carries the delegations that authorized it, plus, for a policy +session, the recorded owner-authority closure; a ticket with more than 16 is +refused (`403 hook-ticket-authority-too-large`). An open `/hooks/events` +stream re-checks them (one closure query plus one revocation query) before +delivering an event, at most every `hooks.authority_recheck_seconds` (5), and +every `hooks.idle_authority_recheck_seconds` (30) while idle. It closes on a +revocation, or after three consecutive failed checks, each at least an +interval after the last; until a check succeeds, events are withheld (dropped) +without querying, so a burst of events during a brief database error doesn't +close the stream. Streams don't see a JSON-route +`RootRevocation` (it writes no core revocation) or an authority expiry; those +end at the ticket deadline (at most 300 seconds), as before. diff --git a/tinycloud-core/src/db.rs b/tinycloud-core/src/db.rs index 6cbc4b40..b67080b2 100644 --- a/tinycloud-core/src/db.rs +++ b/tinycloud-core/src/db.rs @@ -708,15 +708,11 @@ where if roots.is_empty() { return Ok(true); } - let chain_ids = revocation::ancestor_chain_ids_for_roots(&self.conn, &roots) + // TC-1423: one closure query plus one alias-aware revocation query, + // whatever the depth; hook streams re-check this while open. + crate::policy_owner_authority::chains_unrevoked(&self.conn, &roots) .await - .map_err(|error| DbErr::Custom(error.to_string()))?; - for id in chain_ids { - if revocation::is_revoked(&self.conn, &id).await? { - return Ok(false); - } - } - Ok(true) + .map_err(|error| DbErr::Custom(error.to_string())) } /// Load and reparse a delegation from its exact signed Authorization diff --git a/tinycloud-core/src/policy_owner_authority.rs b/tinycloud-core/src/policy_owner_authority.rs index 6c96499d..4d5884dc 100644 --- a/tinycloud-core/src/policy_owner_authority.rs +++ b/tinycloud-core/src/policy_owner_authority.rs @@ -123,3 +123,92 @@ pub async fn migration_applied_at( .await? .map(|row| row.applied_at)) } + +/// Whether nothing in the ancestor closure of `roots` is revoked (directly +/// or under an alias CID): one recursive closure query plus one revocation +/// query, whatever the depth. Fails closed past the traversal budget. +pub async fn chains_unrevoked( + db: &C, + roots: &[Hash], +) -> Result { + if roots.is_empty() { + return Ok(true); + } + let (ids, _) = crate::auth_graph::load_closure_edges(db, roots).await?; + Ok(revoked_ids(db, &ids).await?.is_empty()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::hash::hash; + use crate::migrations::Migrator; + use crate::models::actor; + use crate::relationships::parent_delegations; + use sea_orm::{ActiveModelTrait, ActiveValue::Set, ConnectOptions, Database}; + use sea_orm_migration::MigratorTrait; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::Arc; + + #[tokio::test] + async fn chain_check_costs_two_queries_at_any_depth() { + let mut db = Database::connect(ConnectOptions::new("sqlite::memory:".to_string())) + .await + .unwrap(); + Migrator::up(&db, None).await.unwrap(); + actor::ActiveModel { + id: Set("did:key:actor".to_string()), + } + .insert(&db) + .await + .unwrap(); + let ids: Vec = (0..6) + .map(|i| hash(format!("chain-{i}").as_bytes())) + .collect(); + for id in &ids { + delegation::ActiveModel { + id: Set(*id), + delegator: Set("did:key:actor".to_string()), + delegatee: Set("did:key:actor".to_string()), + expiry: Set(None), + issued_at: Set(None), + not_before: Set(None), + facts: Set(None), + serialization: Set(id.as_ref().to_vec()), + } + .insert(&db) + .await + .unwrap(); + } + for pair in ids.windows(2) { + parent_delegations::ActiveModel { + child: Set(pair[0]), + parent: Set(pair[1]), + } + .insert(&db) + .await + .unwrap(); + } + let queries = Arc::new(AtomicUsize::new(0)); + let counter = Arc::clone(&queries); + db.set_metric_callback(move |_| { + counter.fetch_add(1, Ordering::SeqCst); + }); + assert!(chains_unrevoked(&db, &ids[..1]).await.unwrap()); + assert_eq!(queries.load(Ordering::SeqCst), 2); + + revocation::ActiveModel { + id: Set(hash(b"revoke-chain-top")), + revoker: Set("did:key:actor".to_string()), + revoked: Set(ids[5]), + serialization: Set(b"revocation".to_vec()), + revoked_at: Set(None), + } + .insert(&db) + .await + .unwrap(); + queries.store(0, Ordering::SeqCst); + assert!(!chains_unrevoked(&db, &ids[..1]).await.unwrap()); + assert_eq!(queries.load(Ordering::SeqCst), 2); + } +} diff --git a/tinycloud-node-server/src/config.rs b/tinycloud-node-server/src/config.rs index 43870ba7..559caf38 100644 --- a/tinycloud-node-server/src/config.rs +++ b/tinycloud-node-server/src/config.rs @@ -921,6 +921,22 @@ pub struct HooksConfig { pub webhook_timeout_seconds: u64, #[serde(default = "default_hooks_webhook_max_attempts")] pub webhook_max_attempts: usize, + /// An open event stream re-checks its ticket's authority before + /// delivering an event, at most this often. + #[serde(default = "default_hooks_authority_recheck_seconds")] + pub authority_recheck_seconds: u64, + /// An open event stream re-checks its ticket's authority this often + /// even when no event arrives. + #[serde(default = "default_hooks_idle_authority_recheck_seconds")] + pub idle_authority_recheck_seconds: u64, +} + +fn default_hooks_authority_recheck_seconds() -> u64 { + 5 +} + +fn default_hooks_idle_authority_recheck_seconds() -> u64 { + 30 } fn default_hooks_max_ticket_ttl_seconds() -> u64 { @@ -962,6 +978,8 @@ impl Default for HooksConfig { ), webhook_timeout_seconds: default_hooks_webhook_timeout_seconds(), webhook_max_attempts: default_hooks_webhook_max_attempts(), + authority_recheck_seconds: default_hooks_authority_recheck_seconds(), + idle_authority_recheck_seconds: default_hooks_idle_authority_recheck_seconds(), } } } @@ -972,6 +990,9 @@ impl HooksConfig { if self.webhook_max_attempts > 10 { return Err("hooks.webhook_max_attempts must not exceed 10"); } + if self.authority_recheck_seconds == 0 || self.idle_authority_recheck_seconds == 0 { + return Err("hooks authority re-check intervals must be at least 1 second"); + } Ok(()) } } @@ -1314,6 +1335,12 @@ pub struct PolicyV3Config { /// `TINYCLOUD_POLICY_V3__OWNER_AUTHORITY_EXPIRY`. #[serde(default)] pub owner_authority_expiry: OwnerAuthorityExpiry, + /// Operator escape: serve policies registered before the legacy cutover + /// for which no owner authority is recorded with the root-only checks; + /// when off they are denied. Off by default; logged at boot and per + /// policy when used. Env `TINYCLOUD_POLICY_V3__LEGACY_UNKNOWN_ALLOW`. + #[serde(default)] + pub legacy_unknown_allow: bool, #[serde(default)] pub owner_binding_backfill: OwnerBindingBackfillConfig, } diff --git a/tinycloud-node-server/src/policy_v3.rs b/tinycloud-node-server/src/policy_v3.rs index a5746105..36ae844e 100644 --- a/tinycloud-node-server/src/policy_v3.rs +++ b/tinycloud-node-server/src/policy_v3.rs @@ -298,6 +298,10 @@ impl PolicyV3Runtime { return Err("policy-session-root-graph-mismatch"); } } + // TC-1423: the owner authority must hold; its code passes + // through. + self.validate_owner_authority(&self.conn, ®istration, OffsetDateTime::now_utc()) + .await?; Ok(()) } @@ -363,12 +367,15 @@ impl PolicyV3Runtime { Ok(()) } + /// `Ok(None)` for an ordinary invocation; for a policy-session chain, + /// `Ok(Some(closure))` with the recorded owner-authority closure it + /// passed (empty for root authority). pub async fn authorize_invocation( &self, tinycloud: &crate::TinyCloud, invocation: &InvocationInfo, now: OffsetDateTime, - ) -> Result { + ) -> Result>, &'static str> { if invocation.parents.len() != 1 { for parent in invocation.parents.iter().copied() { if tinycloud @@ -381,7 +388,7 @@ impl PolicyV3Runtime { return Err("policy-session-requires-one-immediate-parent"); } } - return Ok(false); + return Ok(None); } let parent_cid = invocation.parents[0]; let loaded = tinycloud.load_signed_delegation(parent_cid).await; @@ -390,9 +397,9 @@ impl PolicyV3Runtime { Err(_) if self.is_policy_protected_parent(parent_cid).await? => { return Err("policy-session-invalid"); } - Err(_) => return Ok(false), + Err(_) => return Ok(None), }) else { - return Ok(false); + return Ok(None); }; // Sibling roots are stored in the ordinary graph only so S0 can have // normal proof edges. Neither root is invocation authority by itself. @@ -405,7 +412,7 @@ impl PolicyV3Runtime { return Err("policy-root-cannot-authorize-invocation"); } if !is_policy_session(&immediate.0) { - return Ok(false); + return Ok(None); } if immediate_row.delegatee != invocation.invoker || immediate.0.delegate != invocation.invoker @@ -413,12 +420,12 @@ impl PolicyV3Runtime { // Principal/audience mismatch is an ordinary graph authorization // failure. Defer it to the existing authorization path so TC-405 // does not change its status contract or error vocabulary. - return Ok(false); + return Ok(None); } if !capabilities_are_contained(&invocation.capabilities, &immediate.0.capabilities) { return Err("policy-invocation-capability-widening"); } - let (row, session) = self + let (row, session, owner_authority) = self .validate_policy_chain(tinycloud, invocation.parents[0]) .await?; let payload = invocation.invocation.payload(); @@ -528,7 +535,13 @@ impl PolicyV3Runtime { } validate_root_liveness(&self.conn, &root, &root_cid, &self.node_did, now, true).await?; } - Ok(true) + Ok(Some( + owner_authority + .closure + .iter() + .map(|id| id.to_cid(0x55)) + .collect(), + )) } async fn is_policy_protected_parent( @@ -568,7 +581,14 @@ impl PolicyV3Runtime { &self, tinycloud: &crate::TinyCloud, start: tinycloud_auth::ipld_core::cid::Cid, - ) -> Result<(policy_v3_session::Model, Delegation), &'static str> { + ) -> Result< + ( + policy_v3_session::Model, + Delegation, + owner_authority::OwnerAuthority, + ), + &'static str, + > { let mut current_cid = start; let mut child: Option = None; let mut visited = std::collections::HashSet::new(); @@ -583,9 +603,12 @@ impl PolicyV3Runtime { else { return Err("policy-session-parent-missing"); }; - verify_signed_delegation(&event.0.delegation) + verify_delegation_signature(&event.0.delegation) .await .map_err(|_| "policy-session-signature-invalid")?; + if !delegation_time_valid(&event.0.delegation) { + return Err("policy-session-expired"); + } self.validate_graph_projection(&row, &event).await?; if !is_policy_session(&event.0) { return Err("policy-session-protected-fact-missing"); @@ -645,7 +668,8 @@ impl PolicyV3Runtime { { return Err("policy-session-proof-index-mismatch"); } - self.authorize_roots(®istration, OffsetDateTime::now_utc()) + let owner_authority = self + .authorize_roots(®istration, OffsetDateTime::now_utc()) .await?; for root_cid in event.0.parents.iter().copied() { let Some((_, root)) = tinycloud @@ -659,7 +683,7 @@ impl PolicyV3Runtime { return Err("policy-session-capability-widening"); } } - return Ok((index, event)); + return Ok((index, event, owner_authority)); } if event.0.parents.len() != 1 { return Err("policy-session-parent-count-invalid"); @@ -894,9 +918,22 @@ async fn validate_persisted_root( if fact(&event.0.delegation, "role") != Some(expected_role) { return Err("policy-root-role-invalid"); } - verify_signed_delegation(&event.0.delegation) + verify_delegation_signature(&event.0.delegation) .await .map_err(|_| "policy-root-signature-invalid")?; + if !delegation_time_valid(&event.0.delegation) { + return Err( + if event + .0 + .expiry + .is_some_and(|expiry| OffsetDateTime::now_utc() >= expiry) + { + "root-expired" + } else { + "root-not-live" + }, + ); + } for space in event.0.spaces() { tinycloud_core::request_telemetry::observe_space(space); } @@ -1469,6 +1506,10 @@ pub async fn authorize_delivery( .await .map_err(|_| (Status::Forbidden, "delivery-authorization-invalid".into()))?; } + runtime + .validate_owner_authority(&runtime.conn, ®istration, now) + .await + .map_err(|_| (Status::Forbidden, "delivery-authorization-invalid".into()))?; let resource = envelope .get("contentSource") .and_then(Value::as_object) @@ -2238,24 +2279,27 @@ pub async fn mint( )?; let enforcer_did = fact(&enforcement_root.0.delegation, "enforcerDid") .ok_or((Status::Forbidden, "sibling-root-mismatch".into()))?; - runtime + let owner_authority = runtime .authorize_roots(®istration, now) .await .map_err(|e| (Status::Forbidden, e.into()))?; - // Policies registered before owners had to hold what they grant are - // checked here too: the owner must have held every root capability when + // A policy served under the legacy escape has no recorded authority, so + // the owner must be shown to have held every root capability when // the policy was registered. (Owner sessions may since have expired.) - let registered_at = parse_time(®istration.registered_at) - .map_err(|_| (Status::Forbidden, "policy-registration-invalid".into()))?; - for root in [&policy_root, &enforcement_root] { - owner_holds_capabilities( - &runtime.conn, - ®istration.owner_did, - &root.0.capabilities, - registered_at, - ) - .await - .map_err(|error| (Status::Forbidden, error.into()))?; + // A recorded authority covers this check for other policies. + if owner_authority.escaped { + let registered_at = parse_time(®istration.registered_at) + .map_err(|_| (Status::Forbidden, "policy-registration-invalid".into()))?; + for root in [&policy_root, &enforcement_root] { + owner_holds_capabilities( + &runtime.conn, + ®istration.owner_did, + &root.0.capabilities, + registered_at, + ) + .await + .map_err(|error| (Status::Forbidden, error.into()))?; + } } for (cid, expected) in [ (®istration.policy_root_cid, &policy_root), @@ -2638,6 +2682,16 @@ pub async fn mint( validate_root_liveness(&txn, &root, root_cid, &runtime.node_did, now, true) .await .map_err(|error| (Status::Forbidden, error.into()))?; + // The recorded owner authority, rechecked on the locked transaction. + // Only a revoke landing between the earlier check and this + // transaction reaches it, so no test pins it (it is race-only). + if !tinycloud_core::policy_owner_authority::revoked_ids(&txn, &owner_authority.closure) + .await + .map_err(db_error)? + .is_empty() + { + return Err((Status::Forbidden, "policy-owner-authority-revoked".into())); + } let current_event = decode_delegation( std::str::from_utf8(&root.authorization_bytes) .map_err(|_| (Status::Forbidden, "policy-root-invalid".into()))?, @@ -2827,7 +2881,7 @@ impl PolicyV3Runtime { &self, registration: &policy_v3_registration::Model, now: OffsetDateTime, - ) -> Result<(), &'static str> { + ) -> Result { let policy = validate_registration_projection(registration) .map_err(|_| "policy-registration-projection-mismatch")?; let mut roots = Vec::new(); @@ -2880,11 +2934,10 @@ impl PolicyV3Runtime { now, ) .map_err(|_| "enforcer-binding-invalid")?; - // TC-1423: classify a registration made before owner-authority - // bindings existed, in the background. Recorded only; it does not - // gate this request or add work to it. - self.note_owner_binding(registration); - Ok(()) + // TC-1423: the owner authority the roots were registered under must + // hold. + self.validate_owner_authority(&self.conn, registration, now) + .await } } @@ -2959,13 +3012,12 @@ fn validate_stored_root_status( ) .map_err(|_| "root-status-invalid")?; let advertised_expiry = root_advertised_expiry(root)?; - if checked > now - || checked >= advertised_expiry - || fresh != advertised_expiry - || (require_fresh && advertised_expiry <= now) - { + if checked > now || checked >= advertised_expiry || fresh != advertised_expiry { return Err("root-not-live"); } + if require_fresh && advertised_expiry <= now { + return Err("root-expired"); + } let previous_digest = object .get("previousCheckpointDigestHex") .and_then(Value::as_str); @@ -4433,6 +4485,21 @@ fn decode_delegation(value: &str) -> Result Result<(), ()> { + let TinyCloudDelegation::Ucan(ucan) = delegation else { + return Err(()); + }; + tinycloud_core::did_resolution::verify_ucan_signature(ucan) + .await + .map_err(|_| ()) +} + +/// Whether a UCAN is within its signed validity window now. +fn delegation_time_valid(delegation: &TinyCloudDelegation) -> bool { + matches!(delegation, TinyCloudDelegation::Ucan(ucan) if ucan.payload().validate_time(None).is_ok()) +} + async fn verify_signed_delegation(delegation: &TinyCloudDelegation) -> Result<(), ()> { let TinyCloudDelegation::Ucan(ucan) = delegation else { return Err(()); @@ -8711,6 +8778,7 @@ mod tests { #[tokio::test] async fn policy_v2_admits_v3_account_and_v4_accountless_receivers() -> anyhow::Result<()> { + // Boxed: the flow's future is too large for a test thread's stack. Box::pin(run_policy_v2_flow(PolicyOwner::SpaceOwner)).await } @@ -8719,6 +8787,7 @@ mod tests { /// space owner revokes that delegation, no new session can be minted. #[tokio::test] async fn policy_owner_may_be_a_session_the_space_owner_delegated_to() -> anyhow::Result<()> { + // Boxed: the flow's future is too large for a test thread's stack. Box::pin(run_policy_v2_flow(PolicyOwner::DelegatedSession)).await } @@ -8727,6 +8796,7 @@ mod tests { /// Registration must refuse it, so no session can ever read the bytes. #[tokio::test] async fn policy_registration_requires_the_owner_to_hold_what_it_grants() -> anyhow::Result<()> { + // Boxed: the flow's future is too large for a test thread's stack. Box::pin(run_policy_v2_flow(PolicyOwner::Unauthorized)).await } @@ -10471,8 +10541,25 @@ mod tests { assert_eq!(replay.status(), Status::Unauthorized); if let Some(owner_delegation_cid) = owner_delegation_cid.as_deref() { - // Revoking the delegation the policy owner held stops new - // sessions, although the policy and its roots are untouched. + // Revoking the delegation the policy owner held ends the policy: + // the already-admitted session, new challenges and new sessions, + // although the policy and its roots are untouched. + // A challenge issued before the revoke, to try the mint with after it. + let pre_revoke_challenge = client + .post("/policy/v3/challenges") + .header(ContentType::JSON) + .body( + json!({ + "policyCid": policy_cid, + "recipientDid": holder_did, + "requestedCapabilities": requested, + }) + .to_string(), + ) + .dispatch() + .await; + assert_eq!(pre_revoke_challenge.status(), Status::Ok); + let fresh_challenge: Value = pre_revoke_challenge.into_json().await.unwrap(); let mut capabilities = Capabilities::::new(); capabilities.with_action( format!("urn:cid:{owner_delegation_cid}").parse()?, @@ -10510,6 +10597,37 @@ mod tests { "owner session revoke: {revoked_body}" ); + let post_owner_revoke_read = Payload { + issuer: reader_vm.parse()?, + audience: reader_did.parse()?, + not_before: Some(NumericDate::try_from_seconds( + OffsetDateTime::now_utc().unix_timestamp() as f64, + )?), + expiration: NumericDate::try_from_seconds( + (OffsetDateTime::now_utc() + Duration::seconds(30)).unix_timestamp() as f64, + )?, + nonce: Some("tc1423-read-after-owner-revoke".into()), + facts: Some(Vec::::new()), + proof: vec![child_cid], + attenuation: serde_json::from_value::>( + attenuation_for_policy_capabilities(&requested) + .map_err(|(_, error)| anyhow::anyhow!(error))?, + )?, + } + .sign(Algorithm::EdDSA, &reader_jwk)?; + let refused_read = client + .post("/invoke") + .header(rocket::http::Header::new( + "Authorization", + post_owner_revoke_read.encode()?, + )) + .dispatch() + .await; + assert_eq!(refused_read.status(), Status::Forbidden); + assert_eq!( + refused_read.into_string().await.unwrap_or_default(), + "policy-owner-authority-revoked" + ); let challenge_response = client .post("/policy/v3/challenges") .header(ContentType::JSON) @@ -10523,8 +10641,12 @@ mod tests { ) .dispatch() .await; - assert_eq!(challenge_response.status(), Status::Ok); - let fresh_challenge: Value = challenge_response.into_json().await.unwrap(); + assert_eq!(challenge_response.status(), Status::Forbidden); + assert_eq!( + challenge_response.into_string().await.unwrap_or_default(), + "policy-owner-authority-revoked" + ); + // A mint with the challenge issued before the revoke is refused too. let mut fresh = presentation.clone(); fresh["schema"] = json!(CREDENTIAL_PRESENTATION_V4_SCHEMA); fresh["jti"] = json!("presentation-tc-597-after-owner-revoke"); @@ -10569,7 +10691,38 @@ mod tests { Status::Forbidden, "mint after owner revoke: {refused_body}" ); - assert_eq!(refused_body, "policy-owner-not-authorized"); + assert_eq!(refused_body, "policy-owner-authority-revoked"); + // Delivery checks the same authority: the retry that returned the + // stored receipt above is refused. + let owner_revoked_delivery = make_invocation( + [( + content_resource.clone(), + ["tinycloud.kv/get".parse::()?], + )], + &sender_cid, + &delivery_jwk, + &format!("{holder_did}#{}", holder_did.trim_start_matches("did:key:")), + (OffsetDateTime::now_utc() + Duration::seconds(45)).unix_timestamp() as f64, + InvocationOptions { + nonce: Some("tc1423-delivery-after-owner-revoke".into()), + ..InvocationOptions::default() + }, + )?; + let refused_delivery = client + .post("/policy/v3/deliveries/authorize") + .header(ContentType::JSON) + .header(rocket::http::Header::new( + "Authorization", + owner_revoked_delivery.encode()?, + )) + .body(serde_json::to_string(&delivery_request)?) + .dispatch() + .await; + assert_eq!(refused_delivery.status(), Status::Forbidden); + assert_eq!( + refused_delivery.into_string().await.unwrap_or_default(), + "delivery-authorization-invalid" + ); } // SDK revocation writes the generic graph. Policy/v3 joins that diff --git a/tinycloud-node-server/src/policy_v3/owner_authority.rs b/tinycloud-node-server/src/policy_v3/owner_authority.rs index bcd2605a..b423dbb9 100644 --- a/tinycloud-node-server/src/policy_v3/owner_authority.rs +++ b/tinycloud-node-server/src/policy_v3/owner_authority.rs @@ -51,14 +51,13 @@ const OWNER_FIRST_ROWS: u64 = 64; const LISTING_PAGE: u64 = 100; /// How long a background classification waits for the SQLite writer. const LAZY_WRITE_LOCK_WAIT: std::time::Duration = std::time::Duration::from_millis(250); -/// After a classification that could not be stored, or that ended -/// `starved`, the request path doesn't start another one for this long. +/// A classification computed for a request that isn't final (its write +/// failed, or it ended `starved`) is reused for this long before it is +/// recomputed. const RETRY_BACKOFF: std::time::Duration = std::time::Duration::from_secs(60); /// Bound on the process-local sets below; a full set is cleared. const MAX_TRACKED_POLICIES: usize = 65_536; -/// Background classifications running at once, process-wide. A first use -/// that finds no free slot starts none (and doesn't back off); a later use -/// retries. +/// Background classification writes running at once, process-wide. const MAX_BACKGROUND_CLASSIFICATIONS: usize = 3; #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -169,6 +168,46 @@ impl RegisteredMode { Self::Backfill => "backfill", } } + + fn parse(value: &str) -> Option { + Some(match value { + "record" => Self::Record, + "enforce" => Self::Enforce, + "backfill" => Self::Backfill, + _ => return None, + }) + } +} + +/// A classification as a policy use checks it. +#[derive(Clone, Debug)] +pub(super) struct Binding { + state: BindingState, + reason: Option, + mode: RegisteredMode, + authority_expires_at: Option, + /// The recorded closure (empty unless `bound*`). + closure: Arc<[Hash]>, +} + +impl Binding { + fn from_classification(classification: &Classification, mode: RegisteredMode) -> Self { + Self { + state: classification.state, + reason: classification.reason, + mode, + authority_expires_at: classification.authority_expires_at, + closure: classification.closure.iter().map(|(id, _)| *id).collect(), + } + } +} + +/// The owner authority a policy use passed: the closure to watch for +/// revocation, and whether it is served under the legacy escape. +#[derive(Clone, Debug, Default)] +pub struct OwnerAuthority { + pub closure: Arc<[Hash]>, + pub escaped: bool, } /// How a registration's owner held what its roots grant. @@ -696,59 +735,116 @@ impl CutoverSource { pub(super) struct OwnerAuthorityState { configured_cutover: Option, expiry: OwnerAuthorityExpiry, + legacy_unknown_allow: bool, migration_cutover: tokio::sync::OnceCell, - /// Policies whose stored classification is final. - final_policies: std::sync::Mutex>, - /// Policies with a background classification running. + /// Policies whose stored classification is final, with their closure. + /// Liveness is never cached, only what to check. + final_policies: std::sync::Mutex>, + /// Classifications computed for a request but not (yet) final, served + /// until the instant given, so a failed write or `starved` is not + /// recomputed on every request. + transient: std::sync::Mutex>, + /// Policies with a background write running. in_flight: std::sync::Mutex>, - /// Slots for background classifications. + /// Slots for background writes. background: BackgroundSlots, - /// Policies whose last classification is not final, until when. - retry_after: std::sync::Mutex>, + /// The last log time of each escaped policy. + escape_logged: std::sync::Mutex>, + /// Request-path classifications running, one per policy; concurrent uses + /// of the policy wait for it and share its result. + classifying: std::sync::Mutex>>>, + /// Classifications computed (tests count them). + #[cfg(test)] + pub(super) classifications: std::sync::atomic::AtomicUsize, +} + +type ClassifyResult = Result<(Classification, RegisteredMode), &'static str>; + +/// Where a use's classification came from. +enum Classified { + /// Remembered from a classification that finished meanwhile. + InMemory(Binding), + /// Computed (or awaited) by this use. + Computed(Classification, RegisteredMode), +} + +/// Insert into a bounded map, clearing it when full. +fn bounded_insert(map: &mut HashMap, key: &str, value: V) { + if map.len() >= MAX_TRACKED_POLICIES && !map.contains_key(key) { + map.clear(); + } + map.insert(key.to_owned(), value); } impl OwnerAuthorityState { - fn is_final(&self, policy_cid: &str) -> bool { - self.final_policies + fn cached(&self, policy_cid: &str) -> Option { + if let Some(binding) = self + .final_policies .lock() .expect("owner-authority cache") - .contains_key(policy_cid) - } - - fn remember_final(&self, policy_cid: &str, state: BindingState) { - let mut cache = self.final_policies.lock().expect("owner-authority cache"); - if cache.len() >= MAX_TRACKED_POLICIES { - cache.clear(); + .get(policy_cid) + { + return Some(binding.clone()); } - cache.insert(policy_cid.to_owned(), state); - } - - fn backing_off(&self, policy_cid: &str) -> bool { - let mut retry = self.retry_after.lock().expect("owner-authority backoff"); - match retry.get(policy_cid) { - Some(until) if *until > tokio::time::Instant::now() => true, + let mut transient = self.transient.lock().expect("owner-authority cache"); + match transient.get(policy_cid) { + Some((binding, until)) if *until > tokio::time::Instant::now() => Some(binding.clone()), Some(_) => { - retry.remove(policy_cid); - false + transient.remove(policy_cid); + None } - None => false, + None => None, } } - fn back_off(&self, policy_cid: &str) { - let mut retry = self.retry_after.lock().expect("owner-authority backoff"); - if retry.len() >= MAX_TRACKED_POLICIES { - let now = tokio::time::Instant::now(); - retry.retain(|_, until| *until > now); - if retry.len() >= MAX_TRACKED_POLICIES { - retry.clear(); - } - } - retry.insert( - policy_cid.to_owned(), - tokio::time::Instant::now() + RETRY_BACKOFF, + fn remember_final(&self, policy_cid: &str, binding: Binding) { + self.transient + .lock() + .expect("owner-authority cache") + .remove(policy_cid); + bounded_insert( + &mut self.final_policies.lock().expect("owner-authority cache"), + policy_cid, + binding, + ); + } + + fn remember_transient(&self, policy_cid: &str, binding: Binding) { + bounded_insert( + &mut self.transient.lock().expect("owner-authority cache"), + policy_cid, + (binding, tokio::time::Instant::now() + RETRY_BACKOFF), ); } + + /// Whether this escaped use should be logged (once per policy per hour). + fn log_escape(&self, policy_cid: &str) -> bool { + let mut logged = self + .escape_logged + .lock() + .expect("owner-authority escape log"); + let now = tokio::time::Instant::now(); + if logged + .get(policy_cid) + .is_some_and(|at| now.duration_since(*at) < std::time::Duration::from_secs(3600)) + { + return false; + } + bounded_insert(&mut logged, policy_cid, now); + true + } +} + +/// The recorded closure of a policy. +async fn recorded_closure( + conn: &C, + policy_cid: &str, +) -> Result, DbErr> { + policy_v3_owner_authority::Entity::find() + .filter(policy_v3_owner_authority::Column::PolicyCid.eq(policy_cid)) + .all(conn) + .await + .map(|rows| rows.into_iter().map(|row| row.delegation_id).collect()) } /// The process-wide limit on background classifications. @@ -831,6 +927,7 @@ impl PolicyV3Runtime { self.owner_authority = Arc::new(OwnerAuthorityState { configured_cutover: config.legacy_cutover(), expiry: config.owner_authority_expiry, + legacy_unknown_allow: config.legacy_unknown_allow, ..OwnerAuthorityState::default() }); self @@ -898,11 +995,17 @@ impl PolicyV3Runtime { "policy owner-authority legacy cutover is unavailable; legacy policies are classified when it can be read" ), } + if self.owner_authority.legacy_unknown_allow { + ::tracing::warn!( + event = "policy_owner_authority", + legacy_unknown_allow = true, + "policy_v3.legacy_unknown_allow is on: legacy policies without owner authority are served with root-only checks" + ); + } } - /// The stored classification of `registration`, classifying it first if - /// none is stored (or a stored `unknown` came from an older classifier). - /// A classification that cannot be stored is returned, unstored. + /// The stored classification of `registration`, classifying (and + /// storing) it first if none is final. Used by the sweep. pub(super) async fn owner_binding( &self, registration: &policy_v3_registration::Model, @@ -912,18 +1015,27 @@ impl PolicyV3Runtime { .await .map_err(|_| "policy-owner-authority-unavailable")?; let stale = match &existing { - Some(row) if row_is_final(row) => { - let classification = Classification::from_row(row)?; - self.owner_authority - .remember_final(®istration.policy_cid, classification.state); - return Ok(classification); - } + Some(row) if row_is_final(row) => return Classification::from_row(row), Some(_) => true, None => false, }; let (classification, mode) = self.classify_registration(registration).await?; + Ok(self + .store_classification(®istration.policy_cid, classification, mode, stale) + .await) + } + + /// Store a computed classification claim-first, count it, and remember + /// it when final. Returns what is in force. + async fn store_classification( + &self, + policy_cid: &str, + classification: Classification, + mode: RegisteredMode, + stale: bool, + ) -> Classification { match self - .store_lazy_classification(®istration.policy_cid, &classification, mode, stale) + .store_lazy_classification(policy_cid, &classification, mode, stale) .await { LazyWrite::Written => { @@ -940,55 +1052,213 @@ impl PolicyV3Runtime { ..classification }; if classification.is_final() { - self.owner_authority - .remember_final(®istration.policy_cid, classification.state); + self.owner_authority.remember_final( + policy_cid, + Binding::from_classification(&classification, mode), + ); } - Ok(classification) + classification } + // The winner's closure is read on the next use. LazyWrite::Lost(winner) => { - let classification = Classification::from_row(&winner)?; - self.owner_authority - .remember_final(®istration.policy_cid, classification.state); - Ok(classification) + Classification::from_row(&winner).unwrap_or(Classification { + persisted: true, + ..classification + }) } LazyWrite::Failed => { crate::prometheus::observe_policy_owner_binding_backfill("write_failed_served"); - Ok(classification) + classification } } } - /// Request path: make sure `registration` gets classified, without - /// waiting for it. A policy whose stored classification is final costs - /// nothing; otherwise one background task per policy (and at most - /// [`MAX_BACKGROUND_CLASSIFICATIONS`] in all) classifies it, and a result - /// that is not final (a failed write, `starved`) holds off the next - /// attempt for [`RETRY_BACKOFF`]. - pub(super) fn note_owner_binding( + /// What a use of `registration` checks. A final stored classification is + /// loaded once (row and closure) and remembered. A policy without one is + /// classified in the request, served from that result, and stored in the + /// background; a result that isn't final is reused for + /// [`RETRY_BACKOFF`] rather than recomputed. + async fn binding_for_use( &self, registration: &policy_v3_registration::Model, - ) -> Option> { + ) -> Result { let state = &self.owner_authority; let policy_cid = ®istration.policy_cid; - if state.is_final(policy_cid) || state.backing_off(policy_cid) { - return None; + if let Some(binding) = state.cached(policy_cid) { + return Ok(binding); } - let claim = InFlight::claim(state, policy_cid)?; - let runtime = self.clone(); - let registration = registration.clone(); - Some(tokio::spawn(async move { - let _claim = claim; - let result = runtime.owner_binding(®istration).await; - if !result.is_ok_and(|classification| classification.is_final()) { - runtime.owner_authority.back_off(®istration.policy_cid); + let existing = policy_v3_owner_binding::Entity::find_by_id(policy_cid.clone()) + .one(&self.conn) + .await + .map_err(|_| "policy-owner-authority-unavailable")?; + if let Some(row) = existing.as_ref().filter(|row| row_is_final(row)) { + let classification = Classification::from_row(row)?; + let closure = match classification.state { + BindingState::Bound | BindingState::BoundUncapped => { + recorded_closure(&self.conn, policy_cid) + .await + .map_err(|_| "policy-owner-authority-unavailable")? + } + BindingState::RootAuthority | BindingState::Unknown => Vec::new(), + }; + let binding = Binding { + state: classification.state, + reason: classification.reason, + mode: RegisteredMode::parse(&row.registered_mode) + .ok_or("policy-owner-binding-invalid")?, + authority_expires_at: classification.authority_expires_at, + closure: closure.into(), + }; + state.remember_final(policy_cid, binding.clone()); + return Ok(binding); + } + let (classification, mode) = match self.classify_once(registration).await? { + Classified::InMemory(binding) => return Ok(binding), + Classified::Computed(classification, mode) => (classification, mode), + }; + let binding = Binding::from_classification(&classification, mode); + // The write runs in the background (one per policy, a few at once). + // Without a free slot this request uses its result; the write + // is retried once the in-memory result expires. + if let Some(claim) = InFlight::claim(state, policy_cid) { + let runtime = self.clone(); + let policy_cid = policy_cid.clone(); + let stale = existing.is_some(); + tokio::spawn(async move { + let _claim = claim; + runtime + .store_classification(&policy_cid, classification, mode, stale) + .await; + }); + } + Ok(binding) + } + + /// Classify a policy for a use, one computation at a time per policy: + /// concurrent uses wait for it and share its result. A result is + /// remembered in memory before the computation's entry is dropped, and a + /// use that finds no entry checks memory under the same lock, so a use + /// arriving just after a computation reuses it rather than starting + /// another. No lock is held across an await; if the computing request is + /// cancelled, a waiting one takes over. + async fn classify_once( + &self, + registration: &policy_v3_registration::Model, + ) -> Result { + let state = &self.owner_authority; + let policy_cid = ®istration.policy_cid; + let cell = { + let mut classifying = state + .classifying + .lock() + .expect("owner-authority classifications"); + match classifying.get(policy_cid) { + Some(cell) => cell.clone(), + None => { + if let Some(binding) = state.cached(policy_cid) { + return Ok(Classified::InMemory(binding)); + } + classifying.entry(policy_cid.clone()).or_default().clone() + } + } + }; + let result = cell + .get_or_init(|| async { + let result = self.classify_registration(registration).await; + if let Ok((classification, mode)) = &result { + state.remember_transient( + policy_cid, + Binding::from_classification(classification, *mode), + ); + } + result + }) + .await + .clone(); + let mut classifying = state + .classifying + .lock() + .expect("owner-authority classifications"); + if classifying + .get(policy_cid) + .is_some_and(|current| Arc::ptr_eq(current, &cell)) + { + classifying.remove(policy_cid); + } + result.map(|(classification, mode)| Classified::Computed(classification, mode)) + } + + /// Check a policy use against its recorded owner authority: every + /// recorded delegation and ancestor must be unrevoked (as of `conn`, the + /// caller's transaction where it has one), and a `bound` authority + /// unexpired. Legacy registrations are never expiry-checked. A policy + /// without authority is denied, or served with root-only checks under + /// `policy_v3.legacy_unknown_allow` when it is legacy. + pub(super) async fn validate_owner_authority( + &self, + conn: &C, + registration: &policy_v3_registration::Model, + now: OffsetDateTime, + ) -> Result { + let binding = self.binding_for_use(registration).await?; + match binding.state { + BindingState::RootAuthority => Ok(OwnerAuthority::default()), + BindingState::Unknown => { + let reason = binding.reason.map_or("missing", UnknownReason::as_str); + if binding.mode == RegisteredMode::Backfill + && self.owner_authority.legacy_unknown_allow + { + crate::prometheus::observe_policy_owner_authority_unknown_allowed(reason); + if self.owner_authority.log_escape(®istration.policy_cid) { + ::tracing::warn!( + event = "policy_owner_authority", + outcome = "unknown_allowed", + policy_cid = %registration.policy_cid, + reason, + "legacy policy without owner authority served under policy_v3.legacy_unknown_allow" + ); + } + return Ok(OwnerAuthority { + closure: Arc::from([]), + escaped: true, + }); + } + crate::prometheus::observe_policy_owner_authority_denied("unknown"); + Err("policy-owner-authority-unknown") + } + BindingState::Bound | BindingState::BoundUncapped => { + let revoked = authority_graph::revoked_ids(conn, &binding.closure) + .await + .map_err(|_| "policy-owner-authority-unavailable")?; + if !revoked.is_empty() { + crate::prometheus::observe_policy_owner_authority_denied("revoked"); + return Err("policy-owner-authority-revoked"); + } + if binding.state == BindingState::Bound + && binding.mode != RegisteredMode::Backfill + && binding + .authority_expires_at + .is_some_and(|expiry| now >= expiry) + { + crate::prometheus::observe_policy_owner_authority_denied("expired"); + return Err("policy-owner-authority-expired"); + } + Ok(OwnerAuthority { + closure: binding.closure, + escaped: false, + }) } - })) + } } async fn classify_registration( &self, registration: &policy_v3_registration::Model, - ) -> Result<(Classification, RegisteredMode), &'static str> { + ) -> ClassifyResult { + #[cfg(test)] + self.owner_authority + .classifications + .fetch_add(1, std::sync::atomic::Ordering::Relaxed); let registered_at = parse_time(®istration.registered_at).map_err(|_| "policy-registration-invalid")?; let (cutover, _) = self @@ -1361,6 +1631,24 @@ pub async fn authorized_policies( let generically_revoked = authority_graph::revoked_ids(conn, &root_hashes) .await .map_err(listing_unavailable)?; + // Live owner-authority state: a recorded closure with a revoked member + // is `revoked`; a non-legacy `bound` authority past its expiry `expired`. + let mut closures: HashMap> = HashMap::new(); + for row in policy_v3_owner_authority::Entity::find() + .filter(policy_v3_owner_authority::Column::PolicyCid.is_in(policy_cids.clone())) + .all(conn) + .await + .map_err(listing_unavailable)? + { + closures + .entry(row.policy_cid) + .or_default() + .push(row.delegation_id); + } + let closure_ids = closures.values().flatten().copied().collect::>(); + let revoked_authority = authority_graph::revoked_ids(conn, &closure_ids) + .await + .map_err(listing_unavailable)?; let now = OffsetDateTime::now_utc(); let mut policies = Vec::with_capacity(policy_cids.len()); for policy_cid in &policy_cids { @@ -1387,7 +1675,24 @@ pub async fn authorized_policies( "rootState": if revoked { "revoked" } else if expired { "expired" } else { "active" }, "ownerAuthorityState": bindings .get(policy_cid) - .map(|binding| binding.state.as_str()) + .map(|binding| { + if closures.get(policy_cid).is_some_and(|closure| { + closure.iter().any(|id| revoked_authority.contains(id)) + }) { + "revoked" + } else if binding.state == BindingState::Bound.as_str() + && binding.registered_mode != RegisteredMode::Backfill.as_str() + && binding + .authority_expires_at + .as_deref() + .and_then(|value| parse_time(value).ok()) + .is_some_and(|expiry| expiry <= now) + { + "expired" + } else { + binding.state.as_str() + } + }) .unwrap_or("pending"), })); } @@ -1458,64 +1763,19 @@ mod tests { .unwrap() } - /// A policy with a final stored classification adds no query to the - /// request path once seen; the first sighting only spawns. - #[tokio::test] - async fn final_classifications_cost_the_request_path_nothing() { - let (runtime, queries) = counted_runtime().await; - let registration = registration(&runtime, "policy-final").await; - let classification = - Classification::from_selection(Ok(Vec::new()), OffsetDateTime::now_utc()); + async fn store(runtime: &PolicyV3Runtime, policy_cid: &str, classification: &Classification) { insert_classification( &runtime.conn, - ®istration.policy_cid, - &classification, + policy_cid, + classification, RegisteredMode::Record, OffsetDateTime::now_utc(), ) .await .unwrap(); - - let before = queries.load(Ordering::SeqCst); - let first = runtime.note_owner_binding(®istration).expect("spawned"); - assert_eq!(queries.load(Ordering::SeqCst), before, "spawn only"); - assert!( - runtime.note_owner_binding(®istration).is_none(), - "deduplicated" - ); - first.await.unwrap(); - - let before = queries.load(Ordering::SeqCst); - for _ in 0..10 { - assert!(runtime.note_owner_binding(®istration).is_none()); - } - tokio::task::yield_now().await; - assert_eq!(queries.load(Ordering::SeqCst), before); - } - - /// At most a few classifications run in the background at once; a use - /// that finds no free slot starts none and doesn't back off. - #[tokio::test] - async fn background_classifications_are_limited_without_backing_off() { - let (runtime, _) = counted_runtime().await; - let registration = registration(&runtime, "policy-busy").await; - let held = (0..MAX_BACKGROUND_CLASSIFICATIONS) - .map(|_| { - Arc::clone(&runtime.owner_authority.background.0) - .try_acquire_owned() - .unwrap() - }) - .collect::>(); - assert!(runtime.note_owner_binding(®istration).is_none()); - assert!(runtime.owner_authority.in_flight.lock().unwrap().is_empty()); - drop(held); - assert!( - runtime.note_owner_binding(®istration).is_some(), - "no back-off after a busy skip" - ); } - /// A background classification that panics releases its policy and + /// A background write that panics releases its policy and /// its slot. #[tokio::test] async fn a_panicking_classification_releases_its_claim() { @@ -1534,24 +1794,125 @@ mod tests { ); } - /// A classification that is not final (here it cannot even be computed) - /// is not retried on every request. + /// Once a final classification is loaded, a use costs one revocation + /// query for a recorded closure and nothing for root authority. #[tokio::test] - async fn non_final_classifications_back_off() { + async fn steady_state_checks_cost_one_query_or_none() { let (runtime, queries) = counted_runtime().await; - // No root row: classification fails. - let registration = registration(&runtime, "policy-unclassifiable").await; - runtime - .note_owner_binding(®istration) - .expect("spawned") + let now = OffsetDateTime::now_utc(); + let root_authority = registration(&runtime, "policy-root-authority").await; + store( + &runtime, + &root_authority.policy_cid, + &Classification::from_selection(Ok(Vec::new()), now), + ) + .await; + let bound = registration(&runtime, "policy-bound").await; + let closure = vec![ClosureNode { + id: tinycloud_core::hash::hash(b"grant"), + depth: 0, + delegator: "did:key:z6MkGrantor".into(), + expiry: None, + not_before: None, + }]; + store( + &runtime, + &bound.policy_cid, + &Classification::from_selection(Ok(closure), now), + ) + .await; + for (registration, steady_queries) in [(&root_authority, 0), (&bound, 1)] { + runtime + .validate_owner_authority(&runtime.conn, registration, now) + .await + .unwrap(); + for _ in 0..5 { + let before = queries.load(Ordering::SeqCst); + runtime + .validate_owner_authority(&runtime.conn, registration, now) + .await + .unwrap(); + assert_eq!( + queries.load(Ordering::SeqCst) - before, + steady_queries, + "{}", + registration.policy_cid + ); + } + } + } + + /// An `unknown` legacy classification is denied with a fixed code, or + /// served (escaped) when the operator allows it. + #[tokio::test] + async fn unknown_legacy_classifications_are_denied_unless_allowed() { + for allow in [false, true] { + let (runtime, _) = counted_runtime().await; + let runtime = runtime.with_policy_config(&crate::config::PolicyV3Config { + legacy_unknown_allow: allow, + ..Default::default() + }); + let registration = registration(&runtime, "policy-unknown").await; + insert_classification( + &runtime.conn, + ®istration.policy_cid, + &Classification::unknown(UnknownReason::Missing), + RegisteredMode::Backfill, + OffsetDateTime::now_utc(), + ) .await .unwrap(); - let before = queries.load(Ordering::SeqCst); - for _ in 0..10 { - assert!(runtime.note_owner_binding(®istration).is_none()); + let result = runtime + .validate_owner_authority(&runtime.conn, ®istration, OffsetDateTime::now_utc()) + .await; + if allow { + assert!(result.unwrap().escaped); + } else { + assert_eq!(result.unwrap_err(), "policy-owner-authority-unknown"); + } + } + } + + /// A `bound` (non-legacy) authority that has expired is denied with the + /// expiry code; a legacy one is not expiry-checked. + #[tokio::test] + async fn expired_bound_authority_is_denied_except_legacy() { + for (mode, expected) in [ + ( + RegisteredMode::Record, + Err("policy-owner-authority-expired"), + ), + (RegisteredMode::Backfill, Ok(())), + ] { + let (runtime, _) = counted_runtime().await; + let registration = registration(&runtime, "policy-expiring").await; + let expiry = OffsetDateTime::now_utc() + Duration::seconds(5); + let closure = vec![ClosureNode { + id: tinycloud_core::hash::hash(b"short-grant"), + depth: 0, + delegator: "did:key:z6MkGrantor".into(), + expiry: Some(expiry), + not_before: None, + }]; + insert_classification( + &runtime.conn, + ®istration.policy_cid, + &Classification::from_selection(Ok(closure), expiry), + mode, + OffsetDateTime::now_utc(), + ) + .await + .unwrap(); + let result = runtime + .validate_owner_authority( + &runtime.conn, + ®istration, + expiry + Duration::seconds(1), + ) + .await + .map(|_| ()); + assert_eq!(result, expected, "{mode:?}"); } - tokio::task::yield_now().await; - assert_eq!(queries.load(Ordering::SeqCst), before); } /// The cap is exact: registration + 31 days + 5 minutes is accepted, one @@ -1737,4 +2098,135 @@ mod tests { assert_eq!(UnknownReason::parse(reason.as_str()), Some(reason)); } } + + /// A stored registration over another owner's network: classifying it + /// needs no root rows. + fn foreign_registration() -> policy_v3_registration::Model { + use super::super::owner_authority_tests::Signer; + let space_owner = Signer::key(); + let policy_owner = Signer::key(); + stored_registration( + &policy_owner, + &content_in_space_of(space_owner.did()), + &format!("urn:tinycloud:encryption:{}:share", policy_owner.did()), + OffsetDateTime::now_utc() - Duration::hours(1), + ) + } + + /// Holds every background slot until dropped. + fn hold_every_slot(state: &Arc) -> Vec { + (0..MAX_BACKGROUND_CLASSIFICATIONS) + .map(|index| InFlight::claim(state, &format!("policy-held-{index}")).expect("a slot")) + .collect() + } + + /// A policy with a final stored classification adds no query to the + /// request path once its row is loaded. + #[tokio::test] + async fn final_classifications_cost_the_request_path_nothing() { + let (runtime, queries) = counted_runtime().await; + let registration = foreign_registration(); + insert_classification( + &runtime.conn, + ®istration.policy_cid, + &Classification::foreign_network(), + RegisteredMode::Backfill, + OffsetDateTime::now_utc(), + ) + .await + .unwrap(); + runtime.binding_for_use(®istration).await.unwrap(); + let before = queries.load(Ordering::SeqCst); + for _ in 0..10 { + runtime.binding_for_use(®istration).await.unwrap(); + } + assert_eq!(queries.load(Ordering::SeqCst), before); + assert_eq!( + runtime + .owner_authority + .classifications + .load(Ordering::SeqCst), + 0, + "a stored final row is not classified again" + ); + } + + /// At most `MAX_BACKGROUND_CLASSIFICATIONS` background writes run at + /// once. A use that finds no free slot is served and starts none; + /// once a slot is free, a use of another policy stores its result. + #[tokio::test] + async fn background_classifications_are_limited_without_backing_off() { + let (runtime, _) = counted_runtime().await; + let state = &runtime.owner_authority; + // Three background writes may run at once; a fourth finds no slot. + let held = (0..3) + .map(|index| { + InFlight::claim(state, &format!("policy-held-{index}")).expect("one of three slots") + }) + .collect::>(); + assert!( + InFlight::claim(state, "policy-over-the-limit").is_none(), + "no slot beyond the third" + ); + let busy = foreign_registration(); + let binding = runtime.binding_for_use(&busy).await.unwrap(); + assert_eq!(binding.reason, Some(UnknownReason::ForeignNetwork)); + tokio::task::yield_now().await; + assert!( + policy_v3_owner_binding::Entity::find_by_id(busy.policy_cid.clone()) + .one(&runtime.conn) + .await + .unwrap() + .is_none(), + "nothing is stored while every slot is held" + ); + drop(held); + let free = foreign_registration(); + runtime.binding_for_use(&free).await.unwrap(); + let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(10); + loop { + let stored = policy_v3_owner_binding::Entity::find_by_id(free.policy_cid.clone()) + .one(&runtime.conn) + .await + .unwrap(); + if stored.is_some() { + break; + } + assert!( + tokio::time::Instant::now() < deadline, + "the background write did not run" + ); + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + } + } + + /// A classification that is not final (here it could not be stored, as + /// every slot is held) is reused for the back-off window: later uses + /// compute nothing and query nothing. + #[tokio::test] + async fn non_final_classifications_back_off() { + let (runtime, queries) = counted_runtime().await; + let _held = hold_every_slot(&runtime.owner_authority); + let registration = foreign_registration(); + runtime.binding_for_use(®istration).await.unwrap(); + assert_eq!( + runtime + .owner_authority + .classifications + .load(Ordering::SeqCst), + 1 + ); + let before = queries.load(Ordering::SeqCst); + for _ in 0..10 { + runtime.binding_for_use(®istration).await.unwrap(); + } + assert_eq!(queries.load(Ordering::SeqCst), before); + assert_eq!( + runtime + .owner_authority + .classifications + .load(Ordering::SeqCst), + 1 + ); + } } diff --git a/tinycloud-node-server/src/policy_v3/owner_authority_tests.rs b/tinycloud-node-server/src/policy_v3/owner_authority_tests.rs index 326b1f32..bbd8854f 100644 --- a/tinycloud-node-server/src/policy_v3/owner_authority_tests.rs +++ b/tinycloud-node-server/src/policy_v3/owner_authority_tests.rs @@ -364,13 +364,23 @@ impl Signer { /// An encoded invocation of `attenuation` citing `proof`. pub fn invoke(&self, attenuation: Capabilities, proof: Cid) -> anyhow::Result { - let Self::Key { jwk, vm, did, .. } = self else { + self.invoke_for(self.did(), attenuation, proof) + } + + /// An encoded invocation addressed to `audience`. + pub fn invoke_for( + &self, + audience: &str, + attenuation: Capabilities, + proof: Cid, + ) -> anyhow::Result { + let Self::Key { jwk, vm, .. } = self else { anyhow::bail!("invocations are signed by keys"); }; let at = now(); Ok(Payload { issuer: vm.parse()?, - audience: did.parse()?, + audience: audience.parse()?, not_before: Some(seconds(at)?), expiration: seconds(at + Duration::seconds(30))?, nonce: Some(unique("tc1423invoke")), @@ -429,7 +439,9 @@ pub(super) fn signed_policy( "policyId": "pending", "ownerDid": owner_did, "createdAt": format_time(created), - "expiresAt": format_time(root.expiry), + // The policy may outlive its roots; the roots' expiry is what the + // tests exercise. + "expiresAt": format_time(root.expiry + Duration::hours(1)), "contentSource": { "shareId": unique("share-tc-1423-"), "kvResource": content_resource, @@ -490,6 +502,8 @@ pub(super) struct Harness { pub(super) struct Boot { /// The `[policy_v3]` section, passed through figment. pub policy_v3: Option, + /// The `[hooks]` section, passed through figment. + pub hooks: Option, } impl Harness { @@ -606,6 +620,9 @@ impl Harness { if let Some(policy_v3) = &boot.policy_v3 { figment = figment.merge(Serialized::default("policy_v3", policy_v3)); } + if let Some(hooks) = &boot.hooks { + figment = figment.merge(Serialized::default("hooks", hooks)); + } let config: crate::config::Config = figment.extract()?; Ok(Client::tracked(crate::app_with_control(&figment, &config, None).await?).await?) } @@ -732,12 +749,24 @@ impl Harness { } fn requested(&self) -> Vec { - vec![json!({ + vec![self.requested_kv(), self.requested_decrypt()] + } + + fn requested_kv(&self) -> Value { + json!({ "kind": "kv", "resource": self.content_resource.to_string(), "selector": "exact", "actions": [KV_GET], - })] + }) + } + + fn requested_decrypt(&self) -> Value { + json!({ + "kind": "encryption", + "resource": self.network, + "action": DECRYPT, + }) } /// Register a policy over the content whose roots span `root`, signed @@ -1032,7 +1061,7 @@ impl Harness { let s0 = decode_delegation(session).map_err(|(_, error)| anyhow::anyhow!(error))?; let invocation = self.holder.invoke( serde_json::from_value( - attenuation_for_policy_capabilities(&self.requested()) + attenuation_for_policy_capabilities(&[self.requested_kv()]) .map_err(|(_, error)| anyhow::anyhow!(error))?, )?, s0.content_hash().to_cid(0x55), @@ -1179,14 +1208,24 @@ impl Harness { Ok(()) } - /// Use the policy (a challenge) and wait for its background - /// classification. + /// Use the policy (a challenge), wait for its background classification, + /// and check that the use is admitted when the policy has authority + /// (`bound`, `bound-uncapped`, `root-authority`: 200) and refused with + /// the fixed code when it has none (`unknown`: 403). pub async fn classify_on_use(&self, policy: &Registered) -> anyhow::Result { let (status, body) = self.challenge(policy).await; - anyhow::ensure!(status == Status::Ok, "challenge: {status} {body}"); - self.wait_for_binding(&policy.policy_cid, std::time::Duration::from_secs(20)) + let row = self + .wait_for_binding(&policy.policy_cid, std::time::Duration::from_secs(20)) .await? - .ok_or_else(|| anyhow::anyhow!("not classified")) + .ok_or_else(|| anyhow::anyhow!("not classified"))?; + match row.state.as_str() { + "unknown" => anyhow::ensure!( + status == Status::Forbidden && body == "policy-owner-authority-unknown", + "challenge on an unknown policy: {status} {body}" + ), + _ => anyhow::ensure!(status == Status::Ok, "challenge: {status} {body}"), + } + Ok(row) } /// Wait until `policy_cid` is classified as `state`, or time out; @@ -1239,6 +1278,7 @@ fn legacy_boot() -> Boot { policy_v3: Some(json!({ "legacy_cutover": format_time(now() + Duration::days(1)), })), + ..Boot::default() } } @@ -2080,6 +2120,7 @@ async fn rollback_and_reupgrade_respect_the_configured_cutover( ); h.reboot(Boot { policy_v3: Some(json!({"legacy_cutover": format_time(cutover)})), + ..Boot::default() }) .await?; assert_eq!( @@ -2236,6 +2277,7 @@ async fn insert_synthetic( fn boot(policy_v3: Value) -> Boot { Boot { policy_v3: Some(policy_v3), + ..Boot::default() } } @@ -2558,3 +2600,1097 @@ async fn post_cutover_rows_outliving_their_authority_are_unknown_under_enforce( Ok(()) } on_both_backends!(post_cutover_rows_outliving_their_authority_are_unknown_under_enforce); + +// --------------------------------------------------------------------------- +// The recorded owner authority is checked whenever a policy is used. +// --------------------------------------------------------------------------- + +const REVOKED: &str = "policy-owner-authority-revoked"; + +/// A recipient body must not name any CID or DID. +fn assert_opaque(label: &str, body: &str) { + assert!( + !body.contains("bafy") && !body.contains("bafk") && !body.contains("did:"), + "{label} leaks an identifier: {body}" + ); +} + +fn session_cid(session: &str) -> anyhow::Result { + Ok(decode_delegation(session) + .map_err(|(_, error)| anyhow::anyhow!(error))? + .content_hash() + .to_cid(0x55)) +} + +fn text(bytes: &[u8]) -> String { + String::from_utf8_lossy(bytes).into_owned() +} + +impl Harness { + /// The holder decrypts through the network route under a session. + pub async fn decrypt(&self, session: &str) -> anyhow::Result<(Status, String)> { + let invocation = self.holder.invoke_for( + &self.node_did(), + serde_json::from_value( + attenuation_for_policy_capabilities(&[self.requested_decrypt()]) + .map_err(|(_, error)| anyhow::anyhow!(error))?, + )?, + session_cid(session)?, + )?; + let response = self + .client + .post(format!("/encryption/networks/{}/decrypt", self.network)) + .header(ContentType::JSON) + .header(Header::new("Authorization", invocation)) + .body("{}") + .dispatch() + .await; + let status = response.status(); + Ok((status, response.into_string().await.unwrap_or_default())) + } + + /// The holder re-shares a session to a fresh reader; returns the reader + /// and the child CID. + pub async fn reshare( + &self, + session: &str, + ) -> anyhow::Result<(Status, String, Signer, Option)> { + let s0 = decode_delegation(session).map_err(|(_, error)| anyhow::anyhow!(error))?; + let reader = Signer::key(); + let child_not_before = s0.0.not_before.unwrap().unix_timestamp() + 1; + let mut facts = match &s0.0.delegation { + TinyCloudDelegation::Ucan(ucan) => ucan.payload().facts.clone().unwrap(), + TinyCloudDelegation::Cacao(_) => unreachable!(), + }; + facts[0]["remainingRedelegationDepth"] = json!(7); + let Signer::Key { jwk, vm, .. } = &self.holder else { + unreachable!() + }; + let child = Payload { + issuer: vm.parse()?, + audience: reader.did().parse()?, + // Whole seconds (sub-microsecond times don't survive PostgreSQL + // timestamps), after the parent's start; we wait for it below. + not_before: Some(NumericDate::try_from_seconds(child_not_before as f64)?), + expiration: NumericDate::try_from_seconds( + (s0.0.expiry.unwrap().unix_timestamp() - 1) as f64, + )?, + nonce: Some(unique("tc1423child")), + facts: Some(facts), + proof: vec![s0.content_hash().to_cid(0x55)], + attenuation: serde_json::from_value( + attenuation_for_policy_capabilities(&[self.requested_kv()]) + .map_err(|(_, error)| anyhow::anyhow!(error))?, + )?, + } + .sign(Algorithm::EdDSA, jwk)?; + // The child must be valid when it is imported. + let wait = child_not_before - OffsetDateTime::now_utc().unix_timestamp(); + if wait >= 0 { + tokio::time::sleep(std::time::Duration::from_millis(wait as u64 * 1000 + 50)).await; + } + let (status, body) = self + .delegate(TinyCloudDelegation::Ucan(Box::new(child)).encode()?) + .await; + let cid = (status == Status::Ok) + .then(|| serde_json::from_str::(&body).ok()) + .flatten() + .and_then(|value| value["cid"].as_str().and_then(|cid| cid.parse().ok())); + Ok((status, body, reader, cid)) + } + + /// `reader` reads under the re-shared child `child`. + pub async fn read_as(&self, reader: &Signer, child: Cid) -> anyhow::Result<(Status, String)> { + let invocation = reader.invoke( + serde_json::from_value( + attenuation_for_policy_capabilities(&[self.requested_kv()]) + .map_err(|(_, error)| anyhow::anyhow!(error))?, + )?, + child, + )?; + let response = self + .client + .post("/invoke") + .header(Header::new("Authorization", invocation)) + .dispatch() + .await; + let status = response.status(); + Ok(( + status, + text(&response.into_bytes().await.unwrap_or_default()), + )) + } + + /// The holder asks for a signed KV URL under a session. + pub async fn signed_kv(&self, session: &str) -> anyhow::Result<(Status, String)> { + let invocation = self.holder.invoke( + serde_json::from_value( + attenuation_for_policy_capabilities(&[self.requested_kv()]) + .map_err(|(_, error)| anyhow::anyhow!(error))?, + )?, + session_cid(session)?, + )?; + let path = "shares/tc-1423/document.txt"; + let response = self + .client + .post("/signed/kv") + .header(ContentType::JSON) + .header(Header::new("Authorization", invocation)) + .body(json!({"space": self.content_space.to_string(), "path": path}).to_string()) + .dispatch() + .await; + let status = response.status(); + Ok((status, response.into_string().await.unwrap_or_default())) + } + + /// `/delegation/status` on `cid`, invoked by `signer`. + pub async fn delegation_status(&self, signer: &Signer, cid: &str) -> anyhow::Result { + let Signer::Key { jwk, vm, did, .. } = signer else { + anyhow::bail!("status is invoked by keys"); + }; + let at = now(); + let invocation = Payload { + issuer: vm.parse()?, + audience: did.parse()?, + not_before: Some(seconds(at)?), + expiration: seconds(at + Duration::seconds(30))?, + nonce: Some(unique("tc1423status")), + facts: Some(Vec::::new()), + proof: Vec::new(), + attenuation: capabilities(&[( + format!("urn:cid:{cid}"), + "tinycloud.delegation/status", + )])?, + } + .sign(Algorithm::EdDSA, jwk)? + .encode()?; + let response = self + .client + .post("/delegation/status") + .header(ContentType::JSON) + .header(Header::new("Authorization", invocation)) + .body(json!({"cid": cid}).to_string()) + .dispatch() + .await; + Ok(response.into_json().await.unwrap_or(Value::Null)) + } + + pub async fn root_status(&self, root_cid: &str) -> Value { + self.client + .get(format!("/policy/v3/status/{root_cid}")) + .dispatch() + .await + .into_json() + .await + .unwrap_or(Value::Null) + } +} + +/// Revoking the owner's grant ends already-admitted sessions: +/// read, decrypt, a re-shared child, challenge, mint, signed-KV mint and new +/// re-share admission, with a fixed, opaque code. +async fn revoking_the_owner_grant_ends_admitted_sessions(backend: Backend) -> anyhow::Result<()> { + let Some(h) = Harness::new( + backend, + Shape::KeySession, + "revoking_the_owner_grant_ends_admitted_sessions", + ) + .await? + else { + return Ok(()); + }; + let authority = h + .grant_owner(Window::until(now() + Duration::days(2))) + .await?; + let policy = h.register_ok(policy_window()).await?; + let session = h.mint_ok(&policy).await?; + let (status, body) = h.read(&session).await?; + assert_eq!((status, body.as_slice()), (Status::Ok, CONTENT)); + let (_, decrypt_before) = h.decrypt(&session).await?; + assert_ne!(decrypt_before, REVOKED); + let (status, body, reader, child) = h.reshare(&session).await?; + assert_eq!(status, Status::Ok, "{body}"); + let child = child.unwrap(); + let (status, body) = h.read_as(&reader, child).await?; + assert_eq!(status, Status::Ok, "re-shared child read: {body}"); + let (status, body) = h.signed_kv(&session).await?; + assert_eq!(status, Status::Ok, "signed KV: {body}"); + + h.revoke(&h.space_owner, &authority).await?; + + let (status, body) = h.read(&session).await?; + assert_eq!((status, text(&body).as_str()), (Status::Forbidden, REVOKED)); + let checks = [ + ("decrypt", h.decrypt(&session).await?), + ("re-shared child read", h.read_as(&reader, child).await?), + ("challenge", h.challenge(&policy).await), + ("signed KV mint", h.signed_kv(&session).await?), + ]; + for (label, (status, body)) in checks { + assert_eq!(status, Status::Forbidden, "{label}: {body}"); + assert_eq!(body, REVOKED, "{label}"); + assert_opaque(label, &body); + } + let (status, body) = h.mint(&policy).await?.unwrap_err(); + assert_eq!((status, body.as_str()), (Status::Forbidden, REVOKED)); + let (status, body, _, _) = h.reshare(&session).await?; + assert_eq!(status, Status::Forbidden, "re-share admission: {body}"); + assert_opaque("re-share admission", &body); + // The listing by the revoked grant reports it. + let (status, page) = h.list(&h.space_owner, &authority, None, None).await?; + assert_eq!(status, Status::Ok); + assert_eq!(page["policies"][0]["ownerAuthorityState"], json!("revoked")); + Ok(()) +} +on_both_backends!(revoking_the_owner_grant_ends_admitted_sessions); + +/// [B-1] The CLI agent flow: wallet W signs a session for key A, A signs the +/// policy roots. W revokes the A session over the generic route; admitted +/// recipients stop, the root itself stays active. +async fn revoking_the_agent_session_ends_its_policies(backend: Backend) -> anyhow::Result<()> { + let Some(h) = Harness::new( + backend, + Shape::PkhSession, + "revoking_the_agent_session_ends_its_policies", + ) + .await? + else { + return Ok(()); + }; + let agent_session = h + .grant_owner(Window::until(now() + Duration::days(2))) + .await?; + let policy = h.register_ok(policy_window()).await?; + let session = h.mint_ok(&policy).await?; + assert_eq!(h.read(&session).await?.0, Status::Ok); + // Policies grant only kv and decrypt, so a policy session never holds a + // hook ticket (no stream to close). + let (status, body) = h.hook_ticket(&[session_cid(&session)?.to_string()]).await?; + assert_ne!(status, Status::Ok, "{body}"); + + h.revoke(&h.space_owner, &agent_session).await?; + + let (status, body) = h.read(&session).await?; + assert_eq!((status, text(&body).as_str()), (Status::Forbidden, REVOKED)); + assert_eq!( + h.decrypt(&session).await?, + (Status::Forbidden, REVOKED.to_owned()) + ); + assert_eq!(h.signed_kv(&session).await?.0, Status::Forbidden); + // The root itself is not revoked. + assert_eq!( + h.root_status(&policy.policy_root_cid).await["state"], + json!("active") + ); + assert_eq!( + h.delegation_status(&h.owner, &agent_session).await?["status"], + json!("revoked") + ); + let (status, page) = h.list(&h.owner, &agent_session, None, None).await?; + assert_eq!(status, Status::Ok, "{page}"); + assert_eq!(page["policies"][0]["ownerAuthorityState"], json!("revoked")); + Ok(()) +} +on_both_backends!(revoking_the_agent_session_ends_its_policies); + +/// Revoking an ancestor of the recorded grant ends sessions too. +async fn revoking_an_ancestor_ends_admitted_sessions(backend: Backend) -> anyhow::Result<()> { + let Some(h) = Harness::new( + backend, + Shape::KeySession, + "revoking_an_ancestor_ends_admitted_sessions", + ) + .await? + else { + return Ok(()); + }; + let intermediary = Signer::key(); + let window = Window::until(now() + Duration::days(2)); + let upstream = h + .import( + h.space_owner + .delegate(intermediary.did(), &h.owner_grants(), window, &[])?, + ) + .await?; + h.import(intermediary.delegate( + h.owner.did(), + &h.owner_grants(), + window, + &[upstream.parse()?], + )?) + .await?; + let policy = h.register_ok(policy_window()).await?; + let session = h.mint_ok(&policy).await?; + assert_eq!(h.read(&session).await?.0, Status::Ok); + h.revoke(&h.space_owner, &upstream).await?; + let (status, body) = h.read(&session).await?; + assert_eq!((status, text(&body).as_str()), (Status::Forbidden, REVOKED)); + Ok(()) +} +on_both_backends!(revoking_an_ancestor_ends_admitted_sessions); + +/// A revocation stored under an alias CID of a closure node counts. +async fn a_revocation_under_an_alias_cid_ends_sessions(backend: Backend) -> anyhow::Result<()> { + let Some(h) = Harness::new( + backend, + Shape::KeySession, + "a_revocation_under_an_alias_cid_ends_sessions", + ) + .await? + else { + return Ok(()); + }; + let authority = h + .grant_owner(Window::until(now() + Duration::days(2))) + .await?; + let policy = h.register_ok(policy_window()).await?; + let session = h.mint_ok(&policy).await?; + assert_eq!(h.read(&session).await?.0, Status::Ok); + // A second stored encoding of the same grant, and a revocation that names + // only that encoding. + let canonical = authority.parse::()?.hash().to_bytes(); + let alias = hash(b"tc-1423-alias-encoding") + .to_cid(0x55) + .hash() + .to_bytes(); + let space_owner = h.space_owner.did().to_owned(); + let owner = h.owner.did().to_owned(); + let statements = [ + Query::insert() + .into_table(Alias::new("delegation")) + .columns([ + Alias::new("id"), + Alias::new("delegator"), + Alias::new("delegatee"), + Alias::new("serialization"), + ]) + .values_panic([ + alias.clone().into(), + space_owner.clone().into(), + owner.into(), + b"alias".to_vec().into(), + ]) + .to_owned(), + Query::insert() + .into_table(Alias::new("delegation_cid_alias")) + .columns([ + Alias::new("id"), + Alias::new("canonical"), + Alias::new("encrypted"), + ]) + .values_panic([alias.clone().into(), canonical.into(), false.into()]) + .to_owned(), + Query::insert() + .into_table(Alias::new("revocation")) + .columns([ + Alias::new("id"), + Alias::new("revoker"), + Alias::new("revoked"), + Alias::new("serialization"), + ]) + .values_panic([ + hash(b"tc-1423-alias-revocation") + .to_cid(0x55) + .hash() + .to_bytes() + .into(), + space_owner.into(), + alias.into(), + b"alias-revocation".to_vec().into(), + ]) + .to_owned(), + ]; + for statement in statements { + h.conn().execute(h.backend().build(&statement)).await?; + } + let (status, body) = h.read(&session).await?; + assert_eq!((status, text(&body).as_str()), (Status::Forbidden, REVOKED)); + Ok(()) +} +on_both_backends!(a_revocation_under_an_alias_cid_ends_sessions); + +/// All-of: kv granted by one delegation and decrypt by another; revoking +/// either ends the policy. +async fn each_recorded_grant_is_required(backend: Backend) -> anyhow::Result<()> { + for revoke_kv in [true, false] { + let Some(h) = Harness::new( + backend, + Shape::KeySession, + "each_recorded_grant_is_required", + ) + .await? + else { + return Ok(()); + }; + let window = Window::until(now() + Duration::days(2)); + let grants = h.owner_grants(); + let kv = h.grant_owner_with(&grants[..2], window).await?; + let decrypt = h.grant_owner_with(&grants[2..], window).await?; + let policy = h.register_ok(policy_window()).await?; + let mut expected = vec![kv.clone(), decrypt.clone()]; + expected.sort(); + assert_eq!(h.closure(&policy.policy_cid).await?, expected); + let session = h.mint_ok(&policy).await?; + assert_eq!(h.read(&session).await?.0, Status::Ok); + h.revoke(&h.space_owner, if revoke_kv { &kv } else { &decrypt }) + .await?; + let (status, body) = h.read(&session).await?; + assert_eq!( + (status, text(&body).as_str()), + (Status::Forbidden, REVOKED), + "revoke kv: {revoke_kv}" + ); + } + Ok(()) +} +on_both_backends!(each_recorded_grant_is_required); + +/// A root bound to its authority's expiry: once both pass, reads see the +/// session expired and mints see the root expired. +async fn expiry_codes_are_typed(backend: Backend) -> anyhow::Result<()> { + let Some(h) = Harness::new(backend, Shape::KeySession, "expiry_codes_are_typed").await? else { + return Ok(()); + }; + let expiry = now() + Duration::seconds(12); + h.grant_owner(Window::until(expiry)).await?; + let policy = h.register_ok(Window::until(expiry)).await?; + assert_eq!(h.binding(&policy.policy_cid).await?.unwrap().state, "bound"); + let session = h.mint_ok(&policy).await?; + assert_eq!(h.read(&session).await?.0, Status::Ok); + tokio::time::sleep( + (expiry - OffsetDateTime::now_utc()).unsigned_abs() + std::time::Duration::from_secs(1), + ) + .await; + let (status, body) = h.read(&session).await?; + assert_eq!(status, Status::Forbidden); + assert_eq!(text(&body), "policy-session-expired"); + let (status, body) = h.challenge(&policy).await; + assert_eq!((status, body.as_str()), (Status::Forbidden, "root-expired")); + Ok(()) +} +on_both_backends!(expiry_codes_are_typed); + +/// A legacy registration with no authority is denied; the operator +/// escape restores root-only checks and is counted. +async fn legacy_registrations_without_authority_are_denied_unless_escaped( + backend: Backend, +) -> anyhow::Result<()> { + for allow in [false, true] { + let mut policy_v3 = json!({"legacy_cutover": format_time(now() + Duration::days(1))}); + if allow { + policy_v3["legacy_unknown_allow"] = json!(true); + } + let Some(h) = Harness::with_boot( + backend, + Shape::KeySession, + "legacy_registrations_without_authority_are_denied_unless_escaped", + boot(policy_v3.clone()), + ) + .await? + else { + return Ok(()); + }; + let authority = h + .grant_owner(Window::until(now() + Duration::days(2))) + .await?; + let policy = h.register_ok(policy_window()).await?; + let session = h.mint_ok(&policy).await?; + h.revoke(&h.space_owner, &authority).await?; + h.forget_binding(&policy.policy_cid).await?; + // A fresh process: this one remembers the earlier classification. + let mut h = h; + h.reboot(boot(policy_v3)).await?; + let allowed = counter( + "tinycloud_policy_owner_authority_unknown_allowed_total", + &[("reason", "revoked")], + ); + let (status, body) = h.read(&session).await?; + if allow { + assert_eq!((status, body.as_slice()), (Status::Ok, CONTENT)); + assert!( + counter( + "tinycloud_policy_owner_authority_unknown_allowed_total", + &[("reason", "revoked")], + ) >= allowed + 1.0 + ); + } else { + assert_eq!( + (status, text(&body).as_str()), + (Status::Forbidden, "policy-owner-authority-unknown") + ); + } + let binding = h + .wait_for_binding(&policy.policy_cid, std::time::Duration::from_secs(20)) + .await? + .expect("classified"); + assert_eq!(binding.state, "unknown"); + assert_eq!(binding.unknown_reason.as_deref(), Some("revoked")); + } + Ok(()) +} +on_both_backends!(legacy_registrations_without_authority_are_denied_unless_escaped); + +/// Legacy rows get liveness but no expiry cap: a legacy session +/// whose authority expired since registration reads, and so does a +/// legacy root longer than 31 days. +async fn legacy_registrations_are_never_expiry_capped(backend: Backend) -> anyhow::Result<()> { + let Some(h) = Harness::with_boot( + backend, + Shape::KeySession, + "legacy_registrations_are_never_expiry_capped", + legacy_boot(), + ) + .await? + else { + return Ok(()); + }; + let short_expiry = now() + Duration::seconds(8); + h.grant_owner(Window::until(short_expiry)).await?; + let short = h.register_ok(policy_window()).await?; + let short_session = h.mint_ok(&short).await?; + h.forget_binding(&short.policy_cid).await?; + + let registered_at = now() - Duration::days(40); + h.grant_owner(Window { + not_before: registered_at - Duration::days(1), + expiry: now() + Duration::days(2), + issued_at: None, + }) + .await?; + let long = h.register_ok(policy_window()).await?; + let long_session = h.mint_ok(&long).await?; + h.forget_binding(&long.policy_cid).await?; + backdate(&h, &long.policy_cid, registered_at).await?; + + tokio::time::sleep( + (short_expiry - OffsetDateTime::now_utc()).unsigned_abs() + + std::time::Duration::from_secs(1), + ) + .await; + for (label, session) in [ + ("expired authority", &short_session), + ("41-day root", &long_session), + ] { + let (status, body) = h.read(session).await?; + assert_eq!(status, Status::Ok, "{label}: {}", text(&body)); + assert_eq!(body, CONTENT); + } + Ok(()) +} +on_both_backends!(legacy_registrations_are_never_expiry_capped); + +/// The escape covers legacy rows only: a post-cutover row without +/// authority is denied with the flag on. +async fn the_escape_never_serves_post_cutover_rows(backend: Backend) -> anyhow::Result<()> { + let Some(mut h) = Harness::new( + backend, + Shape::KeySession, + "the_escape_never_serves_post_cutover_rows", + ) + .await? + else { + return Ok(()); + }; + let registered_at = now() - Duration::days(40); + h.grant_owner(Window { + not_before: registered_at - Duration::days(1), + expiry: now() + Duration::days(2), + issued_at: None, + }) + .await?; + let policy = h.register_ok(policy_window()).await?; + let session = h.mint_ok(&policy).await?; + h.forget_binding(&policy.policy_cid).await?; + backdate(&h, &policy.policy_cid, registered_at).await?; + h.reboot(boot(json!({ + "legacy_cutover": format_time(now() - Duration::days(60)), + "legacy_unknown_allow": true, + }))) + .await?; + let (status, body) = h.read(&session).await?; + assert_eq!( + (status, text(&body).as_str()), + (Status::Forbidden, "policy-owner-authority-unknown") + ); + let binding = h.binding(&policy.policy_cid).await?; + let binding = match binding { + Some(binding) => binding, + None => h + .wait_for_binding(&policy.policy_cid, std::time::Duration::from_secs(20)) + .await? + .expect("classified"), + }; + assert_eq!( + ( + binding.registered_mode.as_str(), + binding.unknown_reason.as_deref() + ), + ("record", Some("lifetime-exceeded")) + ); + Ok(()) +} +on_both_backends!(the_escape_never_serves_post_cutover_rows); + +/// An escaped legacy policy serves its admitted sessions, but a mint +/// needs the owner to have held what the policy grants when it +/// registered. +async fn escaped_mints_still_check_the_owner_held_the_capability( + backend: Backend, +) -> anyhow::Result<()> { + let escape = || { + boot(json!({ + "legacy_cutover": format_time(now() + Duration::days(1)), + "legacy_unknown_allow": true, + })) + }; + let Some(mut h) = Harness::with_boot( + backend, + Shape::KeySession, + "escaped_mints_still_check_the_owner_held_the_capability", + escape(), + ) + .await? + else { + return Ok(()); + }; + h.grant_owner(Window { + not_before: now() - Duration::seconds(30), + expiry: now() + Duration::days(2), + issued_at: None, + }) + .await?; + let policy = h.register_ok(policy_window()).await?; + let session = h.mint_ok(&policy).await?; + h.forget_binding(&policy.policy_cid).await?; + // Registered before the owner held the grant: nothing to record. + backdate(&h, &policy.policy_cid, now() - Duration::hours(1)).await?; + h.reboot(escape()).await?; + let (status, body) = h.read(&session).await?; + assert_eq!((status, body.as_slice()), (Status::Ok, CONTENT), "escaped"); + let (status, body) = h.mint(&policy).await?.expect_err("mint refused"); + assert_eq!( + (status, body.as_str()), + (Status::Forbidden, "policy-owner-not-authorized") + ); + Ok(()) +} +on_both_backends!(escaped_mints_still_check_the_owner_held_the_capability); + +/// An admitted session can't be imported again once its owner authority +/// is revoked. +async fn admitted_sessions_are_not_reimported_after_a_revoke( + backend: Backend, +) -> anyhow::Result<()> { + let Some(h) = Harness::new( + backend, + Shape::KeySession, + "admitted_sessions_are_not_reimported_after_a_revoke", + ) + .await? + else { + return Ok(()); + }; + let authority = h + .grant_owner(Window::until(now() + Duration::days(2))) + .await?; + let policy = h.register_ok(policy_window()).await?; + let session = h.mint_ok(&policy).await?; + let (status, body) = h.delegate(session.clone()).await; + assert_eq!(status, Status::Ok, "before the revoke: {body}"); + h.revoke(&h.space_owner, &authority).await?; + let (status, body) = h.delegate(session).await; + assert_eq!((status, body.as_str()), (Status::Forbidden, REVOKED)); + Ok(()) +} +on_both_backends!(admitted_sessions_are_not_reimported_after_a_revoke); + +/// Concurrent first uses of an unclassified policy share one +/// classification. +async fn concurrent_first_uses_share_one_classification(backend: Backend) -> anyhow::Result<()> { + let Some(mut h) = Harness::with_boot( + backend, + Shape::KeySession, + "concurrent_first_uses_share_one_classification", + legacy_boot(), + ) + .await? + else { + return Ok(()); + }; + h.grant_owner(Window::until(now() + Duration::days(2))) + .await?; + let policy = h.register_ok(policy_window()).await?; + let session = h.mint_ok(&policy).await?; + h.forget_binding(&policy.policy_cid).await?; + h.reboot(legacy_boot()).await?; + let classifications = || { + h.client + .rocket() + .state::() + .expect("policy runtime") + .owner_authority + .classifications + .load(std::sync::atomic::Ordering::Relaxed) + }; + let before = classifications(); + let reads = futures::future::join_all((0..8).map(|_| h.read(&session))).await; + for read in reads { + let (status, body) = read?; + assert_eq!((status, body.as_slice()), (Status::Ok, CONTENT)); + } + assert_eq!(classifications() - before, 1); + Ok(()) +} +on_both_backends!(concurrent_first_uses_share_one_classification); + +impl Harness { + /// The space owner grants the holder hooks subscription on the content + /// prefix; returns the grant CIDs. + async fn grant_hooks(&self, count: usize) -> anyhow::Result> { + let resource = self.content_space.clone().to_resource( + "hooks".parse()?, + Some("kv/shares".parse()?), + None, + None, + ); + let mut grants = Vec::new(); + for _ in 0..count { + grants.push( + self.import(self.space_owner.delegate( + self.holder.did(), + &[(resource.to_string(), "tinycloud.hooks/subscribe")], + Window::until(now() + Duration::hours(2)), + &[], + )?) + .await?, + ); + } + Ok(grants) + } + + /// The holder mints a hook ticket on the content prefix citing `proofs`. + async fn hook_ticket(&self, proofs: &[String]) -> anyhow::Result<(Status, String)> { + let resource = self.content_space.clone().to_resource( + "hooks".parse()?, + Some("kv/shares".parse()?), + None, + None, + ); + let Signer::Key { jwk, vm, did, .. } = &self.holder else { + unreachable!() + }; + let at = now(); + let invocation = Payload { + issuer: vm.parse()?, + audience: did.parse()?, + not_before: Some(seconds(at)?), + expiration: seconds(at + Duration::seconds(60))?, + nonce: Some(unique("tc1423hooks")), + facts: Some(Vec::::new()), + proof: proofs + .iter() + .map(|cid| cid.parse()) + .collect::>()?, + attenuation: capabilities(&[(resource.to_string(), "tinycloud.hooks/subscribe")])?, + } + .sign(Algorithm::EdDSA, jwk)? + .encode()?; + let response = self + .client + .post("/hooks/tickets") + .header(ContentType::JSON) + .header(Header::new("Authorization", invocation)) + .body( + json!({ + "subscriptions": [{ + "space": self.content_space.to_string(), + "service": "kv", + "pathPrefix": "shares", + }], + "ttlSeconds": 120, + }) + .to_string(), + ) + .dispatch() + .await; + let status = response.status(); + let body = response.into_string().await.unwrap_or_default(); + Ok((status, body)) + } + + /// The holder writes the content again, emitting a hook event. + async fn write_content(&self, body: &'static [u8]) -> anyhow::Result<()> { + let grant = self + .import(self.space_owner.delegate( + self.holder.did(), + &[(self.content_resource.to_string(), "tinycloud.kv/put")], + Window::until(now() + Duration::hours(1)), + &[], + )?) + .await?; + let Signer::Key { jwk, vm, .. } = &self.holder else { + unreachable!() + }; + let write = make_invocation( + [( + self.content_resource.clone(), + ["tinycloud.kv/put".parse::()?], + )], + &grant.parse()?, + jwk, + vm, + (now() + Duration::seconds(45)).unix_timestamp() as f64, + InvocationOptions { + nonce: Some(unique("tc1423write")), + ..InvocationOptions::default() + }, + )?; + let response = self + .client + .post("/invoke") + .header(Header::new("Authorization", write.encode()?)) + .header(ContentType::Plain) + .body(body) + .dispatch() + .await; + anyhow::ensure!( + response.status() == Status::Ok, + "write: {}", + response.status() + ); + Ok(()) + } +} + +fn fast_hooks() -> Boot { + Boot { + hooks: Some(json!({ + "authority_recheck_seconds": 1, + "idle_authority_recheck_seconds": 2, + })), + ..Boot::default() + } +} + +/// Re-checks before events only: the idle re-check never runs in a test. +fn event_rechecks_only() -> Boot { + Boot { + hooks: Some(json!({ + "authority_recheck_seconds": 1, + "idle_authority_recheck_seconds": 600, + })), + ..Boot::default() + } +} + +/// The SSE field rocket writes for each delivered hook event. +const WRITE_EVENT: &str = "event:write"; + +/// Connect a hook stream (its authority is checked here). +async fn open_stream<'c>( + h: &'c Harness, + ticket: &str, +) -> rocket::local::asynchronous::LocalResponse<'c> { + let response = h + .client + .get(format!("/hooks/events?ticket={ticket}")) + .dispatch() + .await; + assert_eq!(response.status(), Status::Ok); + response +} + +/// Read a connected hook stream to its end, or give up after `limit`. +async fn drain_stream( + response: rocket::local::asynchronous::LocalResponse<'_>, + limit: std::time::Duration, +) -> Option { + tokio::time::timeout(limit, response.into_string()) + .await + .ok() + .map(Option::unwrap_or_default) +} + +/// An open hook stream stops delivering after its grant is revoked: +/// the next event after the re-check interval is withheld and the stream +/// closes (with no idle re-check to close it otherwise); an idle stream +/// closes on the idle re-check. +async fn hook_streams_close_after_their_grant_is_revoked(backend: Backend) -> anyhow::Result<()> { + let Some(mut h) = Harness::with_boot( + backend, + Shape::KeySession, + "hook_streams_close_after_their_grant_is_revoked", + event_rechecks_only(), + ) + .await? + else { + return Ok(()); + }; + let grant = h.grant_hooks(1).await?; + let (status, body) = h.hook_ticket(&grant).await?; + assert_eq!(status, Status::Ok, "{body}"); + let ticket = serde_json::from_str::(&body)?["ticket"] + .as_str() + .unwrap() + .to_owned(); + let stream = drain_stream( + open_stream(&h, &ticket).await, + std::time::Duration::from_secs(15), + ); + let actions = async { + // Delivered: the stream is live and within its interval. + h.write_content(b"before-revoke").await?; + h.revoke(&h.space_owner, &grant[0]).await?; + tokio::time::sleep(std::time::Duration::from_millis(1500)).await; + h.write_content(b"after-revoke").await?; + anyhow::Ok(()) + }; + let (body, actions) = tokio::join!(stream, actions); + actions?; + let body = body.expect("the pre-event re-check closes the stream after the revocation"); + assert_eq!(body.matches(WRITE_EVENT).count(), 1, "{body}"); + + // Idle: nothing is written; the idle re-check closes it. + h.reboot(fast_hooks()).await?; + let grant = h.grant_hooks(1).await?; + let (_, body) = h.hook_ticket(&grant).await?; + let ticket = serde_json::from_str::(&body)?["ticket"] + .as_str() + .unwrap() + .to_owned(); + let stream = drain_stream( + open_stream(&h, &ticket).await, + std::time::Duration::from_secs(10), + ); + let revoke = h.revoke(&h.space_owner, &grant[0]); + let (body, revoked) = tokio::join!(stream, revoke); + revoked?; + assert!( + body.is_some(), + "the idle stream closes after the revocation" + ); + Ok(()) +} +on_both_backends!(hook_streams_close_after_their_grant_is_revoked); + +/// Re-check errors close a stream only after three in a row. +async fn hook_streams_close_after_repeated_check_errors(backend: Backend) -> anyhow::Result<()> { + let Some(h) = Harness::with_boot( + backend, + Shape::KeySession, + "hook_streams_close_after_repeated_check_errors", + fast_hooks(), + ) + .await? + else { + return Ok(()); + }; + let grant = h.grant_hooks(1).await?; + let (_, body) = h.hook_ticket(&grant).await?; + let ticket = serde_json::from_str::(&body)?["ticket"] + .as_str() + .unwrap() + .to_owned(); + let started = tokio::time::Instant::now(); + let stream = drain_stream( + open_stream(&h, &ticket).await, + std::time::Duration::from_secs(15), + ); + let break_checks = async { + tokio::time::sleep(std::time::Duration::from_millis(300)).await; + h.execute("DROP TABLE revocation").await + }; + let (body, dropped) = tokio::join!(stream, break_checks); + dropped?; + assert!(body.is_some(), "three failed checks close the stream"); + // Idle checks every 2 s: three errors take at least ~4 s. + assert!(started.elapsed() >= std::time::Duration::from_secs(4)); + Ok(()) +} +on_both_backends!(hook_streams_close_after_repeated_check_errors); + +/// A ticket would carry more than 16 authority IDs: refused. +async fn hook_tickets_carry_at_most_16_authorities(backend: Backend) -> anyhow::Result<()> { + let Some(h) = Harness::new( + backend, + Shape::KeySession, + "hook_tickets_carry_at_most_16_authorities", + ) + .await? + else { + return Ok(()); + }; + let grants = h.grant_hooks(17).await?; + assert_eq!(h.hook_ticket(&grants[..16]).await?.0, Status::Ok); + let (status, body) = h.hook_ticket(&grants).await?; + assert_eq!(status, Status::Forbidden, "{body}"); + assert_eq!(body, "hook-ticket-authority-too-large"); + Ok(()) +} +on_both_backends!(hook_tickets_carry_at_most_16_authorities); + +/// PostgreSQL latency gate for a policy read (the owner-authority +/// check adds one indexed revocation query per read). Ignored by default; +/// run the same test on the baseline and candidate trees: +/// +/// ```sh +/// TINYCLOUD_TEST_POSTGRES_URL=... cargo test -p tinycloud-node --lib \ +/// postgres_policy_read_latency_gate -- --ignored --nocapture +/// ``` +/// +/// With `TC1423_BENCH_BASELINE_P50_US` and `TC1423_BENCH_BASELINE_P99_US` +/// set from the baseline run, it fails when p50 grows by more than 15% (+2 +/// ms) or p99 by more than 25% (+5 ms). +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "benchmark; run explicitly against PostgreSQL"] +async fn postgres_policy_read_latency_gate() -> anyhow::Result<()> { + let Some(h) = Harness::new( + Backend::Postgres, + Shape::KeySession, + "postgres_policy_read_latency_gate", + ) + .await? + else { + return Ok(()); + }; + h.grant_owner(Window::until(now() + Duration::days(2))) + .await?; + let policy = h.register_ok(policy_window()).await?; + let session = h.mint_ok(&policy).await?; + for _ in 0..30 { + assert_eq!(h.read(&session).await?.0, Status::Ok); + } + let mut samples = Vec::with_capacity(400); + for _ in 0..400 { + let started = std::time::Instant::now(); + let (status, _) = h.read(&session).await?; + samples.push(started.elapsed().as_micros() as u64); + assert_eq!(status, Status::Ok); + } + samples.sort_unstable(); + let p50 = samples[samples.len() / 2]; + let p99 = samples[samples.len() * 99 / 100]; + println!( + "TC1423_BENCH reads={} p50_us={p50} p99_us={p99}", + samples.len() + ); + let baseline = |name: &str| { + std::env::var(name) + .ok() + .and_then(|value| value.parse::().ok()) + }; + if let (Some(base50), Some(base99)) = ( + baseline("TC1423_BENCH_BASELINE_P50_US"), + baseline("TC1423_BENCH_BASELINE_P99_US"), + ) { + assert!( + p50 <= base50 * 115 / 100 + 2_000, + "p50 {p50} µs vs baseline {base50} µs" + ); + assert!( + p99 <= base99 * 125 / 100 + 5_000, + "p99 {p99} µs vs baseline {base99} µs" + ); + } + Ok(()) +} diff --git a/tinycloud-node-server/src/prometheus.rs b/tinycloud-node-server/src/prometheus.rs index 2d9a22c2..a1f7ab3c 100644 --- a/tinycloud-node-server/src/prometheus.rs +++ b/tinycloud-node-server/src/prometheus.rs @@ -505,3 +505,31 @@ pub fn observe_policy_owner_binding_backfill(outcome: &'static str) { .with_label_values(&[outcome]) .inc(); } + +lazy_static! { + static ref POLICY_OWNER_AUTHORITY_DENIED: IntCounterVec = register_int_counter_vec!( + "tinycloud_policy_owner_authority_denied_total", + "Policy/v3 uses denied by the recorded owner authority, by reason.", + &["reason"] + ) + .unwrap(); + static ref POLICY_OWNER_AUTHORITY_UNKNOWN_ALLOWED: IntCounterVec = register_int_counter_vec!( + "tinycloud_policy_owner_authority_unknown_allowed_total", + "Policy/v3 uses of legacy registrations without owner authority served under policy_v3.legacy_unknown_allow, by reason.", + &["reason"] + ) + .unwrap(); +} + +/// `revoked`, `expired` or `unknown`. +pub fn observe_policy_owner_authority_denied(reason: &'static str) { + POLICY_OWNER_AUTHORITY_DENIED + .with_label_values(&[reason]) + .inc(); +} + +pub fn observe_policy_owner_authority_unknown_allowed(reason: &'static str) { + POLICY_OWNER_AUTHORITY_UNKNOWN_ALLOWED + .with_label_values(&[reason]) + .inc(); +} diff --git a/tinycloud-node-server/src/routes/hooks.rs b/tinycloud-node-server/src/routes/hooks.rs index 529fbee6..8849cac6 100644 --- a/tinycloud-node-server/src/routes/hooks.rs +++ b/tinycloud-node-server/src/routes/hooks.rs @@ -26,6 +26,7 @@ use rocket::{ use serde::{Deserialize, Serialize}; use std::{future::Future, net::IpAddr, time::Duration, time::Instant}; use time::{format_description::well_known::Rfc3339, OffsetDateTime}; +use tinycloud_auth::ipld_core::cid::Cid; use tinycloud_core::{ events::Invocation, hash::Blake3Hasher, @@ -46,6 +47,9 @@ use tinycloud_core::{ struct VerifiedHookInvocation { admitted: AdmittedInvocation, policy_session: bool, + /// For a policy session, the recorded owner-authority closure it passed; + /// tickets carry it so open streams watch it too. + policy_authority: Vec, } impl VerifiedHookInvocation { @@ -108,13 +112,14 @@ async fn authorize_hook_request( start.elapsed(), ); let admitted = admitted?; - let policy_session = policy_v3 + let policy_authority = policy_v3 .authorize_invocation(tinycloud, &admitted.invocation().0, now) .await .map_err(|error| (Status::Forbidden, error.to_string()))?; Ok(VerifiedHookInvocation { admitted, - policy_session, + policy_session: policy_authority.is_some(), + policy_authority: policy_authority.unwrap_or_default(), }) } @@ -239,6 +244,19 @@ async fn mint_hook_ticket( )); } + let mut authorizing_delegations = invocation.parents.clone(); + for cid in &verified.policy_authority { + if !authorizing_delegations.contains(cid) { + authorizing_delegations.push(*cid); + } + } + // The ticket travels in the stream URL. + if authorizing_delegations.len() > MAX_TICKET_AUTHORITIES { + return Err(( + Status::Forbidden, + "hook-ticket-authority-too-large".to_string(), + )); + } let claims = HookTicketClaims { v: 2, sub: invocation.invoker.clone(), @@ -246,7 +264,7 @@ async fn mint_hook_ticket( iat: now.unix_timestamp(), exp, parent_exp, - authorizing_delegations: invocation.parents.clone(), + authorizing_delegations, }; let ticket = hooks .sign_ticket(&claims) @@ -259,6 +277,69 @@ async fn mint_hook_ticket( Ok(HookTicketResponse { ticket, expires_at }) } +/// Most authority IDs a ticket carries (it travels in the stream URL). +const MAX_TICKET_AUTHORITIES: usize = 16; + +/// Consecutive failed authority checks after which an open stream closes. +const MAX_CONSECUTIVE_CHECK_ERRORS: u32 = 3; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Recheck { + Live, + /// The check failed; keep the stream and check again later. + Retry, + Close, +} + +/// An open stream's authority re-check: before delivering an event at most +/// once per interval, and on an idle timer. It closes the stream on a +/// definite "revoked", or after three consecutive check errors, each at +/// least an interval apart, so a busy database or a burst of events during a +/// brief error doesn't close streams. While the last check failed, events +/// are withheld without querying. +struct AuthorityRecheck { + interval: Duration, + last: rocket::tokio::time::Instant, + consecutive_errors: u32, +} + +impl AuthorityRecheck { + fn new(interval: Duration) -> Self { + Self { + interval, + last: rocket::tokio::time::Instant::now(), + consecutive_errors: 0, + } + } + + fn due(&self) -> bool { + self.last.elapsed() >= self.interval + } + + fn failing(&self) -> bool { + self.consecutive_errors > 0 + } + + fn record(&mut self, result: Result) -> Recheck { + self.last = rocket::tokio::time::Instant::now(); + match result { + Ok(true) => { + self.consecutive_errors = 0; + Recheck::Live + } + Ok(false) => Recheck::Close, + Err(_) => { + self.consecutive_errors += 1; + if self.consecutive_errors >= MAX_CONSECUTIVE_CHECK_ERRORS { + Recheck::Close + } else { + Recheck::Retry + } + } + } + } +} + async fn hook_stream_deadline( claims: &HookTicketClaims, max_ticket_ttl_seconds: u64, @@ -355,11 +436,18 @@ pub async fn hook_events<'r>( .map_err(|e| (Status::TooManyRequests, e))?; let mut receiver = hooks.bus().subscribe(); + let active_recheck = Duration::from_secs(hooks.config().authority_recheck_seconds); + let idle_recheck = Duration::from_secs(hooks.config().idle_authority_recheck_seconds); let stream = EventStream! { let _lease = lease; let deadline_sleep = rocket::tokio::time::sleep_until(stream_deadline); rocket::tokio::pin!(deadline_sleep); + let mut idle = rocket::tokio::time::interval_at( + rocket::tokio::time::Instant::now() + idle_recheck, + idle_recheck, + ); + let mut recheck = AuthorityRecheck::new(active_recheck); loop { rocket::tokio::select! { @@ -367,10 +455,34 @@ pub async fn hook_events<'r>( _ = &mut deadline_sleep => { break; } + _ = idle.tick() => { + let result = tinycloud + .delegation_chains_unrevoked(&claims.authorizing_delegations) + .await; + if recheck.record(result) == Recheck::Close { + break; + } + } message = receiver.recv() => { match message { Ok(event) => { if claims.scopes.iter().any(|scope| matches_scope(&event, scope)) { + if recheck.due() { + let result = tinycloud + .delegation_chains_unrevoked(&claims.authorizing_delegations) + .await; + match recheck.record(result) { + Recheck::Close => break, + // A failed check withholds this event; + // the next check waits an interval. + Recheck::Retry => continue, + Recheck::Live => {} + } + } else if recheck.failing() { + // Until the next check succeeds, events + // are withheld without querying. + continue; + } yield Event::json(&event) .id(event.id.clone()) .event("write"); @@ -944,6 +1056,29 @@ async fn find_parent_expiry( #[cfg(test)] mod tests { + #[test] + fn authority_recheck_closes_on_revoked_or_three_consecutive_errors() { + let mut recheck = AuthorityRecheck::new(Duration::from_secs(5)); + assert!(!recheck.due()); + assert_eq!(recheck.record::<()>(Ok(true)), Recheck::Live); + assert_eq!(recheck.record(Err(())), Recheck::Retry); + assert!( + !recheck.due() && recheck.failing(), + "after a failed check, events are withheld until the interval passes" + ); + assert_eq!(recheck.record(Err(())), Recheck::Retry); + assert_eq!( + recheck.record::<()>(Ok(true)), + Recheck::Live, + "a success resets" + ); + assert_eq!(recheck.record(Err(())), Recheck::Retry); + assert_eq!(recheck.record(Err(())), Recheck::Retry); + assert_eq!(recheck.record(Err(())), Recheck::Close); + let mut recheck = AuthorityRecheck::new(Duration::from_secs(5)); + assert_eq!(recheck.record::<()>(Ok(false)), Recheck::Close); + } + use super::*; use crate::{ config::HooksConfig, hooks::HookRuntime, @@ -1101,6 +1236,7 @@ mod tests { Ok(VerifiedHookInvocation { admitted: AdmittedInvocation::admit(invocation, 300).await?, policy_session: false, + policy_authority: Vec::new(), }) } diff --git a/tinycloud-node-server/src/routes/mod.rs b/tinycloud-node-server/src/routes/mod.rs index bcc79411..0aba8d4d 100644 --- a/tinycloud-node-server/src/routes/mod.rs +++ b/tinycloud-node-server/src/routes/mod.rs @@ -1874,7 +1874,8 @@ async fn invoke_impl( let policy_session_invocation = policy_v3 .authorize_invocation(tinycloud, &admitted.invocation().0, now) .await - .map_err(|error| (Status::Forbidden, error.to_string()))?; + .map_err(|error| (Status::Forbidden, error.to_string()))? + .is_some(); let decrypt_network = admitted .invocation()