diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md new file mode 100644 index 0000000..d66fde8 --- /dev/null +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -0,0 +1,861 @@ +--- +type: rfc +rfc: '610.0000' +title: Flutter Pluggable CI Design +description: Describes the integration with partner CI systems to trigger partner CI execution on post-submit PRs and display execution details on the post-submit dashboard. +status: draft +created: 2026-10-06T09:00:00Z +updated: 2026-10-15T00:00:00Z +tags: + - 600-infrastructure + - 610-cicd + - 450-custom-embedders +authors: + - '"Dmitry Grand" ' +--- + +# RFC 610.0000: Flutter Pluggable CI Design + +## 1. Overview + +To enable Flutter to scale across external platforms and out-of-tree embedder maintainers, Flutter infrastructure (Cocoon) requires a robust, secure, and decoupled communication model. This system notifies partner CI pipelines of post-submit events on `flutter/flutter` so partners can execute custom test suites on proprietary hardware and report execution status back to the Flutter build dashboard. + +### 1.1. Problem Statement + +To allow partners who maintain out-of-tree embedders to detect regressions immediately after a PR is merged into a Flutter repository by running their CI on every Flutter post-submit event. + +### 1.2. Background & Audience + +Multiple Flutter partners adopt custom embedders to support specific platforms not officially supported by Flutter. Changes in the upstream Flutter repository often cause build failures for custom embedders. Depending on how frequently a partner syncs their fork, isolating the exact breaking change is difficult to manage. This solution mitigates the issue by allowing partners to run their CI pipelines against every Flutter PR post-submit, catching regressions early. Flutter infrastructure cannot allow partner CI execution to block the tree, at least for now, because gardeners responsible for maintaining tree health lack the knowledge or access to partner codebases and infrastructure necessary to resolve failures effectively. + +**Expected Scale:** + +* **Post-submit Event Volume:** ~20-30 post-submit events per day across all target branches on `flutter/flutter`. +* **Partner Integration Volume:** 5–20 configured partner tasks per post-submit event over the next 1–2 years. +* **Partner Execution Duration:** Typically 15 minutes to 1 hour per run, depending on custom hardware availability and test suite scope. + +**Target Audience:** + +* Flutter TLs +* First-party partners +* Third-party partners + +### 1.3. Terminology + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [BCP 14](https://www.rfc-editor.org/info/bcp14) ([RFC 2119](https://www.rfc-editor.org/info/rfc2119/)). + +### 1.4. Glossary + +* **In-Tree Platforms:** Core Flutter platforms that are stored, updated, and managed directly inside the main Flutter repository. This includes the official ports for iOS, Android, macOS, Linux, Windows, and Web. +* **Out-of-Tree Platforms:** External Flutter platforms developed and maintained by third parties in separate, independent repositories. Because they live outside the main codebase, today developers must fork the Flutter tool and reverse-engineer undocumented system channels (like text input or app lifecycles) to make them work. +* **Custom Embedders:** Platform-specific codebases that host the Flutter engine and Dart runtime on non-standard operating systems. They act as out-of-tree ports for custom hardware. +* **Cocoon:** Flutter's custom Continuous Integration (CI) orchestration system that integrates underlying CI services like LUCI with GitHub to schedule tests and report build statuses. It also provides developer tooling such as auto-submit mechanisms, test flakiness tracking, and powers the public Flutter Pre-submit/Build Dashboards. +* **LUCI (Layered Universal Continuous Integration):** The underlying execution engine that runs Flutter's testing and build workloads across diverse platforms like macOS, Windows, and Linux. Orchestrated by Cocoon, LUCI executes specific Python-based "recipes" to securely validate code changes for the Flutter framework, engine, and packages during both pre-submit and post-submit pipelines. +* **Post-submit Event:** An event in Cocoon that is triggered after a PR is merged into the `flutter/flutter` repository. This may be a merge to `master`, `beta`, `stable` or release candidate branches. + +--- + +## 2. Goals & Non-Goals + +### 2.1. Goals + +* **Non-Blocking Post-Submit Execution:** Trigger partner CI runs on post-submit events and display results on the Flutter build dashboard without blocking the tree. +* **Decentralized Security:** Authenticate bidirectional server-to-server requests using [Ed25519](https://www.rfc-editor.org/info/rfc8410/) asymmetric key pairs and public JWKS endpoints without a central authorization server. +* **High Resilience:** Tolerate temporary network and partner infrastructure outages via asynchronous GCP Pub/Sub buffering and exponential backoff retries. +* **Minimal Operational Overhead:** Avoid automated status polling loops by combining push callbacks with optional manual on-demand status reconciliation. + +### 2.2. Non-Goals + +* Defining what kind of CI system out-of-tree platform maintainers must use or what tests they run to validate custom embedders. +* Guaranteeing the delivery of CI execution status from partner CI systems back to Cocoon if the partner infrastructure permanently fails to invoke the callback endpoint and does not support the manual status endpoint. + +--- + +## 3. System Architecture + +Integration with third-party CIs requires two main components: communication and authentication. The recommended architecture is **bi-directional webhook server-to-server communication (Push-Callback)** backed internally by GCP Pub/Sub for fault tolerance and retries, combined with an optional **manual on-demand status reconciliation** mechanism. + +### 3.1. High-Level Architecture + +In this architecture, Cocoon uses standard HTTP webhooks to push event notifications to external partner endpoints. External partners process the event asynchronously on their custom infrastructure and send execution status back via Cocoon's callback REST API. Additionally, partners MAY host an endpoint capable of responding to on-demand status queries from Cocoon to enable manual status checks from the build dashboard, eliminating automated polling loops. + +Below is a high-level flow diagram of the proposed communication between Cocoon and Partner CI: + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + actor GitHub as GitHub Repo + participant Cocoon as Cocoon Server + participant Partner as Partner CI + destroy GitHub + GitHub->>Cocoon: Postsubmit Event + activate Cocoon + Note over Cocoon: Creates Partner task for uid=abcd + Cocoon->>PubSub: Partner notification for uid=abcd + deactivate Cocoon + PubSub->>Cocoon: Partner notification for uid=abcd + activate Cocoon + Cocoon->>Partner: HTTP POST (signed with cocoon key
JWT body with notification for uid=abcd) + activate Partner + Partner-->>Cocoon: Response 200 + deactivate Cocoon + Note over Partner: Executes custom tests + Partner->>Cocoon: HTTP POST Callback (signed with partner key
JWT body with status for uid=abcd) + activate Cocoon + Cocoon->>PubSub: Partner test execution status for uid=abcd + Cocoon-->>Partner: Response 200 + deactivate Cocoon + deactivate Partner + PubSub->>Cocoon: Partner test execution status for uid=abcd + activate Cocoon + Note over Cocoon: Update Partner task status for uid=abcd + deactivate Cocoon +``` + +### 3.2. End-to-End Workflows + +#### 3.2.1. Internal Processing of Post-Submit Event + +1. GitHub notifies Cocoon via `/api/github/webhook-subscription` when a PR with `sha=wxyz` is merged. +2. Cocoon schedules tasks on LUCI and creates records in the `tasks` collection in Firestore for each active (`enabled == true`) partner configuration matching the repository `slug` and target `branch`. +3. For every matching partner configuration, Cocoon: + * Creates a record in the `partner_ci` collection in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, + * Publishes a message containing the `uid` and commit context (`sha`, `branch`, `pr`, `slug`, etc.) to an internal GCP Pub/Sub topic. + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + actor GitHub as GitHub Repo + participant Cocoon as Cocoon Server + participant Firestore@{"type":"database"} + destroy GitHub + GitHub->>+Cocoon: Postsubmit Event + Cocoon->>Firestore: Creates records in `tasks` and `partner_ci` collections
for sha=wxyz with uid=abcd,
partner name, attempt number, etc. + Cocoon->>-PubSub: Schedule partner notification
event for sha=wxyz with uid=abcd +``` + +#### 3.2.2. Cocoon Calls Partner Webhook to Notify About Post-Submit to Run CI, Retrying on Failures + +1. A Pub/Sub push subscriber endpoint on Cocoon: + * receives the partner notification event, + * fetches partner configurations, + * generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, + * issues an HTTP POST webhook request to the partner's configured URL with a strict **30-second HTTP timeout** (partners MUST acknowledge receipt asynchronously before running tests). +2. If webhook invocation times out or the partner server returns `5xx` (Server Error), `408` (Request Timeout), or `429` (Too Many Requests), Cocoon MUST NOT acknowledge the Pub/Sub message and will retry the dispatch. +3. If a partner webhook responds with success code `200`, Cocoon acknowledges the message. +4. If a partner webhook responds with client error code `4xx` (except `408 Request Timeout` and `429 Too Many Requests`), Cocoon logs it as an error. Status code `409` (Conflict) is treated as an already processed notification and logged as a warning. In both cases, Cocoon acknowledges the message and does not retry the dispatch. + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + participant Cocoon as Cocoon Server + participant Partner as Partner CI + + PubSub->>+Cocoon: Receive partner notification
event for sha=wxyz with uid=abcd + Cocoon->>+Partner: HTTP POST (signed with cocoon key,
JWT body with notification for sha=wxyz with uid=abcd) + Partner->>Cocoon: HTTP GET to Cocoon JWKS url if missed in cache + Cocoon-->>Partner: JWKS to verify JWT signature + Note over Partner: Internal Failure during processing the request + Partner-->>-Cocoon: Response 500 + Cocoon-->>-PubSub: nack + + PubSub->>+Cocoon: Retry partner notification
event for sha=wxyz with uid=abcd + Cocoon->>+Partner: Retry HTTP POST (signed with cocoon key,
JWT body with notification for sha=wxyz with uid=abcd) + Partner->>Cocoon: HTTP GET to Cocoon JWKS url if missed in cache + Cocoon-->>Partner: JWKS to verify JWT signature + Partner-->>Cocoon: Response 200 + Cocoon-->>-PubSub: ack + Note over Partner: Start executing custom tests +``` + +#### 3.2.3. Optionally, Partner May Notify Cocoon It Started Test Execution and Provide Test Details + +1. When the partner CI begins test execution for `sha=wxyz` (`uid=abcd`), it MAY send a signed JWT callback to Cocoon with `in_progress` status along with scheduled test names and links to execution details. +2. Cocoon verifies the JWT signature (fetching the partner's JWKS on cache miss), publishes the status to Pub/Sub, responds with `200`, and updates the task status in Firestore to `in_progress` (if the task has not already reached a terminal state, guarding against out-of-order Pub/Sub delivery). +3. This displays the scheduled test names and links to execution details in the Cocoon build dashboard. + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + participant Cocoon as Cocoon Server + participant Partner as Partner CI + + Partner->>+Cocoon: HTTP POST Callback (signed with partner key, JWT body
with `in_progress` status for sha=wxyz with uid=abcd) + Cocoon->>Partner: HTTP GET to Partner JWKS url if missed in cache + Partner-->>Cocoon: JWKS to verify JWT signature + Cocoon->>PubSub: Partner test execution status
for sha=wxyz with uid=abcd + Cocoon-->>-Partner: Response 200 + + PubSub->>+Cocoon: Partner test execution status
for sha=wxyz with uid=abcd + Note over Cocoon: Update Partner task status to `in_progress`. If provided,
display scheduled test names and links to execution details. + deactivate Cocoon +``` + +#### 3.2.4. Optionally, If Supported by Partner, Cocoon Can Poll for Status Updates + +Partners MAY host a status endpoint (e.g., `GET /status?uid=abcd`). + +1. In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard while waiting for execution results. +2. Cocoon makes a `GET` request to the partner's status endpoint, fetches the latest status, and updates the task state accordingly. + +**Note:** Automated polling is strictly rejected to avoid DoS risks and operational overhead. + +```mermaid +sequenceDiagram + participant Cocoon as Cocoon Server + participant Partner as Partner CI + + Note over Cocoon: User requested task status while waiting execution results + Cocoon->>Partner: HTTP GET /status?uid=abcd + Partner-->>Cocoon: Response 200 {"status": "in_progress"} + + Note over Partner: Continue executing custom tests +``` + +#### 3.2.5. Partner Calls Cocoon Webhook to Provide CI Run Status Update, Retrying on Failures + +1. Once the partner CI finishes executing tests on proprietary hardware for `sha=wxyz` (`uid=abcd`), it makes an HTTP POST callback to Cocoon signed with the partner's key containing the terminal execution status (`cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), summary message, and test URLs. + * If the callback request experiences a response timeout or receives a retryable failure response (`5xx`, `408 Request Timeout`, or `429 Too Many Requests`), the partner CI SHOULD retry the HTTP POST callback for `uid=abcd`. + * Upon receiving the callback, Cocoon reads the Key ID (`kid`) from the JWT header and looks up the corresponding cached public key. If the public key is not found in cache, it fetches the partner's JWKS to update the cache, or responds with `401` if the key is not found or JWT signature validation fails. + * If Cocoon successfully validates the payload and publishes the partner test execution status to Pub/Sub, it responds with status code `200`. + * If Cocoon fails to process the request, it responds with a non-retryable error status code, such as `400` for a validation error, `404` if the `uid` is not found, `409` if a terminal status was already processed, or `413` if the body size exceeds 768KB (to stay within Firestore's 1 MiB document size limit). +2. Once Cocoon receives the partner test execution status for `sha=wxyz` (`uid=abcd`) from Pub/Sub, it executes an atomic Firestore transaction that re-verifies `partner_ci.processed == false` before setting `partner_ci.processed = true` and updating the partner task state in `tasks` (idempotently discarding and acknowledging duplicate concurrent messages). + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + participant Cocoon as Cocoon Server + participant Partner as Partner CI + + Note over Partner: Finishes executing custom tests + Partner->>Cocoon: HTTP POST Callback (signed with partner key,
JWT body with status for sha=wxyz with uid=abcd) + Note over Partner: Response timeout + + Partner->>+Cocoon: Retry HTTP POST (signed with partner key,
JWT body with status for sha=wxyz with uid=abcd) + Cocoon->>Partner: HTTP GET to Partner JWKS url if missed in cache + Partner-->>Cocoon: JWKS to verify JWT signature + Cocoon->>PubSub: Partner test execution status
for sha=wxyz with uid=abcd + Cocoon-->>-Partner: Response 200 + + PubSub->>+Cocoon: Partner test execution status
for sha=wxyz with uid=abcd + Note over Cocoon: Update Partner task status for sha=wxyz with uid=abcd + deactivate Cocoon +``` + +#### 3.2.6. If Test Failed, Cocoon May Request Re-run of Failed Task + +1. When a task re-run is initiated for commit `sha=wxyz` (either automatically by Cocoon's flake/retry policy or manually from the build dashboard), Cocoon generates a new unique correlation ID (`uid=efgh`) for the new attempt while retaining the original commit `sha=wxyz`, and publishes a new dispatch message to Pub/Sub. +2. The Pub/Sub subscriber sends a new signed JWT HTTP POST notification to the Partner CI webhook containing `sha=wxyz` and `uid=efgh`. Issuing a distinct `uid` ensures the partner treats the request as a new execution rather than rejecting it as a duplicate (`409 Conflict`) of `uid=abcd`. +3. The Partner CI verifies the new JWT signature, re-executes the custom tests for `sha=wxyz`, and sends a signed JWT callback to Cocoon reporting the updated status (e.g., `succeeded`) associated with `uid=efgh`. +4. Cocoon verifies the callback signature, processes the status update through Pub/Sub, and updates the task status in Firestore to `succeeded` for `sha=wxyz` (`uid=efgh`). + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + participant Cocoon as Cocoon Server + participant Partner as Partner CI + + opt Cocoon calls partner webhook to re-run failed task + Cocoon->>+Cocoon: Re-run for uid=efgh requested + Cocoon->>PubSub: Schedule partner notification
event for sha=wxyz with uid=efgh + PubSub->>Cocoon: Receive partner notification
event for sha=wxyz with uid=efgh + Cocoon->>+Partner: HTTP POST (signed with cocoon key,
JWT body with notification for sha=wxyz with uid=efgh) + Partner->>Cocoon: HTTP GET to Cocoon JWKS url if missed in cache + Cocoon-->>Partner: JWKS to verify JWT signature + Partner-->>Cocoon: Response 200 + Cocoon-->>-PubSub: ack + end + + Note over Partner: Executing custom tests + + opt Partner calls cocoon webhook to provide CI run status update + Partner->>+Cocoon: HTTP POST Callback (signed with partner key,
JWT body with status for sha=wxyz with uid=efgh) + Cocoon->>Partner: HTTP GET to Partner JWKS url if missed in cache + Partner-->>Cocoon: JWKS to verify JWT signature + Cocoon->>PubSub: Partner test execution status
for sha=wxyz with uid=efgh + Cocoon-->>-Partner: Response 200 + deactivate Partner + + PubSub->>+Cocoon: Partner test execution status
for sha=wxyz with uid=efgh + Note over Cocoon: Update Partner task status for sha=wxyz with uid=efgh + deactivate Cocoon + end +``` + +### 3.3. Pros & Cons + +* **Pros:** + * Industry-standard webhook architecture that is easy for partners to consume. + * Push-based trigger is real-time and minimizes latency. + * Decouples Flutter core infrastructure from partner device management. + * Stateless design on Cocoon via Firestore and GCP Pub/Sub. + * Status query endpoints allow robust reconciliation on-demand without automated polling loops. +* **Cons:** + * Requires partners to maintain publicly accessible HTTP endpoints and handle inbound requests. + +--- + +## 4. Design + +### 4.1. Security & Authentication + +This section details the decentralized security model used to authorize server-to-server bidirectional webhook communication without a centralized authorization server. + +#### 4.1.1. Asymmetric Cryptography & Key Management + +Both the Cocoon server and the Partner CI server independently generate their own asymmetric key pairs (using [Ed25519](https://www.rfc-editor.org/info/rfc8410/)). Each system stores its private key in a secure key management system and shares its corresponding public key with the other party. + +By leveraging decentralized asymmetric cryptography, this approach intentionally avoids the need for a central Authorization Server or Identity Provider, eliminating a potential single point of failure and reducing infrastructure complexity. + +#### 4.1.2. Key Distribution Strategy: Public Key Endpoints + +For this authentication model to work, both parties MUST reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of the JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to cache and refresh this key set periodically. This allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. + +Example of JWKS: + +#### 4.1.3. Serialized JWT HTTP Request Format + +When either system sends a webhook or callback HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender MUST construct a `POST` request as follows: + +* **Header**: `Content-Type: application/jwt` +* **Body**: A signed JSON Web Token (JWT) + +For manual on-demand status queries (`GET /status?uid=`), Cocoon passes the signed JWT in the `Authorization: Bearer ` header with `"aud"` set to the partner's configured `status_url` and `"data": {"uid": ""}`. + +#### 4.1.4. JWT Payload Structure + +The JWT consists of a base64url-encoded header, payload, and signature separated by dots. +For example: + +`eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6IjEyMzQ1In0.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovL3BhcnRuZXItY2kuZXhhbXBsZS5jb20vd2ViaG9vayIsImlhdCI6MTc3NDk2MjAwMCwiZXhwIjoxNzc0OTYyMzAwLCJkYXRhIjp7InVpZCI6IjBzQXBMaHdibUd6Mlkwd0JEelBMIiwic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsImJyYW5jaCI6Im1hc3RlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` + +#### 4.1.5. JWT Header + +```json +{ + "alg": "EdDSA", + "typ": "JWT", + "kid": "12345" +} +``` + +Where: + +* **alg**: The cryptographic algorithm used to sign the token (`EdDSA` for [Ed25519](https://www.rfc-editor.org/info/rfc8410/) signatures). +* **typ**: The type of the token, which MUST be set to `JWT`. +* **kid**: Key ID string that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. + +#### 4.1.6. JWT Payload + +Tokens are generated and signed per HTTP request attempt with a short-lived expiration window (e.g., 5 minutes): + +```jsonc +{ + // --- Standard Security Metadata --- + "iss": "cocoon", + "aud": "https://partner-ci.example.com/webhook", + "iat": 1774962000, + "exp": 1774962300, + // --- The Actual Message (Custom Claims) --- + "data": { + "uid": "0sAtLhwbmGz2Y0WBDzPL", + "slug": "flutter/flutter", + "branch": "master", + "pr": 12345, + "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b" + } +} +``` + +#### 4.1.7. JWT Signature + +[Ed25519 signature](https://www.rfc-editor.org/info/rfc8037/#appendix-A.4) as defined in [RFC 8037](https://www.rfc-editor.org/info/rfc8037). + +#### 4.1.8. Validating Incoming Requests + +Webhook listeners MUST validate incoming requests in accordance with standard JWT/JWS/JWKS specifications and security best practices, specifically: + +* [RFC 7519 Section 7.2](https://www.rfc-editor.org/info/rfc7519/#section-7.2) for JWT validation; +* [RFC 7515 Section 5.2](https://www.rfc-editor.org/info/rfc7515/#section-5.2) for signature verification; +* [RFC 7517 Section 4.5](https://www.rfc-editor.org/info/rfc7517/#section-4.5) & [Section 5](https://www.rfc-editor.org/info/rfc7517/#section-5) for `kid` lookup via cached JWKS; +* [RFC 8725 Section 3](https://www.rfc-editor.org/info/rfc8725/#section-3) for algorithm pinning (`EdDSA`) and `iss`/`aud`/`exp` claim checks. + +In addition, Cocoon's callback endpoint validates that `data.uid` exists, has not processed a terminal status yet, and belongs to a partner that is listed in the `iss` claim. + +#### 4.1.9. Outbound Request Security & SSRF Prevention + +Because Cocoon makes outbound HTTP requests to partner-supplied endpoints and renders partner-supplied log URLs on the public Flutter build dashboard, Cocoon MUST enforce the following security controls: + +* **HTTPS-Only Scheme Validation:** All URLs configured in `partner_config` (`webhook_url`, `status_url`, `keys_url`) and all `details[].url` entries submitted in status callbacks MUST use the `https://` scheme. Plaintext `http://`, `javascript:`, `data:`, and other non-HTTPS schemes MUST be rejected with `400 Bad Request`. +* **SSRF Protection:** Cocoon's outbound HTTP client MUST validate resolved IP addresses upon DNS resolution and block connections to loopback (`127.0.0.0/8`, `::1/128`), private (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), and link-local (`169.254.0.0/16`, `fe80::/10`, including the GCP metadata server `169.254.169.254`) address ranges, and MUST NOT follow HTTP redirects to disallowed hosts or schemes. +* **Outbound Response Size & Timeout Limits:** Outbound HTTP responses from partner endpoints MUST be capped at 64 KB for webhook acknowledgments and JWKS responses (with a 30-second timeout for webhooks and 3-second timeout for JWKS) and 768 KB for manual status query responses (with a 30-second timeout). + +### 4.2. API Contracts & Status State Machine + +See [Section 8 (Appendix A)](#8-appendix-a-usage-examples) for full HTTP wire examples of each endpoint. + +* **Outbound Webhook Dispatch (`POST `):** Dispatched by Cocoon with `data` claims `{uid, slug, branch, pr, sha}`. +* **Inbound Status Callback (`POST /api/partner/callback`):** Called by Partner CI with `data` claims `{uid, status, message, details}`. Maximum request body size is 768KB. +* **Manual On-Demand Status Query (`GET ?uid=`):** Called by Cocoon with `Authorization: Bearer ` when a user manually requests status reconciliation from the dashboard; returns a JSON object `{status, message, details}`. + +**Allowed Status Values (`status` & `details[].status`):** + +* `in_progress` — Task or test is currently executing. Used only in optional start-execution callbacks ([Section 3.2.3](#323-optionally-partner-may-notify-cocoon-it-started-test-execution-and-provide-test-details)) and manual status responses ([Section 3.2.4](#324-optionally-if-supported-by-partner-cocoon-can-poll-for-status-updates)). MUST NOT be sent in terminal status callbacks. +* **Terminal Failure States:** `cancelled`, `infra_failure`, `failed`. +* **Terminal Success States:** `succeeded`, `neutral`, `skipped`. + +Once a task transitions to any terminal state, the `partner_ci_status_sub` handler atomically sets `partner_ci.processed = true` and updates `tasks` inside a single Firestore transaction (rejecting or discarding concurrent duplicate deliveries), and subsequent HTTP callbacks for the same `uid` are rejected with `409 Conflict`. + +### 4.3. Data Model (Firestore) + +To support webhook flows, the following Firestore collections and schema extensions are implemented in Cocoon: + +#### 4.3.1. partner_config Collection + +For partner configuration, add a `partner_config` collection with Document ID equal to `task_name`: + +```go +slug string // github repo (e.g. flutter/flutter) to trigger on +branches []string // target branch patterns (e.g. ["master", "beta", "stable"]) to trigger on +task_name string // Name of post submit task to display on dashboard +partner string // 3p partner name (must match JWT iss claim on callbacks) +enabled bool // Whether this partner integration is active (emergency kill-switch) +webhook_url string // HTTPS url to call to request CI +status_url string // Optional manual HTTPS status request url +keys_url string // HTTPS url to read partner key id to public key mapping (JWKS) +``` + +Currently, there will be no UI for editing this collection. For every post-submit event, dashboard checks will be extended with active (`enabled == true`) tasks from this collection matching the event's `slug` and `branch`. All tasks from this collection will be created suppressed without the ability to unsuppress. Later, if a decision is made to let some 1P partner CI block the tree, the `partner_config` collection MAY be extended with a `block_tree` boolean field. + +#### 4.3.2. partner_ci Collection + +Add a `partner_ci` collection with Auto-ID Document ID equal to the `uid` of the `data` payload. This is used to match post-submit tasks to corresponding status responses from partner CI systems: + +```go +commit_sha string // Head sha at a moment of check-run creation ie commit sha +slug string // repository_owner/repository_name (e.g. flutter/flutter) +branch string // Target branch name (e.g. master) +partner string // 3p partner name +task_name string // Name of post submit task to display on dashboard +attempt_number int // Number of re-run attempt +processed bool // Whether terminal status response from partner was already processed +``` + +#### 4.3.3. tasks Collection + +Extend the `tasks` collection with `message` and `details` fields. + +### 4.4. Asynchronous Messaging & Retries (Pub/Sub) + +#### 4.4.1. partner_ci_notification Topic + +To guarantee that Cocoon can notify each partner CI even during network outages or temporary partner infrastructure outages, once Cocoon reads the active post-submit task configurations for a post-submit event, it publishes a message to the `partner_ci_notification` topic for every matching partner integration. Cocoon consumes messages from `partner_ci_notification_sub` via a **GCP Pub/Sub Push subscription** (invoking an internal Cocoon push handler endpoint) and attempts to `POST` each notification to the partner webhook (with a 30-second HTTP timeout) using Pub/Sub exponential backoff on retryable failures (`5xx`, `408`, `429`, or network timeouts). + +To prevent head-of-line blocking ("noisy neighbor" exhaustion) when a single partner endpoint is degraded or timing out: + +* Cocoon enforces a per-partner circuit breaker / concurrency cap so a failing partner endpoint cannot exhaust push worker capacity for healthy partners. +* `partner_ci_notification_sub` is configured with exponential backoff (minimum 60 seconds, maximum 600 seconds), `max_delivery_attempts = 10`, and a **Dead-Letter Topic (`partner_ci_notification_dlq`)**. If a notification exhausts all delivery attempts and is routed to `partner_ci_notification_dlq`, a dead-letter handler marks the corresponding task in Firestore as `infra_failure` with a summary message indicating webhook delivery failure. + +#### 4.4.2. partner_ci_status Topic + +To guarantee that Cocoon can reliably process status updates once the status callback webhook (`/api/partner/callback`) is called by a partner and the payload is successfully validated, the payload is published to the `partner_ci_status` topic and consumed by Cocoon via a **GCP Pub/Sub Push subscription (`partner_ci_status_sub`)**. The push handler executes an idempotent Firestore transaction to update `partner_ci` and the corresponding task status in `tasks`. Unrecoverable messages after `max_delivery_attempts = 10` are routed to a **Dead-Letter Topic (`partner_ci_status_dlq`)** for alerting and inspection. + +--- + +## 5. Alternatives Considered + +### 5.1. Alternative 1: Event-Driven GCP Pub/Sub Integration (Asynchronous Event Bus) + +This architecture eliminates direct server-to-server HTTP calls by leveraging Google Cloud Pub/Sub as an event bus between Cocoon and partner infrastructure. + +#### 5.1.1. End-to-End Workflow Diagram + +```mermaid +sequenceDiagram + actor GitHub as GitHub Repo + participant Cocoon as Cocoon Server + participant PubSub@{"type":"queue"} + participant Partner as Partner CI + + destroy GitHub + GitHub->>Cocoon: Postsubmit Event + Note over Cocoon: Creates Partner task + Cocoon->>PubSub: Publish Event + PubSub->>Partner: Partner Ingestion (JWT+Service Account) + Note over Partner: Executes custom tests + Partner->>PubSub: Publish Results (JWT+Service Account) + PubSub->>Cocoon: Update Partner task status + Note over Cocoon: Update Partner task status +``` + +#### 5.1.2. Workflow + +* **1. Trigger:** GitHub triggers post-commit processing in Cocoon. +* **2. Outbound Event:** Cocoon schedules the partner task and publishes an event to an outbound Pub/Sub topic. +* **3. Partner Ingestion:** Partners consume messages from the topic via cross-project GCP IAM subscription or delegation. +* **4. Execution:** Partner CI processes the event on local hardware. +* **5. Inbound Result:** Partner CI publishes results directly back to Cocoon's inbound Pub/Sub topic. +* **6. Dashboard Update:** Cocoon ingests result messages and updates test execution states in the dashboard. + +#### 5.1.3. Pros & Cons + +* **Pros:** + * Highly asynchronous and resilient; messages are automatically buffered during outages. + * No inbound HTTP ports are required on the partner side; can run behind strict firewalls. + * Removes webhook management, network connection timeouts, and custom HTTP retry logic. +* **Cons:** + * High administration overhead: requires managing cross-project GCP permissions or GCP service account keys for partners. + * Introduces cloud vendor lock-in (GCP) for third-party partners. + * Increases setup and development complexity for partners. + * Cannot validate message body at the time of message submission. + +### 5.2. Alternative 2: Automated Polling Strategy (Push Trigger, Pull Status) + +This hybrid model retains HTTP push for triggering partner CI jobs, but replaces inbound callbacks with periodic polling by Cocoon to check test completion. + +#### 5.2.1. End-to-End Workflow Diagram + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + actor GitHub as GitHub Repo + participant Cocoon as Cocoon Server + participant Partner as Partner CI + destroy GitHub + + GitHub->>Cocoon: Postsubmit Event + Note over Cocoon: Creates Partner task + Cocoon->>PubSub: Schedule partner notification
event for sha=wxyz with uid=abcd + PubSub->>Cocoon: Receive partner notification
event for sha=wxyz with uid=abcd + Cocoon->>Partner: HTTP POST (signed with cocoon key,
JWT body with notification for uid=abcd) + Partner-->>Cocoon: Response 200 + Cocoon-->>PubSub: ack + Cocoon->>PubSub: Schedule pull partner CI status event
for sha=wxyz with uid=abcd + + loop Polling by using PubSub retry + PubSub->>Cocoon: Receive pull partner CI status event
for sha=wxyz with uid=abcd + Note over Partner: Executes custom tests + Cocoon->>Partner: GET /status?uid=abcd + Partner-->>Cocoon: Response 202, Status: RUNNING + Cocoon-->>PubSub: nack + end + + PubSub->>Cocoon: Receive pull partner CI status event
for sha=wxyz with uid=abcd + Cocoon->>Partner: GET /status?uid=abcd + Partner-->>Cocoon: Response 200, Status: COMPLETED + Cocoon-->>PubSub: ack + Cocoon->>PubSub: Schedule Partner test execution status
for sha=wxyz with uid=abcd + PubSub->>Cocoon: Receive Partner test execution status
for sha=wxyz with uid=abcd + Note over Cocoon: Update Partner task status + Cocoon-->>PubSub: ack +``` + +#### 5.2.2. Workflow + +* **1. Trigger & Dispatch:** GitHub triggers Cocoon, which creates partner tasks and pushes a webhook notification to the partner. +* **2. Execution:** The partner CI executes workloads asynchronously. +* **3. Polling Loop:** Cocoon automatically queries the partner's status endpoint periodically (e.g., using Pub/Sub retry loop). +* **4. Inbound Processing:** Once the status endpoint returns a terminal state (`PASSED`/`FAILED`), Cocoon updates the dashboard. + +#### 5.2.3. Pros & Cons + +* **Pros:** + * Prevents partners from needing inbound access or security tokens to write to Cocoon endpoints. + * Simpler implementation for partners with pre-existing REST APIs that expose build status. +* **Cons:** + * High network and resource overhead from frequent, automated polling across numerous partners. + * Increases latency in reporting status updates back to the Flutter dashboard due to polling intervals. + * Scale limitations and risk of denial-of-service (DoS) on partner endpoints under heavy load. + +### 5.3. Comparison of Options + +| Approach Criteria | Primary Design: Webhook Push-Callback | Alternative 1: Event-Driven GCP Pub/Sub | Alternative 2: Automated Polling (Push-Poll) | +| --- | --- | --- | --- | +| **Communication Model** | ✅ Bidirectional HTTP webhooks with manual query backup. | ✅ Asynchronous Event Bus (Pub/Sub). | ✅ HTTP push trigger, automated polling by Cocoon. | +| **Partner Setup Complexity** | ⚠️ **Low to moderate.** Webhooks are easy to consume, but partners must implement retry logic for outbound webhook calls. | ❌ **High.** Requires managing cross-project GCP permissions or GCP service account keys. | ✅ **Low.** Simpler for partners to only set up webhook and status endpoints. | +| **Cloud Vendor Lock-in** | ✅ **Low.** Built on standard REST and HTTP/JWT protocols. | ❌ **High.** Introduces GCP cloud vendor lock-in for third-party partners. | ✅ **Low.** Built on standard REST and HTTP/JWT protocols. | +| **Security & Authentication** | ✅ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | ⚠️ **Secure.** Managed via GCP IAM and service accounts. | ✅ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | +| **Status response validation** | ✅ Can validate payload schema, unknown/duplicated `uid` and notify partner with HTTP response codes. | ⚠️ Can only validate and notify partner about payload schema violation. | ❌ Cannot notify partner of any payload schema or data validation issues. | +| **Latency / Status Reporting** | ✅ **Real-time.** Push-based callback minimizes status reporting latency. | ✅ **Real-time.** Push-based consumption minimizes latency. | ❌ **Higher latency.** Updates are delayed based on Cocoon's polling intervals. | +| **Resilience & Overhead** | ✅ **Highly resilient.** Retries are backed by Pub/Sub; Status query endpoint prevents polling loops. | ✅ **Highly resilient.** Messages are automatically buffered during partner infrastructure outages. | ❌ **High overhead.** Constant automated polling creates network load and risks DoS on partner endpoints. | + +--- + +## 6. Testing, Rollout & Documentation Plan + +### 6.1. Testing Plan + +* **Unit Tests:** Validate Ed25519 JWT generation and signature verification, JWKS `kid` lookup and cache-miss refresh cooldown logic, SSRF/HTTPS URL validation, payload schema validation, and task status state transitions (including idempotency and out-of-order `in_progress` vs. terminal status handling). +* **Integration Tests:** Verify Firestore (`partner_config`, `partner_ci`, `tasks`) atomic transactions and Pub/Sub (`partner_ci_notification`, `partner_ci_status`, and dead-letter topics) message publishing, acknowledgment, and retry behavior. +* **End-to-End Tests:** Implement a dummy partner CI service to test the complete post-submit dispatch, `in_progress` callback, manual status reconciliation, terminal callback, and task re-run workflows. + +### 6.2. Rollout & Rollback Plan + +* **Phased Rollout:** + 1. **Dummy partner CI validation:** Deploy the Pub/Sub topics, Firestore schema changes, and Cocoon endpoints using the dummy partner CI service. + 2. **Pilot Onboarding:** Onboard a single first-party or pilot partner in production (`enabled = true` in `partner_config`) and monitor webhook delivery latency, callback success rates, and dashboard rendering for 2 weeks. + 3. **General Availability:** Gradually onboard additional partners by adding documents to `partner_config`. +* **Emergency Kill-Switch & Rollback:** + * Setting `enabled = false` on any document in `partner_config` immediately stops Cocoon from scheduling new tasks or dispatching webhooks for that partner without requiring a code deployment or deleting configuration state. + * Because all partner tasks are non-blocking (created suppressed), disabling or rolling back the feature has zero impact on Flutter tree health or PR merge velocity. + +### 6.3. Observability, Privacy & Data Retention + +* **Metrics & Alerting:** Cocoon will emit Cloud Monitoring metrics partitioned by `partner` and `task_name`: + * Outbound webhook dispatch count, HTTP status code distribution, and latency. + * Inbound callback request count, HTTP status code distribution, and JWT verification failure count. + * On-demand JWKS refresh count, latency, and failure rate. + * Dead-letter queue (`partner_ci_notification_dlq` and `partner_ci_status_dlq`) message counts. +* **Operational Ownership:** Flutter EngProd monitors Cocoon pipeline health and Pub/Sub DLQs; individual partner teams are responsible for monitoring and triaging their own CI infrastructure and test failures. +* **Privacy & Data Retention:** + * Because the Flutter build dashboard is public, partner callback payloads (`message` and `details[].url`) MUST NOT contain Personally Identifiable Information (PII), internal hostnames, or confidential partner secrets. + +### 6.4. Partner Onboarding & Documentation Plan + +Provide partner integration documentation detailing: + +* Generating Ed25519 key pairs and hosting a public JWKS endpoint. +* Implementing the inbound webhook listener and JWT verification. +* Implementing the outbound status callback with exponential backoff retries on `5xx`, `408`, and `429` responses. +* Optionally hosting the manual on-demand `GET /status` endpoint. + +--- + +## 7. Open Questions + +* What other fields should Cocoon provide to the partner webhook to request a CI run? +* Can the `tasks` collection primary key `__` be used as `uid`? In that case Cocoon would not need a separate `partner_ci` collection. +* How should Cocoon handle partner tasks that never receive a terminal status callback (e.g., marking tasks stuck in `New` or `in_progress` as `infra_failure` after a configurable timeout)? +* Do we need to provide branch name (e.g. stable or master) to the partner webhook? Or just the SHA and task name? Technically the branch name could be derived from the sha. + +--- + +## 8. Appendix A. Usage Examples + +### 8.1. Communication Example: Cocoon and 3rd Party Partner + +#### 8.1.1. Webhook Dispatch Request (Cocoon -> Partner) + +When a post-submit event is processed, Cocoon posts a signed JWT to the partner's webhook URL. +See: [JSON Web Token (JWT) RFC 7519](https://www.rfc-editor.org/info/rfc7519/) for more details about JWT. + +##### 8.1.1.1. HTTP POST Request + +```http +POST /webhook HTTP/1.1 +Host: partner-ci.example.com +Content-Type: application/jwt + +.. +``` + +See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc7515/) for more details about Signature (JWS). + +##### 8.1.1.2. Decoded JWT Payload + +```json +{ + "iss": "cocoon", + "aud": "https://partner-ci.example.com/webhook", + "iat": 1774962000, + "exp": 1774962300, + "data": { + "uid": "0sAtLhwbmGz2Y0WBDzPL", + "slug": "flutter/flutter", + "branch": "master", + "pr": 12345, + "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b" + } +} +``` + +Where: + +* **iss**: Issuer claim identifying the caller that issued the token (e.g., `cocoon`). +* **aud**: Audience claim specifying the intended recipient URL of the request. +* **iat**: Issued At timestamp (Unix epoch time) indicating when the token was created. +* **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. +* **data**: Object containing custom payload claims and event details: + * **uid**: The unique correlation identifier to match asynchronous CI execution request to future status response. + * **slug**: The target repository identifier (e.g., `flutter/flutter`). + * **branch**: The target git branch name (e.g., `master`). + * **pr**: The Pull Request number associated with the event. + * **sha**: The target git commit SHA hash. + +##### 8.1.1.3. HTTP Response + +```http +HTTP/1.1 200 OK +Content-Type: application/json + +{} +``` + +#### 8.1.2. Status Callback Request (Partner -> Cocoon) + +After executing custom test suites on proprietary hardware, the partner sends the final status back to Cocoon. + +##### 8.1.2.1. HTTP POST Request + +```http +POST /api/partner/callback HTTP/1.1 +Host: cocoon.flutter.dev +Content-Type: application/jwt + +.. +``` + +##### 8.1.2.2. Decoded Payload + +```json +{ + "iss": "3p-partner-ci", + "aud": "https://cocoon.flutter.dev/api/partner/callback", + "iat": 1774963500, + "exp": 1774963800, + "data": { + "uid": "0sAtLhwbmGz2Y0WBDzPL", + "status": "failed", + "message": "Some optional summary if needed", + "details": [ + { + "name": "test_1", + "status": "failed", + "url": "https://partner-ci.example.com/test_1/log" + }, + { + "name": "test_2", + "status": "succeeded", + "url": "https://partner-ci.example.com/test_2/log" + }, + { + "name": "test_3", + "status": "skipped", + "url": "https://partner-ci.example.com/test_3/log" + }, + { + "name": "test_4", + "status": "failed", + "url": "https://partner-ci.example.com/test_4/log" + }, + { + "name": "test_5", + "status": "cancelled", + "url": "https://partner-ci.example.com/test_5/log" + } + ] + } +} +``` + +Where: + +* **iss**: Issuer claim identifying the caller that issued the token (e.g., partner). +* **aud**: Audience claim specifying the intended recipient URL of the request. +* **iat**: Issued At timestamp (Unix epoch time) indicating when the token was created. +* **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. +* **data**: Object containing custom payload claims and event details: + * **uid**: The unique correlation identifier to match which CI execution request this status response belongs to. + * **status**: + * `in_progress` - Task is in progress. Used only for optional `in_progress` status notification. Cannot be in terminal status notification. + * `cancelled` - Task is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during tests execution (failure state). + * `failed` - Some or all of the tests are failed (failure state). + * `succeeded` - All tests passed (success state). + * `neutral` - Only some flaky tests failed (success state). + * `skipped` - Some or all tests are skipped (success state). + * **message**: Optional Markdown formatted message with a summary of CI run. Expected to be present when status is `infra_failure` or `failed`. + * **details**: Granular test execution details. + * **name**: Test name. + * **status**: + * `in_progress` - Test is in progress. Used only for optional `in_progress` status notification. Cannot be in terminal status notification. + * `cancelled` - Test is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during test execution (failure state). + * `failed` - Test failed (failure state). + * `succeeded` - Test passed (success state). + * `neutral` - Test failed but considered flaky (success state). + * `skipped` - Test skipped (success state). + * **url**: The publicly accessible task execution details or logs URL. + +##### 8.1.2.3. HTTP Response + +```http +HTTP/1.1 200 OK +Content-Type: application/json + +{} +``` + +#### 8.1.3. Manual On-Demand Status Reconciliation Request + +For on-demand execution triggered by Tree Gardeners, Cocoon can manually query the partner's status endpoint. + +##### 8.1.3.1. HTTP GET Request + +```http +GET /status?uid=0sAtLhwbmGz2Y0WBDzPL HTTP/1.1 +Host: partner-ci.example.com +Authorization: Bearer +``` + +##### 8.1.3.2. HTTP Response + +```http +HTTP/1.1 200 OK +Content-Type: application/json + +{ + "status": "in_progress", + "message": "Some optional summary if needed", + "details": [ + { + "name": "test_1", + "status": "failed", + "url": "https://partner-ci.example.com/test_1/log" + }, + { + "name": "test_2", + "status": "succeeded", + "url": "https://partner-ci.example.com/test_2/log" + }, + { + "name": "test_3", + "status": "skipped", + "url": "https://partner-ci.example.com/test_3/log" + }, + { + "name": "test_4", + "status": "failed", + "url": "https://partner-ci.example.com/test_4/log" + }, + { + "name": "test_5", + "status": "in_progress", + "url": null + } + ] +} +``` + +Where: + +* **status**: The status of CI execution: + * `in_progress` - Partner CI starts the execution of this task (in progress state). + * `cancelled` - Task is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during test execution (failure state). + * `failed` - Some or all of the tests are failed (failure state). + * `succeeded` - All tests passed (success state). + * `neutral` - Only some flaky tests failed (success state). + * `skipped` - Some or all tests are skipped (success state). +* **message**: Markdown formatted message with a summary of CI run. +* **details**: Granular test execution details: + * **name**: Test name. + * **status**: Current test execution status: + * `in_progress` - Partner CI starts the execution of this task (in progress state). + * `cancelled` - Test is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during test execution (failure state). + * `failed` - Test failed (failure state). + * `succeeded` - Test passed (success state). + * `neutral` - Test failed but considered flaky (success state). + * `skipped` - Test skipped (success state). + * **url**: The publicly accessible task execution details or logs URL.