From 0d6c4f9892c741bda6b8ed15fe77e5d92482fb49 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Tue, 6 Oct 2026 14:01:10 -0700 Subject: [PATCH 01/35] converting design doc into rfc --- rfc/600.0000-flutter-pluggable-ci.md | 658 +++++++++++++++++++++++++++ 1 file changed, 658 insertions(+) create mode 100644 rfc/600.0000-flutter-pluggable-ci.md diff --git a/rfc/600.0000-flutter-pluggable-ci.md b/rfc/600.0000-flutter-pluggable-ci.md new file mode 100644 index 0000000..01184aa --- /dev/null +++ b/rfc/600.0000-flutter-pluggable-ci.md @@ -0,0 +1,658 @@ +--- +type: rfc +rfc: '600.0000' +title: Flutter Pluggable CI Design +description: Describes the integration with partner CI systems to trigger partner CI execution on post-submit PRs and display execution details on the post-submit dashboard. +status: draft +created: 2026-10-06T09:00:00Z +updated: 2026-10-06T09:00:00Z +tags: + - 600-infrastructure +authors: + - '"Dmitry Grand" ' +--- + +# RFC 600.0000: Flutter Pluggable CI Design + +## Problem Statement + +To allow partners who maintain off-tree embedders to detect regression immediately after a commit lands on the flutter main branch by running their CI on every flutter PR post-submit. + +## Background + +Multiple Flutter partners adopt custom embedders to support specific platforms not officially supported by Flutter. Changes in the upstream Flutter repository often cause build failures for custom embedders. Depending on how frequently a partner syncs their fork, isolating the exact breaking change is difficult to manage. This solution mitigates the issue by allowing partners to run their CI pipelines against every Flutter PR post-submit, catching regressions early. We cannot allow partner CI execution to block the tree, at least for now, because gardeners responsible for maintaining tree health lack the knowledge or access to partner codebases and infrastructure necessary to resolve failures effectively. + +### Audience + +* Flutter TLs +* First-party partners + +### Glossary + +* **In-Tree Platforms:** These are core Flutter platforms that are stored, updated, and managed directly inside the main Flutter repository. This includes the official ports for iOS, Android, macOS, Linux, and Windows. +* **Out-of-Tree Platforms:** These are external Flutter platforms developed and maintained by third parties in separate, independent repositories. Because they live outside the main codebase, today developers must fork the Flutter tool and reverse-engineer undocumented system channels (like text input or app lifecycles) to make them work. +* **Custom Embedders:** These are platform-specific codebases that host the Flutter engine and Dart runtime on non-standard operating systems. They act as out-of-tree ports for custom hardware. +* **Cocoon:** Flutter's custom Continuous Integration (CI) orchestration system that integrates underlying CI services like LUCI with GitHub to schedule tests and report build statuses. It also provides developer tooling such as auto-submit mechanisms, test flakiness tracking, and powers the public Flutter Pre-submit/Build Dashboards. +* **LUCI (Layered Universal Continuous Integration):** The underlying execution engine that runs Flutter's testing and build workloads across diverse platforms like macOS, Windows, and Linux. Orchestrated by Cocoon, LUCI executes specific Python-based "recipes" to securely validate code changes for the Flutter framework, engine, and packages during both pre-submit and post-submit pipelines. + +## Overview + +To enable Flutter to scale across external platforms and out-of-tree embedder maintainers, Flutter infrastructure (Cocoon) requires a robust, secure, and decoupled communication model. This system notifies partner CI pipelines of post-submit commits on `flutter/flutter` so partners can execute custom test suites on proprietary hardware and report execution status back to the Flutter build dashboard. + +Key architectural goals include non-blocking test execution, strict security via Ed25519 public/private key authentication, high resilience against network outages, and minimal operational overhead for Flutter infrastructure maintainers. + +The proposed solution is to use webhooks for bidirectional asynchronous communication backed by GCP Pub/Sub for fault tolerance and retries. + +Below is a high-level flow diagram of the proposed communication between Cocoon and Partner CI: + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + actor GitHub as GitHub Repo + participant Cocoon as Cocoon Server + participant Partner as Partner CI + destroy GitHub + GitHub->>Cocoon: Postsubmit Event + activate Cocoon + Note over Cocoon: Creates Partner task for uid=abcd + Cocoon->>PubSub: Partner notification for uid=abcd + deactivate Cocoon + PubSub->>Cocoon: Partner notification for uid=abcd + activate Cocoon + Cocoon->>Partner: HTTP POST (signed with cocoon key
JWT body with notification for uid=abcd) + activate Partner + Partner-->>Cocoon: Response 200 + deactivate Cocoon + Note over Partner: Executes custom tests + Partner->>Cocoon: HTTP POST Callback (signed with partner key
JWT body with status for uid=abcd) + activate Cocoon + Cocoon->>PubSub: Partner test execution status for uid=abcd + Cocoon-->>Partner: Response 200 + deactivate Cocoon + deactivate Partner + PubSub->>Cocoon: Partner test execution status for uid=abcd + activate Cocoon + Note over Cocoon: Update Partner task status for uid=abcd + deactivate Cocoon +``` + +### Non-goals + +This design does not cover what kind of CI for out-of-tree platforms to use and what tests to run to validate custom embedders. Nor does it cover how to guarantee the delivery of CI execution status from partner CI systems back to Cocoon. + +## Usage Examples + +### Communication Example: Cocoon and 3rd Party Partner + +#### 1. Webhook Dispatch Request (Cocoon -> Partner) + +When a post-submit commit is processed, Cocoon posts a signed JWT to the partner's webhook URL. +See: [JSON Web Token (JWT) RFC 7519](https://www.rfc-editor.org/info/rfc7519/) for more details about JWT. + +##### HTTP POST Request: + +```http +POST /webhook HTTP/1.1 +Host: partner-ci.example.com +Content-Type: application/jwt + +.. +``` + +See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc7515/) for more details about Signature (JWS). + +##### Decoded JWT Payload: + +```json +{ + "iss": "cocoon", + "aud": "https://partner-ci.com/webhook", + "iat": 1774962000, + "exp": 1774962300, + "data": { + "uid": "0sAtLhwbmGz2Y0WBDzPL", + "slug": "flutter/flutter", + "pr": 12345, + "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b", + "task_name": "some_awesome_partners_test" + } +} +``` + +Where: + +* **iss**: Issuer claim identifying the caller that issued the token (e.g., `cocoon`). +* **aud**: Audience claim specifying the intended recipient URL of the request. +* **iat**: Issued At timestamp (Unix epoch time) indicating when the token was created. +* **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. +* **data**: Object containing custom payload claims and event details: + * **uid**: The unique correlation identifier to match asynchronous CI execution request to future status response. + * **slug**: The target repository identifier (e.g., `flutter/flutter`). + * **pr**: The Pull Request number associated with the event. + * **sha**: The target git commit SHA hash. + * **task_name**: Name of task to be displayed on dashboard, to potentially support several CI types or platforms per partner. For only one task per partner can be ignored. + +##### HTTP Response: + +```http +HTTP/1.1 200 OK +Content-Type: application/json + +{} +``` + +#### 2. Status Callback Request (Partner -> Cocoon) + +After executing custom test suites on proprietary hardware, the partner sends the final status back to Cocoon. + +##### HTTP POST Request: + +```http +POST /api/partner/callback HTTP/1.1 +Host: cocoon.flutter.dev +Content-Type: application/jwt + +.. +``` + +##### Decoded Payload: + +```json +{ + "iss": "3p-partner-ci", + "aud": "https://dashboard.flutter.dev/api/partner/callback", + "iat": 1774963500, + "exp": 1774963800, + "data": { + "uid": "0sAtLhwbmGz2Y0WBDzPL", + "status": "failed", + "message": "Some optional summary if needed", + "details": [ + { + "name": "test_1", + "status": "failed", + "log_url": "https://parnter.com/test_1/log" + }, + { + "name": "test_2", + "status": "success", + "log_url": "https://parnter-ci.com/test_2/log" + }, + { + "name": "test_3", + "status": "skipped", + "log_url": "https://parnter-ci.com/test_3/log" + }, + { + "name": "test_4", + "status": "failed", + "log_url": "https://parnter-ci.com/test_4/log" + }, + { + "name": "test_5", + "status": "cancelled", + "log_url": "https://parnter-ci.com/test_5/log" + } + ] + } +} +``` + +Where: + +* **iss**: Issuer claim identifying the caller that issued the token (e.g., partner). +* **aud**: Audience claim specifying the intended recipient URL of the request. +* **iat**: Issued At timestamp (Unix epoch time) indicating when the token was created. +* **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. +* **data**: Object containing custom payload claims and event details: + * **uid**: The unique correlation identifier to match which CI execution request this status response belongs to. + * **status**: The terminal status of CI execution (cannot be `in_progress`): + * `cancelled` - Task is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during tests execution (failure state). + * `failed` - Some or all of the tests are failed (failure state). + * `succeeded` - All tests passed (success state). + * `neutral` - Only some flaky tests failed (success state). + * `skipped` - Some or all tests are skipped (success state). + * **message**: Markdown formatted message with a summary of CI run. + * **details**: Granular test execution details: + * **name**: Test name. + * **status**: Terminal test status (cannot be `in_progress`): + * `cancelled` - Test is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during test execution (failure state). + * `failed` - Test failed (failure state). + * `succeeded` - Test passed (success state). + * `neutral` - Test failed but considered flaky (success state). + * `skipped` - Test skipped (success state). + * **log_url**: The publicly accessible task execution log URL. + +##### HTTP Response: + +```http +HTTP/1.1 200 OK +Content-Type: application/json + +{} +``` + +#### 3. Manual On-Demand Status Reconciliation Request + +Cocoon can manually query the partner's status endpoint. + +##### HTTP GET Request: + +```http +GET /status?uid=0sAtLhwbmGz2Y0WBDzPL HTTP/1.1 +Host: partner-ci.example.com +Authorization: Bearer +``` + +##### HTTP Response: + +```http +HTTP/1.1 200 OK +Content-Type: application/json + +{ + "status": "in_progress", + "message": "Some optional summary if needed", + "details": [ + { + "name": "test_1", + "status": "failed", + "log_url": "https://parnter.com/test_1/log" + }, + { + "name": "test_2", + "status": "success", + "log_url": "https://parnter-ci.com/test_2/log" + }, + { + "name": "test_3", + "status": "skipped", + "log_url": "https://parnter-ci.com/test_3/log" + }, + { + "name": "test_4", + "status": "failed", + "log_url": "https://parnter-ci.com/test_4/log" + }, + { + "name": "test_5", + "status": "in_progress", + "log_url": null + } + ] +} +``` + +Where: + +* **status**: The status of CI execution: + * `in_progress` - Partner CI starts the execution of this task (in progress state). + * `cancelled` - Task is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during test execution (failure state). + * `failed` - Some or all of the tests are failed (failure state). + * `succeeded` - All tests passed (success state). + * `neutral` - Only some flaky tests failed (success state). + * `skipped` - Some or all tests are skipped (success state). +* **message**: Markdown formatted message with a summary of CI run. +* **details**: Granular test execution details: + * **name**: Test name. + * **status**: Current test execution status: + * `in_progress` - Partner CI starts the execution of this task (in progress state). + * `cancelled` - Test is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during test execution (failure state). + * `failed` - Test failed (failure state). + * `succeeded` - Test passed (success state). + * `neutral` - Test failed but considered flaky (success state). + * `skipped` - Test skipped (success state). + * **log_url**: The publicly accessible task execution log URL. + +## Detailed Design + +Integration with third-party CIs requires two main points of discussion: communication and authentication. For external system notification, the recommended approach is a bi-directional webhook server-to-server communication, combined with a manual on-demand status reconciliation mechanism. + +### Primary Design: Webhook Server-to-Server Communication (Push-Callback) with Manual Status Request + +In this architecture, Cocoon uses standard HTTP webhooks to push event notifications to external partner endpoints. External partners process the event asynchronously on their custom infrastructure and send execution status back via Cocoon's callback REST API. Additionally, partners must host an endpoint capable of responding to on-demand status queries from Cocoon to enable manual status checks from the build dashboard, eliminating automated polling loops. + +#### End-to-End Workflow Diagram + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + actor GitHub as GitHub Repo + participant Cocoon as Cocoon Server + participant Partner as Partner CI + destroy GitHub + GitHub->>+Cocoon: Postsubmit Event + Note over Cocoon: Creates Partner task with uid=abcd + Cocoon->>-PubSub: Notify partner message with uid=abcd + + alt Cocoon calls partner webhook to notify about post submit to run CI, retrying on failures + PubSub->>+Cocoon: Notify partner message with uid=abcd + Cocoon->>+Partner: HTTP POST (signed with cocoon key, JWT body with notification for uid=abcd) + Partner->>Cocoon: HTTP GET of Cocoon JWKS url + Cocoon-->>Partner: JWKS to verify JWT signature + Note over Partner: Internal Failure during processing the request + Partner-->>-Cocoon: Response 500 + Cocoon-->>-PubSub: nack + + PubSub->>+Cocoon: Retry notify partner message with uid=abcd + Cocoon->>+Partner: Retry HTTP POST (signed with cocoon key, JWT body with notification for uid=abcd) + Partner->>Cocoon: HTTP GET of Cocoon JWKS url + Cocoon-->>Partner: JWKS to verify JWT signature + Partner-->>Cocoon: Response 200 + Cocoon-->>-PubSub: ack + end + + opt Manual status request + Note over Cocoon: User requested task status while waiting execution results + Cocoon->>Partner: HTTP GET /status?uid=abcd + Partner-->>Cocoon: Response 202 {"status": "in_progress"} + end + + Note over Partner: Executing custom tests + + alt Partner calls cocoon webhook to provide CI run status update, retrying on failures + Partner->>Cocoon: HTTP POST Callback (signed with partner key, JWT body with status for uid=abcd) + Note over Partner: Response timeout + + Partner->>+Cocoon: Retry HTTP POST (signed with partner key, JWT body with status for uid=abcd) + Cocoon->>Partner: HTTP GET of Partner JWKS url + Partner-->>Cocoon: JWKS to verify JWT signature + Cocoon->>PubSub: Partner test execution status for uid=abcd + Cocoon-->>-Partner: Response 200 + deactivate Partner + + PubSub->>+Cocoon: Partner test execution status for uid=abcd + Note over Cocoon: Update Partner task status for uid=abcd + deactivate Cocoon + end +``` + +#### Flow Breakdown + +* **1. Trigger:** GitHub notifies Cocoon via `/api/github/webhook-subscription` when a commit lands on `main`. +* **2. Scheduling:** Cocoon schedules tasks on LUCI and creates dummy tasks in Firestore/Redis for each configured partner. +* **3. Outbound Queueing:** For every configured partner, Cocoon creates a record in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, and publishes a message containing the `uid` and commit context (`commit_sha`, `pr_number`, `repo`, etc.) to an internal GCP Pub/Sub topic for each partner. +* **4. Partner Dispatch:** A Pub/Sub subscriber fetches partner configurations, generates a short-lived JWT signed with an Ed25519 private key, and issues an HTTP POST webhook request to the partner's configured URL. Cocoon remains stateless and retries the dispatch via Pub/Sub if it timed out waiting for status response, receives 5xx (Server error), or 408 (Request Timeout) response. +* **5. Terminal unsuccessful response:** If a partner webhook responds with client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. +* **6. Execution & Callback:** The partner CI executes tests on proprietary hardware and makes an API callback to Cocoon with the execution status (`in_progress`, `cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), message, and log URLs. Cocoon reads the Key ID from the JWT header and does a lookup for the corresponding cached public key; if the public key is not found it attempts to re-read partner's JWKS and update cache; if the key is not found it responds with 401. If Cocoon successfully validates the payload and sends the partner status to Pub/Sub, it will respond with status code 200. Failure responses indicating that the partner CI system should retry sending the CI execution status are: 5xx and 408. In case of a JWT signature validation failure, Cocoon will respond with 401. If there is a validation error, it responds with 400; if the `uid` is not found, it responds with 404; if already processed, it responds with 409; if body size exceeds 8MB it responds with 413. +* **7. Processing response:** Once Cocoon receives the partner status from Pub/Sub, it updates the partner task state in Firestore/Redis. +* **8. Manual On-Demand Status Reconciliation:** Partners are required to host a status endpoint (e.g., `GET /status?uid=abcd`). In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. Automated polling is strictly rejected to avoid DoS risks and operational overhead. + +#### Security & Authentication + +This section details the decentralized security model used to authorize server-to-server bidirectional webhook communication without a centralized authorization server. + +##### Asymmetric Cryptography & Key Management + +Both the Cocoon server and the Partner CI server independently generate their own asymmetric key pairs (using Ed25519). Each system stores its private key in a secure key management system and shares its corresponding public key with the other party. + +By leveraging decentralized asymmetric cryptography, this approach intentionally avoids the need for a central Authorization Server or Identity Provider, eliminating a potential single point of failure and reducing infrastructure complexity. + +##### Key Distribution Strategy: Public Key Endpoints + +For this authentication model to work, both parties must reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to fetch or refresh this key periodically. This is highly secure because it allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. + +Example of JWKS: `https://firebaseappcheck.googleapis.com/v1/jwks` + +##### Serialized JWT HTTP Body + +When either system sends an HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender constructs a POST request as follows: + +* **Header**: `Content-Type: application/jwt` +* **Body**: A signed JSON Web Token (JWT) + +##### JWT Payload Structure + +The JWT consists of a base64 encoded header, payload, and signature separated by dots. +For example: + +```text +eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA +``` + +##### Decoded Header: + +```json +{ + "alg": "EdDSA", + "typ": "JWT", + "kid": 12345 +} +``` + +Where: + +* **alg**: The cryptographic algorithm used to sign the token (e.g., `EdDSA` for Ed25519 signatures). +* **typ**: The type of the token, which is set to `JWT`. +* **kid**: Key ID that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. + +##### Decoded Payload: + +```jsonc +{ + // --- Standard Security Metadata --- + "iss": "cocoon", + "aud": "https://3p-partner-ci.com/webhook", + "iat": 1774962000, + "exp": 1774962300, + // --- The Actual Message (Custom Claims) --- + "data": { + "slug": "flutter/flutter", + "pr": 12345, + "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b", + "task_name": "some_awesome_test" + } +} +``` + +##### Decoded Signature: + +```text +Ed25519Sign(privateKeyEd25519, ASCII_OR_UTF8_BYTES(encodedHeader + "." + encodedPayload)) +``` + +Where: + +* **Ed25519Sign** is an EdDSA/Ed25519 function for deterministic signatures. +* **privateKeyEd25519** is an Ed25519 private key. +* **encodedHeader** is a `Base64UrlEncode(JSON.stringify(header))`. +* **encodedPayload** is a `Base64UrlEncode(JSON.stringify(payload))`. + +##### Validating Incoming Requests + +The webhook listener receives the JWT in the body of the request, extracts the Key ID (`kid`), fetches the provider's public key from their JWKS URL, and verifies the signature. This allows each side to validate the caller's identity and ensure request integrity independently without relying on a third-party authority. Cocoon's webhook should also validate that the `uid` provided by the partner exists and has not been processed yet. + +#### Pros & Cons + +* **Pros:** + * Industry-standard webhook architecture that is easy for partners to consume. + * Push-based trigger is real-time and minimizes latency. + * Decouples Flutter core infrastructure from partner device management. + * Stateless design on Cocoon via Firestore/Redis and GCP Pub/Sub. + * Status query endpoints allow robust reconciliation on-demand without automated polling loops. +* **Cons:** + * Requires partners to maintain publicly accessible HTTP endpoints and handle inbound requests. + * Downtime of partner endpoints requires backoff and retry queues on Cocoon. + * Exposes inbound REST APIs on Cocoon that require JWT validation. + +### Alternatives Considered + +#### Alternative 1: Event-Driven GCP Pub/Sub Integration (Asynchronous Event Bus) + +This architecture eliminates direct server-to-server HTTP calls by leveraging Google Cloud Pub/Sub as an event bus between Cocoon and partner infrastructure. + +##### End-to-End Workflow Diagram + +```mermaid +sequenceDiagram + actor GitHub as GitHub Repo + participant Cocoon as Cocoon Server + participant PubSub@{"type":"queue"} + participant Partner as Partner CI + + destroy GitHub + GitHub->>Cocoon: Postsubmit Event + Note over Cocoon: Creates Partner task + Cocoon->>PubSub: Publish Event + PubSub->>Partner: Partner Ingestion (JWT+Service Account) + Note over Partner: Executes custom tests + Partner->>PubSub: Publish Results (JWT+Service Account) + PubSub->>Cocoon: Update Partner task status + Note over Cocoon: Update Partner task status +``` + +##### Workflow + +* **1. Trigger:** GitHub triggers post-commit processing in Cocoon. +* **2. Outbound Event:** Cocoon schedules the partner task and publishes an event to an outbound Pub/Sub topic. +* **3. Partner Ingestion:** Partners consume messages from the topic via cross-project GCP IAM subscription or delegation. +* **4. Execution:** Partner CI processes the event on local hardware. +* **5. Inbound Result:** Partner CI publishes results directly back to Cocoon's inbound Pub/Sub topic. +* **6. Dashboard Update:** Cocoon ingests result messages and updates test execution states in the dashboard. + +##### Pros & Cons + +* **Pros:** + * Highly asynchronous and resilient; messages are automatically buffered during outages. + * No inbound HTTP ports are required on the partner side; can run behind strict firewalls. + * Removes webhook management, network connection timeouts, and custom HTTP retry logic. +* **Cons:** + * High administration overhead: requires managing cross-project GCP permissions or GCP service account keys for partners. + * Introduces cloud vendor lock-in (GCP) for third-party partners. + * Increases setup and development complexity for partners. + * Cannot validate message body at the time of message submission. + +#### Alternative 2: Automated Polling Strategy (Push Trigger, Pull Status) + +This hybrid model retains HTTP push for triggering partner CI jobs, but replaces inbound callbacks with periodic polling by Cocoon to check test completion. + +##### End-to-End Workflow Diagram + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + actor GitHub as GitHub Repo + participant Cocoon as Cocoon Server + participant Partner as Partner CI + destroy GitHub + + GitHub->>Cocoon: Postsubmit Event + Note over Cocoon: Creates Partner task + Cocoon->>PubSub: Notify partner message for uid=abcd + PubSub->>Cocoon: Notify partner message for uid=abcd + Cocoon->>Partner: HTTP POST (signed with cocoon key, JWT body with notification for uid=abcd) + Partner-->>Cocoon: Response 200 + Cocoon-->>PubSub: ack + Cocoon->>PubSub: Pull partner CI status for uid=abcd + + loop Polling by using PubSub retry + PubSub->>Cocoon: Pull partner CI status for uid=abcd + Note over Partner: Executes custom tests + Cocoon->>Partner: GET /status?uid=abcd + Partner-->>Cocoon: Response 202, Status: RUNNING + Cocoon-->>PubSub: nack + end + + PubSub->>Cocoon: Pull partner CI status for uid=abcd + Cocoon->>Partner: GET /status?uid=abcd + Partner-->>Cocoon: Response 200, Status: COMPLETED + Cocoon-->>PubSub: ack + Cocoon->>PubSub: Partner test execution status for uid=abcd + PubSub->>Cocoon: Partner test execution status for uid=abcd + Note over Cocoon: Update Partner task status + Cocoon-->>PubSub: ack +``` + +##### Workflow + +* **1. Trigger & Dispatch:** GitHub triggers Cocoon, which creates partner tasks and pushes a webhook notification to the partner. +* **2. Execution:** The partner CI executes workloads asynchronously. +* **3. Polling Loop:** Cocoon automatically queries the partner's status endpoint periodically (e.g., using Pub/Sub retry loop). +* **4. Inbound Processing:** Once the status endpoint returns a terminal state (`PASSED`/`FAILED`), Cocoon updates the dashboard. + +##### Pros & Cons + +* **Pros:** + * Prevents partners from needing inbound access or security tokens to write to Cocoon endpoints. + * Simpler implementation for partners with pre-existing REST APIs that expose build status. +* **Cons:** + * High network and resource overhead from frequent, automated polling across numerous partners. + * Increases latency in reporting status updates back to the Flutter dashboard due to polling intervals. + * Scale limitations and risk of denial-of-service (DoS) on partner endpoints under heavy load. + +### Comparison of Options + +| Approach Criteria | Primary Design: Webhook Push-Callback | Alternative 1: Event-Driven GCP Pub/Sub | Alternative 2: Automated Polling (Push-Poll) | +| --- | --- | --- | --- | +| **Communication Model** | Bidirectional HTTP webhooks with manual query backup. | Asynchronous Event Bus (Pub/Sub). | HTTP push trigger, automated polling by Cocoon. | +| **Partner Setup Complexity** | **Low to moderate.** Webhooks are easy to consume, but partners must implement retry logic for outbound webhook calls. | **High.** Requires managing cross-project GCP permissions or GCP service account keys. | **Low.** Simpler for partners to only set up webhook and status endpoints. | +| **Cloud Vendor Lock-in** | **Low.** Built on standard REST and HTTP/JWT protocols. | **High.** Introduces GCP cloud vendor lock-in for third-party partners. | **Low.** Built on standard REST and HTTP/JWT protocols. | +| **Security & Authentication** | **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | **Secure.** Managed via GCP IAM and service accounts. | **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | +| **Status response validation** | Can validate payload schema, unknown/duplicated `uid` and notify partner with HTTP response codes. | Can only validate and notify partner about payload schema violation. | Cannot notify partner of any payload schema or data validation issues. | +| **Latency / Status Reporting** | **Real-time.** Push-based callback minimizes status reporting latency. | **Real-time.** Push-based consumption minimizes latency. | **Higher latency.** Updates are delayed based on Cocoon's polling intervals. | +| **Resilience & Overhead** | **Highly resilient.** Retries are backed by Pub/Sub; Status query endpoint prevents polling loops. | **Highly resilient.** Messages are automatically buffered during partner infrastructure outages. | **High overhead.** Constant automated polling creates network load and risks DoS on partner endpoints. | + +## Integration with Existing Features + +To support webhook flows, the following must be implemented in Cocoon: + +### Firestore + +#### partner_config collection + +For partner configuration, add a `partner_config` collection with Document ID equal to `task_name`. + +```go +slug string // github repo (e.g. flutter/flutter or flutter/packages) to trigger on +task_name string // Name of post submit task +partner string // 3p partner name +webhook_url string // Url to call to request CI +status_url string // Manual status request url +keys_url string // Url to read partner key id to public key mapping +``` + +Currently, there will be no UI for editing this table. For every post-submit dashboard checks will be extended with tasks from this list. All tasks from this list would be created suppressed without the ability to unsuppress. Later if a decision will be made to let some 1P partner CI block the tree we may extend this table with a `block_tree` boolean variable. + +#### partner_ci collection + +Add a `partner_ci` collection with Auto-ID Document ID that will be equal to the `uid` of the `data` payload. This is used to match post-submit tasks to corresponding status responses from partner CI systems. + +```go +commit_sha string // Head sha at a moment of check-run creation ie commit sha +slug string // repository_owner/repository_name (e.g. flutter/flutter) +partner string // 3p partner name +task_name string // Name of post submit task +attempt_number int // Number of re-run attempt +processed bool // Whether status response from partner was already processed +``` + +#### tasks + +Extend `tasks` collection with `message` and `log_urls` fields. + +### Pub/Sub + +#### partner_ci_notification topic + +To guarantee that Cocoon can notify each partner CI even during network outages or temporary partner infrastructure outages, once Cocoon reads the post-submit task configuration for a post-submit event, it will send a message to the `partner_ci_notification` topic for every partner integration. A separate instance of Cocoon will receive that message from `partner_ci_notification_sub` and attempt to post it to the partner webhook. + +#### partner_ci_status topic + +To guarantee that Cocoon can properly process status updates once the status webhook is called by a partner and the payload is successfully validated, the payload will be sent to the `partner_ci_status` topic and later received by Cocoon from `partner_ci_status_sub` to update the corresponding task status. + +## Open Questions + +* What other fields should we provide to partners webhook to request CI run? +* Can tasks PK `__` be used as `uid`? In that case we would not need a `partner_ci` collection. + +## Testing Plan + +Implement a dummy partner CI to test the end-to-end workflow. + +## Documentation Plan + +Provide documentation for partners detailing integration setup and best practices for retry mechanisms. From 68663651322d2be137ad325870d809a96ececbc4 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Tue, 6 Oct 2026 14:47:40 -0700 Subject: [PATCH 02/35] change rfc from 600 to 610 --- ...tter-pluggable-ci.md => 610.0000-flutter-pluggable-ci.md} | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) rename rfc/{600.0000-flutter-pluggable-ci.md => 610.0000-flutter-pluggable-ci.md} (99%) diff --git a/rfc/600.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md similarity index 99% rename from rfc/600.0000-flutter-pluggable-ci.md rename to rfc/610.0000-flutter-pluggable-ci.md index 01184aa..28d80e6 100644 --- a/rfc/600.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -1,6 +1,6 @@ --- type: rfc -rfc: '600.0000' +rfc: '610.0000' title: Flutter Pluggable CI Design description: Describes the integration with partner CI systems to trigger partner CI execution on post-submit PRs and display execution details on the post-submit dashboard. status: draft @@ -8,11 +8,12 @@ created: 2026-10-06T09:00:00Z updated: 2026-10-06T09:00:00Z tags: - 600-infrastructure + - 610-cicd authors: - '"Dmitry Grand" ' --- -# RFC 600.0000: Flutter Pluggable CI Design +# RFC 610.0000: Flutter Pluggable CI Design ## Problem Statement From 7dd486a5bf6125616f1e30f65b065c06036e5740 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 09:32:42 -0700 Subject: [PATCH 03/35] use html table to have colored background --- rfc/610.0000-flutter-pluggable-ci.md | 63 ++++++++++++++++++++++++---- 1 file changed, 54 insertions(+), 9 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 28d80e6..f3d49ae 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -587,15 +587,60 @@ sequenceDiagram ### Comparison of Options -| Approach Criteria | Primary Design: Webhook Push-Callback | Alternative 1: Event-Driven GCP Pub/Sub | Alternative 2: Automated Polling (Push-Poll) | -| --- | --- | --- | --- | -| **Communication Model** | Bidirectional HTTP webhooks with manual query backup. | Asynchronous Event Bus (Pub/Sub). | HTTP push trigger, automated polling by Cocoon. | -| **Partner Setup Complexity** | **Low to moderate.** Webhooks are easy to consume, but partners must implement retry logic for outbound webhook calls. | **High.** Requires managing cross-project GCP permissions or GCP service account keys. | **Low.** Simpler for partners to only set up webhook and status endpoints. | -| **Cloud Vendor Lock-in** | **Low.** Built on standard REST and HTTP/JWT protocols. | **High.** Introduces GCP cloud vendor lock-in for third-party partners. | **Low.** Built on standard REST and HTTP/JWT protocols. | -| **Security & Authentication** | **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | **Secure.** Managed via GCP IAM and service accounts. | **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | -| **Status response validation** | Can validate payload schema, unknown/duplicated `uid` and notify partner with HTTP response codes. | Can only validate and notify partner about payload schema violation. | Cannot notify partner of any payload schema or data validation issues. | -| **Latency / Status Reporting** | **Real-time.** Push-based callback minimizes status reporting latency. | **Real-time.** Push-based consumption minimizes latency. | **Higher latency.** Updates are delayed based on Cocoon's polling intervals. | -| **Resilience & Overhead** | **Highly resilient.** Retries are backed by Pub/Sub; Status query endpoint prevents polling loops. | **Highly resilient.** Messages are automatically buffered during partner infrastructure outages. | **High overhead.** Constant automated polling creates network load and risks DoS on partner endpoints. | + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Approach CriteriaPrimary Design: Webhook Push-CallbackAlternative 1: Event-Driven GCP Pub/SubAlternative 2: Automated Polling (Push-Poll)
Communication ModelBidirectional HTTP webhooks with manual query backup.Asynchronous Event Bus (Pub/Sub).HTTP push trigger, automated polling by Cocoon.
Partner Setup ComplexityLow to moderate. Webhooks are easy to consume, but partners must implement retry logic for outbound webhook calls.High. Requires managing cross-project GCP permissions or GCP service account keys.Low. Simpler for partners to only set up webhook and status endpoints.
Cloud Vendor Lock-inLow. Built on standard REST and HTTP/JWT protocols.High. Introduces GCP cloud vendor lock-in for third-party partners.Low. Built on standard REST and HTTP/JWT protocols.
Security & AuthenticationHighly secure. Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server.Secure. Managed via GCP IAM and service accounts.Highly secure. Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server.
Status response validationCan validate payload schema, unknown/duplicated uid and notify partner with HTTP response codes.Can only validate and notify partner about payload schema violation.Cannot notify partner of any payload schema or data validation issues.
Latency / Status ReportingReal-time. Push-based callback minimizes status reporting latency.Real-time. Push-based consumption minimizes latency.Higher latency. Updates are delayed based on Cocoon's polling intervals.
Resilience & OverheadHighly resilient. Retries are backed by Pub/Sub; Status query endpoint prevents polling loops.Highly resilient. Messages are automatically buffered during partner infrastructure outages.High overhead. Constant automated polling creates network load and risks DoS on partner endpoints.
## Integration with Existing Features From cf5efb17066cf0e1cacb1cf95bb1fa3fc9dc66d8 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 09:53:10 -0700 Subject: [PATCH 04/35] added tags --- rfc/610.0000-flutter-pluggable-ci.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index f3d49ae..862a6c0 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -115,7 +115,12 @@ See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc751 "slug": "flutter/flutter", "pr": 12345, "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b", - "task_name": "some_awesome_partners_test" + "task_name": "some_awesome_partners_test", + "tags": { + "attempt": "1", + "key2": "value2", + "key3": "value3" + } } } ``` @@ -132,6 +137,7 @@ Where: * **pr**: The Pull Request number associated with the event. * **sha**: The target git commit SHA hash. * **task_name**: Name of task to be displayed on dashboard, to potentially support several CI types or platforms per partner. For only one task per partner can be ignored. + * **tags**: Optional key-value string pairs, for future usage. Should be returned as is in the status response. ##### HTTP Response: @@ -166,6 +172,11 @@ Content-Type: application/jwt "exp": 1774963800, "data": { "uid": "0sAtLhwbmGz2Y0WBDzPL", + "tags": { + "attempt": "1", + "key2": "value2", + "key3": "value3" + }, "status": "failed", "message": "Some optional summary if needed", "details": [ @@ -207,6 +218,7 @@ Where: * **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. * **data**: Object containing custom payload claims and event details: * **uid**: The unique correlation identifier to match which CI execution request this status response belongs to. + * **tags**: Optional key-value string pairs, for internal usage by Flutter CI. Should be returned as is if received in request. * **status**: The terminal status of CI execution (cannot be `in_progress`): * `cancelled` - Task is cancelled by a Partner CI (failure state). * `infra_failure` - Infrastructure failure during tests execution (failure state). From 78e97d7bc63b2cb4989abe3f987d3dbc3d1fc833 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 10:01:17 -0700 Subject: [PATCH 05/35] fix trailing punctuation --- rfc/610.0000-flutter-pluggable-ci.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 862a6c0..71fe4e4 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -90,7 +90,7 @@ This design does not cover what kind of CI for out-of-tree platforms to use and When a post-submit commit is processed, Cocoon posts a signed JWT to the partner's webhook URL. See: [JSON Web Token (JWT) RFC 7519](https://www.rfc-editor.org/info/rfc7519/) for more details about JWT. -##### HTTP POST Request: +##### HTTP POST Request ```http POST /webhook HTTP/1.1 @@ -102,7 +102,7 @@ Content-Type: application/jwt See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc7515/) for more details about Signature (JWS). -##### Decoded JWT Payload: +##### Decoded JWT Payload ```json { @@ -139,7 +139,7 @@ Where: * **task_name**: Name of task to be displayed on dashboard, to potentially support several CI types or platforms per partner. For only one task per partner can be ignored. * **tags**: Optional key-value string pairs, for future usage. Should be returned as is in the status response. -##### HTTP Response: +##### HTTP Response ```http HTTP/1.1 200 OK @@ -152,7 +152,7 @@ Content-Type: application/json After executing custom test suites on proprietary hardware, the partner sends the final status back to Cocoon. -##### HTTP POST Request: +##### HTTP POST Request ```http POST /api/partner/callback HTTP/1.1 @@ -162,7 +162,7 @@ Content-Type: application/jwt .. ``` -##### Decoded Payload: +##### Decoded Payload ```json { @@ -238,7 +238,7 @@ Where: * `skipped` - Test skipped (success state). * **log_url**: The publicly accessible task execution log URL. -##### HTTP Response: +##### HTTP Response ```http HTTP/1.1 200 OK @@ -251,7 +251,7 @@ Content-Type: application/json Cocoon can manually query the partner's status endpoint. -##### HTTP GET Request: +##### HTTP GET Request ```http GET /status?uid=0sAtLhwbmGz2Y0WBDzPL HTTP/1.1 @@ -259,7 +259,7 @@ Host: partner-ci.example.com Authorization: Bearer ``` -##### HTTP Response: +##### HTTP Response ```http HTTP/1.1 200 OK @@ -427,7 +427,7 @@ For example: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA ``` -##### Decoded Header: +##### Decoded Header ```json { @@ -443,7 +443,7 @@ Where: * **typ**: The type of the token, which is set to `JWT`. * **kid**: Key ID that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. -##### Decoded Payload: +##### Decoded Payload ```jsonc { From d8899645a06a5065072ba0a68e604772328d7aa9 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 11:27:37 -0700 Subject: [PATCH 06/35] fix Multiple top-level headings and table background --- rfc/610.0000-flutter-pluggable-ci.md | 103 ++++++++------------------- 1 file changed, 29 insertions(+), 74 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 71fe4e4..603100c 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -90,7 +90,7 @@ This design does not cover what kind of CI for out-of-tree platforms to use and When a post-submit commit is processed, Cocoon posts a signed JWT to the partner's webhook URL. See: [JSON Web Token (JWT) RFC 7519](https://www.rfc-editor.org/info/rfc7519/) for more details about JWT. -##### HTTP POST Request +**HTTP POST Request** ```http POST /webhook HTTP/1.1 @@ -102,7 +102,7 @@ Content-Type: application/jwt See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc7515/) for more details about Signature (JWS). -##### Decoded JWT Payload +**Decoded JWT Payload** ```json { @@ -139,7 +139,7 @@ Where: * **task_name**: Name of task to be displayed on dashboard, to potentially support several CI types or platforms per partner. For only one task per partner can be ignored. * **tags**: Optional key-value string pairs, for future usage. Should be returned as is in the status response. -##### HTTP Response +**HTTP Response** ```http HTTP/1.1 200 OK @@ -152,7 +152,7 @@ Content-Type: application/json After executing custom test suites on proprietary hardware, the partner sends the final status back to Cocoon. -##### HTTP POST Request +**HTTP POST Request** ```http POST /api/partner/callback HTTP/1.1 @@ -162,7 +162,7 @@ Content-Type: application/jwt .. ``` -##### Decoded Payload +**Decoded Payload** ```json { @@ -238,7 +238,7 @@ Where: * `skipped` - Test skipped (success state). * **log_url**: The publicly accessible task execution log URL. -##### HTTP Response +**HTTP Response** ```http HTTP/1.1 200 OK @@ -251,7 +251,7 @@ Content-Type: application/json Cocoon can manually query the partner's status endpoint. -##### HTTP GET Request +**HTTP GET Request** ```http GET /status?uid=0sAtLhwbmGz2Y0WBDzPL HTTP/1.1 @@ -259,7 +259,7 @@ Host: partner-ci.example.com Authorization: Bearer ``` -##### HTTP Response +**HTTP Response** ```http HTTP/1.1 200 OK @@ -329,7 +329,7 @@ Integration with third-party CIs requires two main points of discussion: communi In this architecture, Cocoon uses standard HTTP webhooks to push event notifications to external partner endpoints. External partners process the event asynchronously on their custom infrastructure and send execution status back via Cocoon's callback REST API. Additionally, partners must host an endpoint capable of responding to on-demand status queries from Cocoon to enable manual status checks from the build dashboard, eliminating automated polling loops. -#### End-to-End Workflow Diagram +**End-to-End Workflow Diagram** ```mermaid sequenceDiagram @@ -418,7 +418,7 @@ When either system sends an HTTP request (Cocoon dispatching a webhook or a Part * **Header**: `Content-Type: application/jwt` * **Body**: A signed JSON Web Token (JWT) -##### JWT Payload Structure +**JWT Payload Structure** The JWT consists of a base64 encoded header, payload, and signature separated by dots. For example: @@ -427,7 +427,7 @@ For example: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA ``` -##### Decoded Header +**Decoded Header** ```json { @@ -443,7 +443,7 @@ Where: * **typ**: The type of the token, which is set to `JWT`. * **kid**: Key ID that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. -##### Decoded Payload +**Decoded Payload** ```jsonc { @@ -462,7 +462,7 @@ Where: } ``` -##### Decoded Signature: +**Decoded Signature** ```text Ed25519Sign(privateKeyEd25519, ASCII_OR_UTF8_BYTES(encodedHeader + "." + encodedPayload)) @@ -475,7 +475,7 @@ Where: * **encodedHeader** is a `Base64UrlEncode(JSON.stringify(header))`. * **encodedPayload** is a `Base64UrlEncode(JSON.stringify(payload))`. -##### Validating Incoming Requests +**Validating Incoming Requests** The webhook listener receives the JWT in the body of the request, extracts the Key ID (`kid`), fetches the provider's public key from their JWKS URL, and verifies the signature. This allows each side to validate the caller's identity and ensure request integrity independently without relying on a third-party authority. Cocoon's webhook should also validate that the `uid` provided by the partner exists and has not been processed yet. @@ -498,7 +498,7 @@ The webhook listener receives the JWT in the body of the request, extracts the K This architecture eliminates direct server-to-server HTTP calls by leveraging Google Cloud Pub/Sub as an event bus between Cocoon and partner infrastructure. -##### End-to-End Workflow Diagram +**End-to-End Workflow Diagram** ```mermaid sequenceDiagram @@ -518,7 +518,7 @@ sequenceDiagram Note over Cocoon: Update Partner task status ``` -##### Workflow +**Workflow** * **1. Trigger:** GitHub triggers post-commit processing in Cocoon. * **2. Outbound Event:** Cocoon schedules the partner task and publishes an event to an outbound Pub/Sub topic. @@ -527,7 +527,7 @@ sequenceDiagram * **5. Inbound Result:** Partner CI publishes results directly back to Cocoon's inbound Pub/Sub topic. * **6. Dashboard Update:** Cocoon ingests result messages and updates test execution states in the dashboard. -##### Pros & Cons +**Pros & Cons** * **Pros:** * Highly asynchronous and resilient; messages are automatically buffered during outages. @@ -543,7 +543,7 @@ sequenceDiagram This hybrid model retains HTTP push for triggering partner CI jobs, but replaces inbound callbacks with periodic polling by Cocoon to check test completion. -##### End-to-End Workflow Diagram +**End-to-End Workflow Diagram** ```mermaid sequenceDiagram @@ -580,14 +580,14 @@ sequenceDiagram Cocoon-->>PubSub: ack ``` -##### Workflow +**Workflow** * **1. Trigger & Dispatch:** GitHub triggers Cocoon, which creates partner tasks and pushes a webhook notification to the partner. * **2. Execution:** The partner CI executes workloads asynchronously. * **3. Polling Loop:** Cocoon automatically queries the partner's status endpoint periodically (e.g., using Pub/Sub retry loop). * **4. Inbound Processing:** Once the status endpoint returns a terminal state (`PASSED`/`FAILED`), Cocoon updates the dashboard. -##### Pros & Cons +**Pros & Cons** * **Pros:** * Prevents partners from needing inbound access or security tokens to write to Cocoon endpoints. @@ -599,60 +599,15 @@ sequenceDiagram ### Comparison of Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Approach CriteriaPrimary Design: Webhook Push-CallbackAlternative 1: Event-Driven GCP Pub/SubAlternative 2: Automated Polling (Push-Poll)
Communication ModelBidirectional HTTP webhooks with manual query backup.Asynchronous Event Bus (Pub/Sub).HTTP push trigger, automated polling by Cocoon.
Partner Setup ComplexityLow to moderate. Webhooks are easy to consume, but partners must implement retry logic for outbound webhook calls.High. Requires managing cross-project GCP permissions or GCP service account keys.Low. Simpler for partners to only set up webhook and status endpoints.
Cloud Vendor Lock-inLow. Built on standard REST and HTTP/JWT protocols.High. Introduces GCP cloud vendor lock-in for third-party partners.Low. Built on standard REST and HTTP/JWT protocols.
Security & AuthenticationHighly secure. Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server.Secure. Managed via GCP IAM and service accounts.Highly secure. Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server.
Status response validationCan validate payload schema, unknown/duplicated uid and notify partner with HTTP response codes.Can only validate and notify partner about payload schema violation.Cannot notify partner of any payload schema or data validation issues.
Latency / Status ReportingReal-time. Push-based callback minimizes status reporting latency.Real-time. Push-based consumption minimizes latency.Higher latency. Updates are delayed based on Cocoon's polling intervals.
Resilience & OverheadHighly resilient. Retries are backed by Pub/Sub; Status query endpoint prevents polling loops.Highly resilient. Messages are automatically buffered during partner infrastructure outages.High overhead. Constant automated polling creates network load and risks DoS on partner endpoints.
+| Approach Criteria | Primary Design: Webhook Push-Callback | Alternative 1: Event-Driven GCP Pub/Sub | Alternative 2: Automated Polling (Push-Poll) | +| --- | --- | --- | --- | +| **Communication Model** | $\color{green}{\text{■}}$ Bidirectional HTTP webhooks with manual query backup. | $\color{green}{\text{■}}$ Asynchronous Event Bus (Pub/Sub). | $\color{green}{\text{■}}$ HTTP push trigger, automated polling by Cocoon. | +| **Partner Setup Complexity** | $\color{yellow}{\text{■}}$ **Low to moderate.** Webhooks are easy to consume, but partners must implement retry logic for outbound webhook calls. | $\color{red}{\text{■}}$ **High.** Requires managing cross-project GCP permissions or GCP service account keys. | $\color{green}{\text{■}}$ **Low.** Simpler for partners to only set up webhook and status endpoints. | +| **Cloud Vendor Lock-in** | $\color{green}{\text{■}}$ **Low.** Built on standard REST and HTTP/JWT protocols. | $\color{red}{\text{■}}$ **High.** Introduces GCP cloud vendor lock-in for third-party partners. | $\color{green}{\text{■}}$ **Low.** Built on standard REST and HTTP/JWT protocols. | +| **Security & Authentication** | $\color{green}{\text{■}}$ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | $\color{yellow}{\text{■}}$ **Secure.** Managed via GCP IAM and service accounts. | $\color{green}{\text{■}}$ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | +| **Status response validation** | $\color{green}{\text{■}}$ Can validate payload schema, unknown/duplicated `uid` and notify partner with HTTP response codes. | $\color{yellow}{\text{■}}$ Can only validate and notify partner about payload schema violation. | $\color{green}{\text{■}}$ Cannot notify partner of any payload schema or data validation issues. | +| **Latency / Status Reporting** | $\color{green}{\text{■}}$ **Real-time.** Push-based callback minimizes status reporting latency. | $\color{green}{\text{■}}$ **Real-time.** Push-based consumption minimizes latency. | $\color{red}{\text{■}}$ **Higher latency.** Updates are delayed based on Cocoon's polling intervals. | +| **Resilience & Overhead** | $\color{green}{\text{■}}$ **Highly resilient.** Retries are backed by Pub/Sub; Status query endpoint prevents polling loops. | $\color{green}{\text{■}}$ **Highly resilient.** Messages are automatically buffered during partner infrastructure outages. | $\color{red}{\text{■}}$ **High overhead.** Constant automated polling creates network load and risks DoS on partner endpoints. | ## Integration with Existing Features From 23865eba4eb105139c11b0421b7883c88a538971 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 11:32:40 -0700 Subject: [PATCH 07/35] changed colored boxes to emojis --- rfc/610.0000-flutter-pluggable-ci.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 603100c..276adc1 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -601,13 +601,13 @@ sequenceDiagram | Approach Criteria | Primary Design: Webhook Push-Callback | Alternative 1: Event-Driven GCP Pub/Sub | Alternative 2: Automated Polling (Push-Poll) | | --- | --- | --- | --- | -| **Communication Model** | $\color{green}{\text{■}}$ Bidirectional HTTP webhooks with manual query backup. | $\color{green}{\text{■}}$ Asynchronous Event Bus (Pub/Sub). | $\color{green}{\text{■}}$ HTTP push trigger, automated polling by Cocoon. | -| **Partner Setup Complexity** | $\color{yellow}{\text{■}}$ **Low to moderate.** Webhooks are easy to consume, but partners must implement retry logic for outbound webhook calls. | $\color{red}{\text{■}}$ **High.** Requires managing cross-project GCP permissions or GCP service account keys. | $\color{green}{\text{■}}$ **Low.** Simpler for partners to only set up webhook and status endpoints. | -| **Cloud Vendor Lock-in** | $\color{green}{\text{■}}$ **Low.** Built on standard REST and HTTP/JWT protocols. | $\color{red}{\text{■}}$ **High.** Introduces GCP cloud vendor lock-in for third-party partners. | $\color{green}{\text{■}}$ **Low.** Built on standard REST and HTTP/JWT protocols. | -| **Security & Authentication** | $\color{green}{\text{■}}$ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | $\color{yellow}{\text{■}}$ **Secure.** Managed via GCP IAM and service accounts. | $\color{green}{\text{■}}$ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | -| **Status response validation** | $\color{green}{\text{■}}$ Can validate payload schema, unknown/duplicated `uid` and notify partner with HTTP response codes. | $\color{yellow}{\text{■}}$ Can only validate and notify partner about payload schema violation. | $\color{green}{\text{■}}$ Cannot notify partner of any payload schema or data validation issues. | -| **Latency / Status Reporting** | $\color{green}{\text{■}}$ **Real-time.** Push-based callback minimizes status reporting latency. | $\color{green}{\text{■}}$ **Real-time.** Push-based consumption minimizes latency. | $\color{red}{\text{■}}$ **Higher latency.** Updates are delayed based on Cocoon's polling intervals. | -| **Resilience & Overhead** | $\color{green}{\text{■}}$ **Highly resilient.** Retries are backed by Pub/Sub; Status query endpoint prevents polling loops. | $\color{green}{\text{■}}$ **Highly resilient.** Messages are automatically buffered during partner infrastructure outages. | $\color{red}{\text{■}}$ **High overhead.** Constant automated polling creates network load and risks DoS on partner endpoints. | +| **Communication Model** | ✅ Bidirectional HTTP webhooks with manual query backup. | ✅ Asynchronous Event Bus (Pub/Sub). | ✅ HTTP push trigger, automated polling by Cocoon. | +| **Partner Setup Complexity** | ⚠️ **Low to moderate.** Webhooks are easy to consume, but partners must implement retry logic for outbound webhook calls. | ❌ **High.** Requires managing cross-project GCP permissions or GCP service account keys. | ✅ **Low.** Simpler for partners to only set up webhook and status endpoints. | +| **Cloud Vendor Lock-in** | ✅ **Low.** Built on standard REST and HTTP/JWT protocols. | ❌ **High.** Introduces GCP cloud vendor lock-in for third-party partners. | ✅ **Low.** Built on standard REST and HTTP/JWT protocols. | +| **Security & Authentication** | ✅ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | ⚠️ **Secure.** Managed via GCP IAM and service accounts. | ✅ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | +| **Status response validation** | ✅ Can validate payload schema, unknown/duplicated `uid` and notify partner with HTTP response codes. | ⚠️ Can only validate and notify partner about payload schema violation. | ✅ Cannot notify partner of any payload schema or data validation issues. | +| **Latency / Status Reporting** | ✅ **Real-time.** Push-based callback minimizes status reporting latency. | ✅ **Real-time.** Push-based consumption minimizes latency. | ❌ **Higher latency.** Updates are delayed based on Cocoon's polling intervals. | +| **Resilience & Overhead** | ✅ **Highly resilient.** Retries are backed by Pub/Sub; Status query endpoint prevents polling loops. | ✅ **Highly resilient.** Messages are automatically buffered during partner infrastructure outages. | ❌ **High overhead.** Constant automated polling creates network load and risks DoS on partner endpoints. | ## Integration with Existing Features From c2719274c46cffdd21409cb3b30476bbebf88ceb Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 12:27:16 -0700 Subject: [PATCH 08/35] validation test --- rfc/610.0000-flutter-pluggable-ci.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 276adc1..3d77bf1 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -90,7 +90,7 @@ This design does not cover what kind of CI for out-of-tree platforms to use and When a post-submit commit is processed, Cocoon posts a signed JWT to the partner's webhook URL. See: [JSON Web Token (JWT) RFC 7519](https://www.rfc-editor.org/info/rfc7519/) for more details about JWT. -**HTTP POST Request** +**HTTP POST Request:** ```http POST /webhook HTTP/1.1 @@ -152,7 +152,7 @@ Content-Type: application/json After executing custom test suites on proprietary hardware, the partner sends the final status back to Cocoon. -**HTTP POST Request** +**HTTP POST Request:** ```http POST /api/partner/callback HTTP/1.1 From 940fdd8460e2c90246574dda2c201959d197d769 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 12:35:05 -0700 Subject: [PATCH 09/35] Only check sibling headings --- .markdownlint.yaml | 2 ++ rfc/610.0000-flutter-pluggable-ci.md | 40 ++++++++++++++-------------- 2 files changed, 22 insertions(+), 20 deletions(-) diff --git a/.markdownlint.yaml b/.markdownlint.yaml index 528e09e..7c20fc4 100644 --- a/.markdownlint.yaml +++ b/.markdownlint.yaml @@ -1,5 +1,7 @@ default: true # Enable all standard markdownlint rules by default MD013: false # Do not enforce line lengths (diff churning, table lengths, diagrams etc) +MD024: + siblings_only: true MD025: front_matter_title: "" # Do not treat frontmatter title as H1 heading (RFC H1 is in body) MD033: false # Allow inline HTML (badges, centered logos/images, details/summary folds) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 3d77bf1..2d2401f 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -90,7 +90,7 @@ This design does not cover what kind of CI for out-of-tree platforms to use and When a post-submit commit is processed, Cocoon posts a signed JWT to the partner's webhook URL. See: [JSON Web Token (JWT) RFC 7519](https://www.rfc-editor.org/info/rfc7519/) for more details about JWT. -**HTTP POST Request:** +##### HTTP POST Request ```http POST /webhook HTTP/1.1 @@ -102,7 +102,7 @@ Content-Type: application/jwt See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc7515/) for more details about Signature (JWS). -**Decoded JWT Payload** +##### Decoded JWT Payload ```json { @@ -139,7 +139,7 @@ Where: * **task_name**: Name of task to be displayed on dashboard, to potentially support several CI types or platforms per partner. For only one task per partner can be ignored. * **tags**: Optional key-value string pairs, for future usage. Should be returned as is in the status response. -**HTTP Response** +##### HTTP Response ```http HTTP/1.1 200 OK @@ -152,7 +152,7 @@ Content-Type: application/json After executing custom test suites on proprietary hardware, the partner sends the final status back to Cocoon. -**HTTP POST Request:** +##### HTTP POST Request ```http POST /api/partner/callback HTTP/1.1 @@ -162,7 +162,7 @@ Content-Type: application/jwt .. ``` -**Decoded Payload** +##### Decoded Payload ```json { @@ -238,7 +238,7 @@ Where: * `skipped` - Test skipped (success state). * **log_url**: The publicly accessible task execution log URL. -**HTTP Response** +##### HTTP Response ```http HTTP/1.1 200 OK @@ -251,7 +251,7 @@ Content-Type: application/json Cocoon can manually query the partner's status endpoint. -**HTTP GET Request** +##### HTTP GET Request ```http GET /status?uid=0sAtLhwbmGz2Y0WBDzPL HTTP/1.1 @@ -259,7 +259,7 @@ Host: partner-ci.example.com Authorization: Bearer ``` -**HTTP Response** +##### HTTP Response ```http HTTP/1.1 200 OK @@ -329,7 +329,7 @@ Integration with third-party CIs requires two main points of discussion: communi In this architecture, Cocoon uses standard HTTP webhooks to push event notifications to external partner endpoints. External partners process the event asynchronously on their custom infrastructure and send execution status back via Cocoon's callback REST API. Additionally, partners must host an endpoint capable of responding to on-demand status queries from Cocoon to enable manual status checks from the build dashboard, eliminating automated polling loops. -**End-to-End Workflow Diagram** +#### End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -418,7 +418,7 @@ When either system sends an HTTP request (Cocoon dispatching a webhook or a Part * **Header**: `Content-Type: application/jwt` * **Body**: A signed JSON Web Token (JWT) -**JWT Payload Structure** +##### JWT Payload Structure The JWT consists of a base64 encoded header, payload, and signature separated by dots. For example: @@ -427,7 +427,7 @@ For example: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA ``` -**Decoded Header** +##### Decoded Header ```json { @@ -443,7 +443,7 @@ Where: * **typ**: The type of the token, which is set to `JWT`. * **kid**: Key ID that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. -**Decoded Payload** +##### Decoded Payload ```jsonc { @@ -462,7 +462,7 @@ Where: } ``` -**Decoded Signature** +##### Decoded Signature: ```text Ed25519Sign(privateKeyEd25519, ASCII_OR_UTF8_BYTES(encodedHeader + "." + encodedPayload)) @@ -475,7 +475,7 @@ Where: * **encodedHeader** is a `Base64UrlEncode(JSON.stringify(header))`. * **encodedPayload** is a `Base64UrlEncode(JSON.stringify(payload))`. -**Validating Incoming Requests** +##### Validating Incoming Requests The webhook listener receives the JWT in the body of the request, extracts the Key ID (`kid`), fetches the provider's public key from their JWKS URL, and verifies the signature. This allows each side to validate the caller's identity and ensure request integrity independently without relying on a third-party authority. Cocoon's webhook should also validate that the `uid` provided by the partner exists and has not been processed yet. @@ -498,7 +498,7 @@ The webhook listener receives the JWT in the body of the request, extracts the K This architecture eliminates direct server-to-server HTTP calls by leveraging Google Cloud Pub/Sub as an event bus between Cocoon and partner infrastructure. -**End-to-End Workflow Diagram** +##### End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -518,7 +518,7 @@ sequenceDiagram Note over Cocoon: Update Partner task status ``` -**Workflow** +##### Workflow * **1. Trigger:** GitHub triggers post-commit processing in Cocoon. * **2. Outbound Event:** Cocoon schedules the partner task and publishes an event to an outbound Pub/Sub topic. @@ -527,7 +527,7 @@ sequenceDiagram * **5. Inbound Result:** Partner CI publishes results directly back to Cocoon's inbound Pub/Sub topic. * **6. Dashboard Update:** Cocoon ingests result messages and updates test execution states in the dashboard. -**Pros & Cons** +##### Pros & Cons * **Pros:** * Highly asynchronous and resilient; messages are automatically buffered during outages. @@ -543,7 +543,7 @@ sequenceDiagram This hybrid model retains HTTP push for triggering partner CI jobs, but replaces inbound callbacks with periodic polling by Cocoon to check test completion. -**End-to-End Workflow Diagram** +##### End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -580,14 +580,14 @@ sequenceDiagram Cocoon-->>PubSub: ack ``` -**Workflow** +##### Workflow * **1. Trigger & Dispatch:** GitHub triggers Cocoon, which creates partner tasks and pushes a webhook notification to the partner. * **2. Execution:** The partner CI executes workloads asynchronously. * **3. Polling Loop:** Cocoon automatically queries the partner's status endpoint periodically (e.g., using Pub/Sub retry loop). * **4. Inbound Processing:** Once the status endpoint returns a terminal state (`PASSED`/`FAILED`), Cocoon updates the dashboard. -**Pros & Cons** +##### Pros & Cons * **Pros:** * Prevents partners from needing inbound access or security tokens to write to Cocoon endpoints. From 2cc06acd5d02f83c047d927b7b1397bc55f3be68 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 12:36:39 -0700 Subject: [PATCH 10/35] fix trailing punctuation --- rfc/610.0000-flutter-pluggable-ci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 2d2401f..f916878 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -462,7 +462,7 @@ Where: } ``` -##### Decoded Signature: +##### Decoded Signature ```text Ed25519Sign(privateKeyEd25519, ASCII_OR_UTF8_BYTES(encodedHeader + "." + encodedPayload)) From 72813782bce259aef760289d0b32d0f3657cd36c Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 16:36:52 -0700 Subject: [PATCH 11/35] enumerate paragrahps --- .markdownlint.yaml | 2 - rfc/610.0000-flutter-pluggable-ci.md | 108 +++++++++++++-------------- 2 files changed, 54 insertions(+), 56 deletions(-) diff --git a/.markdownlint.yaml b/.markdownlint.yaml index 7c20fc4..528e09e 100644 --- a/.markdownlint.yaml +++ b/.markdownlint.yaml @@ -1,7 +1,5 @@ default: true # Enable all standard markdownlint rules by default MD013: false # Do not enforce line lengths (diff churning, table lengths, diagrams etc) -MD024: - siblings_only: true MD025: front_matter_title: "" # Do not treat frontmatter title as H1 heading (RFC H1 is in body) MD033: false # Allow inline HTML (badges, centered logos/images, details/summary folds) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index f916878..ce87357 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -15,20 +15,20 @@ authors: # RFC 610.0000: Flutter Pluggable CI Design -## Problem Statement +## 1. Problem Statement To allow partners who maintain off-tree embedders to detect regression immediately after a commit lands on the flutter main branch by running their CI on every flutter PR post-submit. -## Background +## 2. Background Multiple Flutter partners adopt custom embedders to support specific platforms not officially supported by Flutter. Changes in the upstream Flutter repository often cause build failures for custom embedders. Depending on how frequently a partner syncs their fork, isolating the exact breaking change is difficult to manage. This solution mitigates the issue by allowing partners to run their CI pipelines against every Flutter PR post-submit, catching regressions early. We cannot allow partner CI execution to block the tree, at least for now, because gardeners responsible for maintaining tree health lack the knowledge or access to partner codebases and infrastructure necessary to resolve failures effectively. -### Audience +### 2.1. Audience * Flutter TLs * First-party partners -### Glossary +### 2.2. Glossary * **In-Tree Platforms:** These are core Flutter platforms that are stored, updated, and managed directly inside the main Flutter repository. This includes the official ports for iOS, Android, macOS, Linux, and Windows. * **Out-of-Tree Platforms:** These are external Flutter platforms developed and maintained by third parties in separate, independent repositories. Because they live outside the main codebase, today developers must fork the Flutter tool and reverse-engineer undocumented system channels (like text input or app lifecycles) to make them work. @@ -36,7 +36,7 @@ Multiple Flutter partners adopt custom embedders to support specific platforms n * **Cocoon:** Flutter's custom Continuous Integration (CI) orchestration system that integrates underlying CI services like LUCI with GitHub to schedule tests and report build statuses. It also provides developer tooling such as auto-submit mechanisms, test flakiness tracking, and powers the public Flutter Pre-submit/Build Dashboards. * **LUCI (Layered Universal Continuous Integration):** The underlying execution engine that runs Flutter's testing and build workloads across diverse platforms like macOS, Windows, and Linux. Orchestrated by Cocoon, LUCI executes specific Python-based "recipes" to securely validate code changes for the Flutter framework, engine, and packages during both pre-submit and post-submit pipelines. -## Overview +## 3. Overview To enable Flutter to scale across external platforms and out-of-tree embedder maintainers, Flutter infrastructure (Cocoon) requires a robust, secure, and decoupled communication model. This system notifies partner CI pipelines of post-submit commits on `flutter/flutter` so partners can execute custom test suites on proprietary hardware and report execution status back to the Flutter build dashboard. @@ -77,20 +77,20 @@ sequenceDiagram deactivate Cocoon ``` -### Non-goals +### 3.1 Non-goals This design does not cover what kind of CI for out-of-tree platforms to use and what tests to run to validate custom embedders. Nor does it cover how to guarantee the delivery of CI execution status from partner CI systems back to Cocoon. -## Usage Examples +## 4. Usage Examples -### Communication Example: Cocoon and 3rd Party Partner +### 4.1. Communication Example: Cocoon and 3rd Party Partner -#### 1. Webhook Dispatch Request (Cocoon -> Partner) +#### 4.1.1. Webhook Dispatch Request (Cocoon -> Partner) When a post-submit commit is processed, Cocoon posts a signed JWT to the partner's webhook URL. See: [JSON Web Token (JWT) RFC 7519](https://www.rfc-editor.org/info/rfc7519/) for more details about JWT. -##### HTTP POST Request +##### 4.1.1.1. HTTP POST Request ```http POST /webhook HTTP/1.1 @@ -102,7 +102,7 @@ Content-Type: application/jwt See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc7515/) for more details about Signature (JWS). -##### Decoded JWT Payload +##### 4.1.1.2. Decoded JWT Payload ```json { @@ -139,7 +139,7 @@ Where: * **task_name**: Name of task to be displayed on dashboard, to potentially support several CI types or platforms per partner. For only one task per partner can be ignored. * **tags**: Optional key-value string pairs, for future usage. Should be returned as is in the status response. -##### HTTP Response +##### 4.1.1.3. HTTP Response ```http HTTP/1.1 200 OK @@ -148,11 +148,11 @@ Content-Type: application/json {} ``` -#### 2. Status Callback Request (Partner -> Cocoon) +#### 4.1.2. Status Callback Request (Partner -> Cocoon) After executing custom test suites on proprietary hardware, the partner sends the final status back to Cocoon. -##### HTTP POST Request +##### 4.1.2.1. HTTP POST Request ```http POST /api/partner/callback HTTP/1.1 @@ -162,7 +162,7 @@ Content-Type: application/jwt .. ``` -##### Decoded Payload +##### 4.1.2.2. Decoded Payload ```json { @@ -238,7 +238,7 @@ Where: * `skipped` - Test skipped (success state). * **log_url**: The publicly accessible task execution log URL. -##### HTTP Response +##### 4.1.2.3. HTTP Response ```http HTTP/1.1 200 OK @@ -247,11 +247,11 @@ Content-Type: application/json {} ``` -#### 3. Manual On-Demand Status Reconciliation Request +#### 4.1.3. Manual On-Demand Status Reconciliation Request Cocoon can manually query the partner's status endpoint. -##### HTTP GET Request +##### 4.1.3.1. HTTP GET Request ```http GET /status?uid=0sAtLhwbmGz2Y0WBDzPL HTTP/1.1 @@ -259,7 +259,7 @@ Host: partner-ci.example.com Authorization: Bearer ``` -##### HTTP Response +##### 4.1.3.2. HTTP Response ```http HTTP/1.1 200 OK @@ -321,15 +321,15 @@ Where: * `skipped` - Test skipped (success state). * **log_url**: The publicly accessible task execution log URL. -## Detailed Design +## 5. Detailed Design Integration with third-party CIs requires two main points of discussion: communication and authentication. For external system notification, the recommended approach is a bi-directional webhook server-to-server communication, combined with a manual on-demand status reconciliation mechanism. -### Primary Design: Webhook Server-to-Server Communication (Push-Callback) with Manual Status Request +### 5.1. Primary Design: Webhook Server-to-Server Communication (Push-Callback) with Manual Status Request In this architecture, Cocoon uses standard HTTP webhooks to push event notifications to external partner endpoints. External partners process the event asynchronously on their custom infrastructure and send execution status back via Cocoon's callback REST API. Additionally, partners must host an endpoint capable of responding to on-demand status queries from Cocoon to enable manual status checks from the build dashboard, eliminating automated polling loops. -#### End-to-End Workflow Diagram +#### 5.1.1. End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -384,7 +384,7 @@ sequenceDiagram end ``` -#### Flow Breakdown +#### 5.1.2. Flow Breakdown * **1. Trigger:** GitHub notifies Cocoon via `/api/github/webhook-subscription` when a commit lands on `main`. * **2. Scheduling:** Cocoon schedules tasks on LUCI and creates dummy tasks in Firestore/Redis for each configured partner. @@ -395,30 +395,30 @@ sequenceDiagram * **7. Processing response:** Once Cocoon receives the partner status from Pub/Sub, it updates the partner task state in Firestore/Redis. * **8. Manual On-Demand Status Reconciliation:** Partners are required to host a status endpoint (e.g., `GET /status?uid=abcd`). In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. Automated polling is strictly rejected to avoid DoS risks and operational overhead. -#### Security & Authentication +#### 5.1.3. Security & Authentication This section details the decentralized security model used to authorize server-to-server bidirectional webhook communication without a centralized authorization server. -##### Asymmetric Cryptography & Key Management +##### 5.1.3.1. Asymmetric Cryptography & Key Management Both the Cocoon server and the Partner CI server independently generate their own asymmetric key pairs (using Ed25519). Each system stores its private key in a secure key management system and shares its corresponding public key with the other party. By leveraging decentralized asymmetric cryptography, this approach intentionally avoids the need for a central Authorization Server or Identity Provider, eliminating a potential single point of failure and reducing infrastructure complexity. -##### Key Distribution Strategy: Public Key Endpoints +##### 5.1.3.2. Key Distribution Strategy: Public Key Endpoints For this authentication model to work, both parties must reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to fetch or refresh this key periodically. This is highly secure because it allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. Example of JWKS: `https://firebaseappcheck.googleapis.com/v1/jwks` -##### Serialized JWT HTTP Body +##### 5.1.3.3. Serialized JWT HTTP Body When either system sends an HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender constructs a POST request as follows: * **Header**: `Content-Type: application/jwt` * **Body**: A signed JSON Web Token (JWT) -##### JWT Payload Structure +##### 5.1.3.4. JWT Payload Structure The JWT consists of a base64 encoded header, payload, and signature separated by dots. For example: @@ -427,7 +427,7 @@ For example: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA ``` -##### Decoded Header +##### 5.1.3.5. Decoded Header ```json { @@ -443,7 +443,7 @@ Where: * **typ**: The type of the token, which is set to `JWT`. * **kid**: Key ID that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. -##### Decoded Payload +##### 5.1.3.6. Decoded Payload ```jsonc { @@ -462,7 +462,7 @@ Where: } ``` -##### Decoded Signature +##### 5.1.3.7. Decoded Signature ```text Ed25519Sign(privateKeyEd25519, ASCII_OR_UTF8_BYTES(encodedHeader + "." + encodedPayload)) @@ -475,11 +475,11 @@ Where: * **encodedHeader** is a `Base64UrlEncode(JSON.stringify(header))`. * **encodedPayload** is a `Base64UrlEncode(JSON.stringify(payload))`. -##### Validating Incoming Requests +##### 5.1.3.8. Validating Incoming Requests The webhook listener receives the JWT in the body of the request, extracts the Key ID (`kid`), fetches the provider's public key from their JWKS URL, and verifies the signature. This allows each side to validate the caller's identity and ensure request integrity independently without relying on a third-party authority. Cocoon's webhook should also validate that the `uid` provided by the partner exists and has not been processed yet. -#### Pros & Cons +#### 5.1.4. Pros & Cons * **Pros:** * Industry-standard webhook architecture that is easy for partners to consume. @@ -492,13 +492,13 @@ The webhook listener receives the JWT in the body of the request, extracts the K * Downtime of partner endpoints requires backoff and retry queues on Cocoon. * Exposes inbound REST APIs on Cocoon that require JWT validation. -### Alternatives Considered +### 5.2. Alternatives Considered -#### Alternative 1: Event-Driven GCP Pub/Sub Integration (Asynchronous Event Bus) +#### 5.2.1. Event-Driven GCP Pub/Sub Integration (Asynchronous Event Bus) This architecture eliminates direct server-to-server HTTP calls by leveraging Google Cloud Pub/Sub as an event bus between Cocoon and partner infrastructure. -##### End-to-End Workflow Diagram +##### 5.2.1.1. End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -518,7 +518,7 @@ sequenceDiagram Note over Cocoon: Update Partner task status ``` -##### Workflow +##### 5.2.1.2. Workflow * **1. Trigger:** GitHub triggers post-commit processing in Cocoon. * **2. Outbound Event:** Cocoon schedules the partner task and publishes an event to an outbound Pub/Sub topic. @@ -527,7 +527,7 @@ sequenceDiagram * **5. Inbound Result:** Partner CI publishes results directly back to Cocoon's inbound Pub/Sub topic. * **6. Dashboard Update:** Cocoon ingests result messages and updates test execution states in the dashboard. -##### Pros & Cons +##### 5.2.1.3. Pros & Cons * **Pros:** * Highly asynchronous and resilient; messages are automatically buffered during outages. @@ -539,11 +539,11 @@ sequenceDiagram * Increases setup and development complexity for partners. * Cannot validate message body at the time of message submission. -#### Alternative 2: Automated Polling Strategy (Push Trigger, Pull Status) +#### 5.2.2. Alternative 2: Automated Polling Strategy (Push Trigger, Pull Status) This hybrid model retains HTTP push for triggering partner CI jobs, but replaces inbound callbacks with periodic polling by Cocoon to check test completion. -##### End-to-End Workflow Diagram +##### 5.2.2.1. End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -580,14 +580,14 @@ sequenceDiagram Cocoon-->>PubSub: ack ``` -##### Workflow +##### 5.2.2.2. Workflow * **1. Trigger & Dispatch:** GitHub triggers Cocoon, which creates partner tasks and pushes a webhook notification to the partner. * **2. Execution:** The partner CI executes workloads asynchronously. * **3. Polling Loop:** Cocoon automatically queries the partner's status endpoint periodically (e.g., using Pub/Sub retry loop). * **4. Inbound Processing:** Once the status endpoint returns a terminal state (`PASSED`/`FAILED`), Cocoon updates the dashboard. -##### Pros & Cons +##### 5.2.2.3. Pros & Cons * **Pros:** * Prevents partners from needing inbound access or security tokens to write to Cocoon endpoints. @@ -597,7 +597,7 @@ sequenceDiagram * Increases latency in reporting status updates back to the Flutter dashboard due to polling intervals. * Scale limitations and risk of denial-of-service (DoS) on partner endpoints under heavy load. -### Comparison of Options +### 5.3. Comparison of Options | Approach Criteria | Primary Design: Webhook Push-Callback | Alternative 1: Event-Driven GCP Pub/Sub | Alternative 2: Automated Polling (Push-Poll) | | --- | --- | --- | --- | @@ -609,13 +609,13 @@ sequenceDiagram | **Latency / Status Reporting** | ✅ **Real-time.** Push-based callback minimizes status reporting latency. | ✅ **Real-time.** Push-based consumption minimizes latency. | ❌ **Higher latency.** Updates are delayed based on Cocoon's polling intervals. | | **Resilience & Overhead** | ✅ **Highly resilient.** Retries are backed by Pub/Sub; Status query endpoint prevents polling loops. | ✅ **Highly resilient.** Messages are automatically buffered during partner infrastructure outages. | ❌ **High overhead.** Constant automated polling creates network load and risks DoS on partner endpoints. | -## Integration with Existing Features +## 6. Integration with Existing Features To support webhook flows, the following must be implemented in Cocoon: -### Firestore +### 6.1. Firestore -#### partner_config collection +#### 6.1.1. partner_config collection For partner configuration, add a `partner_config` collection with Document ID equal to `task_name`. @@ -630,7 +630,7 @@ keys_url string // Url to read partner key id to public key mapping Currently, there will be no UI for editing this table. For every post-submit dashboard checks will be extended with tasks from this list. All tasks from this list would be created suppressed without the ability to unsuppress. Later if a decision will be made to let some 1P partner CI block the tree we may extend this table with a `block_tree` boolean variable. -#### partner_ci collection +#### 6.1.2. partner_ci collection Add a `partner_ci` collection with Auto-ID Document ID that will be equal to the `uid` of the `data` payload. This is used to match post-submit tasks to corresponding status responses from partner CI systems. @@ -643,29 +643,29 @@ attempt_number int // Number of re-run attempt processed bool // Whether status response from partner was already processed ``` -#### tasks +#### 6.1.3. tasks Extend `tasks` collection with `message` and `log_urls` fields. -### Pub/Sub +### 6.2. Pub/Sub -#### partner_ci_notification topic +#### 6.2.1. partner_ci_notification topic To guarantee that Cocoon can notify each partner CI even during network outages or temporary partner infrastructure outages, once Cocoon reads the post-submit task configuration for a post-submit event, it will send a message to the `partner_ci_notification` topic for every partner integration. A separate instance of Cocoon will receive that message from `partner_ci_notification_sub` and attempt to post it to the partner webhook. -#### partner_ci_status topic +#### 6.2.2. partner_ci_status topic To guarantee that Cocoon can properly process status updates once the status webhook is called by a partner and the payload is successfully validated, the payload will be sent to the `partner_ci_status` topic and later received by Cocoon from `partner_ci_status_sub` to update the corresponding task status. -## Open Questions +## 7. Open Questions * What other fields should we provide to partners webhook to request CI run? * Can tasks PK `__` be used as `uid`? In that case we would not need a `partner_ci` collection. -## Testing Plan +## 8. Testing Plan Implement a dummy partner CI to test the end-to-end workflow. -## Documentation Plan +## 9. Documentation Plan Provide documentation for partners detailing integration setup and best practices for retry mechanisms. From 2a666b3775659edaae703cde0e335bdee9cae8f0 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 16:48:04 -0700 Subject: [PATCH 12/35] diagram theme --- rfc/610.0000-flutter-pluggable-ci.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index ce87357..f7d9c11 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -5,7 +5,7 @@ title: Flutter Pluggable CI Design description: Describes the integration with partner CI systems to trigger partner CI execution on post-submit PRs and display execution details on the post-submit dashboard. status: draft created: 2026-10-06T09:00:00Z -updated: 2026-10-06T09:00:00Z +updated: 2026-10-08T00:00:00Z tags: - 600-infrastructure - 610-cicd @@ -47,6 +47,10 @@ The proposed solution is to use webhooks for bidirectional asynchronous communic Below is a high-level flow diagram of the proposed communication between Cocoon and Partner CI: ```mermaid +--- +config: + theme: redux-color +--- sequenceDiagram participant PubSub@{"type":"queue"} actor GitHub as GitHub Repo From 63e8a6ec24db6b87b87155ee94883439d822cd35 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 16:49:53 -0700 Subject: [PATCH 13/35] default theme --- rfc/610.0000-flutter-pluggable-ci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index f7d9c11..6185873 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -49,7 +49,7 @@ Below is a high-level flow diagram of the proposed communication between Cocoon ```mermaid --- config: - theme: redux-color + theme: default --- sequenceDiagram participant PubSub@{"type":"queue"} From 92e43e9df5ec93feb929dc26ce10c75f0d8caa4f Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 16:53:12 -0700 Subject: [PATCH 14/35] remove theme --- rfc/610.0000-flutter-pluggable-ci.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 6185873..6dc5456 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -47,10 +47,6 @@ The proposed solution is to use webhooks for bidirectional asynchronous communic Below is a high-level flow diagram of the proposed communication between Cocoon and Partner CI: ```mermaid ---- -config: - theme: default ---- sequenceDiagram participant PubSub@{"type":"queue"} actor GitHub as GitHub Repo From 826d43fc423d52f0e3ede1b5543d77e6b8bf8969 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Wed, 7 Oct 2026 17:17:27 -0700 Subject: [PATCH 15/35] formatting --- rfc/610.0000-flutter-pluggable-ci.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 6dc5456..caaea91 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -423,9 +423,7 @@ When either system sends an HTTP request (Cocoon dispatching a webhook or a Part The JWT consists of a base64 encoded header, payload, and signature separated by dots. For example: -```text -eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA -``` +`eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` ##### 5.1.3.5. Decoded Header From 8e5bbda29f12baf98b9ee077a4a55c5b4ae45642 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 8 Oct 2026 10:28:13 -0700 Subject: [PATCH 16/35] added re-run failed task diagram --- rfc/610.0000-flutter-pluggable-ci.md | 83 ++++++++++++++++++++++++---- 1 file changed, 72 insertions(+), 11 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index caaea91..7632ab2 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -384,7 +384,68 @@ sequenceDiagram end ``` -#### 5.1.2. Flow Breakdown +#### 5.1.2. Re-run Failed Task Workflow Diagram + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + actor GitHub as GitHub Repo + participant Cocoon as Cocoon Server + participant Partner as Partner CI + destroy GitHub + GitHub->>+Cocoon: Postsubmit Event + Note over Cocoon: Creates Partner task for sha=xyzw with uid=abcd + Cocoon->>-PubSub: Notify partner message for sha=xyzw with uid=abcd + + alt Cocoon calls partner webhook to notify about post submit to run CI + PubSub->>+Cocoon: Notify partner message for sha=xyzw with uid=abcd + Cocoon->>+Partner: HTTP POST (signed with cocoon key, JWT body with notification for sha=xyzw with uid=abcd) + Partner->>Cocoon: HTTP GET of Cocoon JWKS url + Cocoon-->>Partner: JWKS to verify JWT signature + Cocoon-->>Partner: JWKS to verify JWT signature + Partner-->>Cocoon: Response 200 + Cocoon-->>-PubSub: ack + end + + Note over Partner: Executing custom tests + + alt Partner calls cocoon webhook to provide CI run status update + Partner->>+Cocoon: HTTP POST Callback (signed with partner key, JWT body with failed status for sha=xyzw with uid=abcd) + + Partner-->>Cocoon: JWKS to verify JWT signature + Cocoon->>PubSub: Partner test execution status for sha=xyzw with uid=abcd + Cocoon-->>-Partner: Response 200 + deactivate Partner + PubSub->>+Cocoon: Partner test execution status for sha=xyzw with uid=abcd + Note over Cocoon: Update Partner task status to failed for sha=xyzw with uid=abcd + deactivate Cocoon + end + + alt Cocoon calls partner webhook to re-run failed task + PubSub->>+Cocoon: Notify partner message for sha=xyzw with uid=efgh + Cocoon->>+Partner: HTTP POST (signed with cocoon key, JWT body with notification for sha=xyzw with uid=efgh) + Partner->>Cocoon: HTTP GET of Cocoon JWKS url + Cocoon-->>Partner: JWKS to verify JWT signature + Partner-->>Cocoon: Response 200 + Cocoon-->>-PubSub: ack + end + + Note over Partner: Executing custom tests + + alt Partner calls cocoon webhook to provide CI run status update + Partner->>+Cocoon: HTTP POST Callback (signed with partner key, JWT body with success status for sha=xyzw with uid=efgh) + Partner-->>Cocoon: JWKS to verify JWT signature + Cocoon->>PubSub: Partner test execution status for sha=xyzw with uid=efgh + Cocoon-->>-Partner: Response 200 + deactivate Partner + + PubSub->>+Cocoon: Partner test execution status for sha=xyzw with uid=efgh + Note over Cocoon: Update Partner task status to succeeded for sha=xyzw with uid=efgh + deactivate Cocoon + end +``` + +#### 5.1.3. Flow Breakdown * **1. Trigger:** GitHub notifies Cocoon via `/api/github/webhook-subscription` when a commit lands on `main`. * **2. Scheduling:** Cocoon schedules tasks on LUCI and creates dummy tasks in Firestore/Redis for each configured partner. @@ -395,37 +456,37 @@ sequenceDiagram * **7. Processing response:** Once Cocoon receives the partner status from Pub/Sub, it updates the partner task state in Firestore/Redis. * **8. Manual On-Demand Status Reconciliation:** Partners are required to host a status endpoint (e.g., `GET /status?uid=abcd`). In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. Automated polling is strictly rejected to avoid DoS risks and operational overhead. -#### 5.1.3. Security & Authentication +#### 5.1.4. Security & Authentication This section details the decentralized security model used to authorize server-to-server bidirectional webhook communication without a centralized authorization server. -##### 5.1.3.1. Asymmetric Cryptography & Key Management +##### 5.1.4.1. Asymmetric Cryptography & Key Management Both the Cocoon server and the Partner CI server independently generate their own asymmetric key pairs (using Ed25519). Each system stores its private key in a secure key management system and shares its corresponding public key with the other party. By leveraging decentralized asymmetric cryptography, this approach intentionally avoids the need for a central Authorization Server or Identity Provider, eliminating a potential single point of failure and reducing infrastructure complexity. -##### 5.1.3.2. Key Distribution Strategy: Public Key Endpoints +##### 5.1.4.2. Key Distribution Strategy: Public Key Endpoints For this authentication model to work, both parties must reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to fetch or refresh this key periodically. This is highly secure because it allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. Example of JWKS: `https://firebaseappcheck.googleapis.com/v1/jwks` -##### 5.1.3.3. Serialized JWT HTTP Body +##### 5.1.4.3. Serialized JWT HTTP Body When either system sends an HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender constructs a POST request as follows: * **Header**: `Content-Type: application/jwt` * **Body**: A signed JSON Web Token (JWT) -##### 5.1.3.4. JWT Payload Structure +##### 5.1.4.4. JWT Payload Structure The JWT consists of a base64 encoded header, payload, and signature separated by dots. For example: `eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` -##### 5.1.3.5. Decoded Header +##### 5.1.4.5. Decoded Header ```json { @@ -441,7 +502,7 @@ Where: * **typ**: The type of the token, which is set to `JWT`. * **kid**: Key ID that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. -##### 5.1.3.6. Decoded Payload +##### 5.1.4.6. Decoded Payload ```jsonc { @@ -460,7 +521,7 @@ Where: } ``` -##### 5.1.3.7. Decoded Signature +##### 5.1.4.7. Decoded Signature ```text Ed25519Sign(privateKeyEd25519, ASCII_OR_UTF8_BYTES(encodedHeader + "." + encodedPayload)) @@ -473,11 +534,11 @@ Where: * **encodedHeader** is a `Base64UrlEncode(JSON.stringify(header))`. * **encodedPayload** is a `Base64UrlEncode(JSON.stringify(payload))`. -##### 5.1.3.8. Validating Incoming Requests +##### 5.1.4.8. Validating Incoming Requests The webhook listener receives the JWT in the body of the request, extracts the Key ID (`kid`), fetches the provider's public key from their JWKS URL, and verifies the signature. This allows each side to validate the caller's identity and ensure request integrity independently without relying on a third-party authority. Cocoon's webhook should also validate that the `uid` provided by the partner exists and has not been processed yet. -#### 5.1.4. Pros & Cons +#### 5.1.5. Pros & Cons * **Pros:** * Industry-standard webhook architecture that is easy for partners to consume. From 22341efdb13a3b5c849b809ead394732f45be89d Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 8 Oct 2026 10:34:39 -0700 Subject: [PATCH 17/35] changed re-run to opt --- rfc/610.0000-flutter-pluggable-ci.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 7632ab2..6b54d41 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -421,7 +421,7 @@ sequenceDiagram deactivate Cocoon end - alt Cocoon calls partner webhook to re-run failed task + opt Cocoon calls partner webhook to re-run failed task PubSub->>+Cocoon: Notify partner message for sha=xyzw with uid=efgh Cocoon->>+Partner: HTTP POST (signed with cocoon key, JWT body with notification for sha=xyzw with uid=efgh) Partner->>Cocoon: HTTP GET of Cocoon JWKS url @@ -432,7 +432,7 @@ sequenceDiagram Note over Partner: Executing custom tests - alt Partner calls cocoon webhook to provide CI run status update + opt Partner calls cocoon webhook to provide CI run status update Partner->>+Cocoon: HTTP POST Callback (signed with partner key, JWT body with success status for sha=xyzw with uid=efgh) Partner-->>Cocoon: JWKS to verify JWT signature Cocoon->>PubSub: Partner test execution status for sha=xyzw with uid=efgh From b0cddde3ceacb318af419c6d51fe50d72180378e Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 8 Oct 2026 16:32:20 -0700 Subject: [PATCH 18/35] Address Codefu's review --- rfc/610.0000-flutter-pluggable-ci.md | 603 +++++++++++++-------------- 1 file changed, 301 insertions(+), 302 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 6b54d41..83e6a08 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -30,7 +30,7 @@ Multiple Flutter partners adopt custom embedders to support specific platforms n ### 2.2. Glossary -* **In-Tree Platforms:** These are core Flutter platforms that are stored, updated, and managed directly inside the main Flutter repository. This includes the official ports for iOS, Android, macOS, Linux, and Windows. +* **In-Tree Platforms:** These are core Flutter platforms that are stored, updated, and managed directly inside the main Flutter repository. This includes the official ports for iOS, Android, macOS, Linux, Windows and Web. * **Out-of-Tree Platforms:** These are external Flutter platforms developed and maintained by third parties in separate, independent repositories. Because they live outside the main codebase, today developers must fork the Flutter tool and reverse-engineer undocumented system channels (like text input or app lifecycles) to make them work. * **Custom Embedders:** These are platform-specific codebases that host the Flutter engine and Dart runtime on non-standard operating systems. They act as out-of-tree ports for custom hardware. * **Cocoon:** Flutter's custom Continuous Integration (CI) orchestration system that integrates underlying CI services like LUCI with GitHub to schedule tests and report build statuses. It also provides developer tooling such as auto-submit mechanisms, test flakiness tracking, and powers the public Flutter Pre-submit/Build Dashboards. @@ -40,9 +40,9 @@ Multiple Flutter partners adopt custom embedders to support specific platforms n To enable Flutter to scale across external platforms and out-of-tree embedder maintainers, Flutter infrastructure (Cocoon) requires a robust, secure, and decoupled communication model. This system notifies partner CI pipelines of post-submit commits on `flutter/flutter` so partners can execute custom test suites on proprietary hardware and report execution status back to the Flutter build dashboard. -Key architectural goals include non-blocking test execution, strict security via Ed25519 public/private key authentication, high resilience against network outages, and minimal operational overhead for Flutter infrastructure maintainers. +Key architectural goals include non-blocking test execution, strict security via [Ed25519](https://www.rfc-editor.org/info/rfc8410/) public/private key authentication, high resilience against network outages, and minimal operational overhead for Flutter infrastructure maintainers. -The proposed solution is to use webhooks for bidirectional asynchronous communication backed by GCP Pub/Sub for fault tolerance and retries. +The proposed solution is to use webhooks for bidirectional asynchronous communication backed internally by GCP Pub/Sub for fault tolerance and retries. Below is a high-level flow diagram of the proposed communication between Cocoon and Partner CI: @@ -81,255 +81,15 @@ sequenceDiagram This design does not cover what kind of CI for out-of-tree platforms to use and what tests to run to validate custom embedders. Nor does it cover how to guarantee the delivery of CI execution status from partner CI systems back to Cocoon. -## 4. Usage Examples - -### 4.1. Communication Example: Cocoon and 3rd Party Partner - -#### 4.1.1. Webhook Dispatch Request (Cocoon -> Partner) - -When a post-submit commit is processed, Cocoon posts a signed JWT to the partner's webhook URL. -See: [JSON Web Token (JWT) RFC 7519](https://www.rfc-editor.org/info/rfc7519/) for more details about JWT. - -##### 4.1.1.1. HTTP POST Request - -```http -POST /webhook HTTP/1.1 -Host: partner-ci.example.com -Content-Type: application/jwt - -.. -``` - -See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc7515/) for more details about Signature (JWS). - -##### 4.1.1.2. Decoded JWT Payload - -```json -{ - "iss": "cocoon", - "aud": "https://partner-ci.com/webhook", - "iat": 1774962000, - "exp": 1774962300, - "data": { - "uid": "0sAtLhwbmGz2Y0WBDzPL", - "slug": "flutter/flutter", - "pr": 12345, - "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b", - "task_name": "some_awesome_partners_test", - "tags": { - "attempt": "1", - "key2": "value2", - "key3": "value3" - } - } -} -``` - -Where: - -* **iss**: Issuer claim identifying the caller that issued the token (e.g., `cocoon`). -* **aud**: Audience claim specifying the intended recipient URL of the request. -* **iat**: Issued At timestamp (Unix epoch time) indicating when the token was created. -* **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. -* **data**: Object containing custom payload claims and event details: - * **uid**: The unique correlation identifier to match asynchronous CI execution request to future status response. - * **slug**: The target repository identifier (e.g., `flutter/flutter`). - * **pr**: The Pull Request number associated with the event. - * **sha**: The target git commit SHA hash. - * **task_name**: Name of task to be displayed on dashboard, to potentially support several CI types or platforms per partner. For only one task per partner can be ignored. - * **tags**: Optional key-value string pairs, for future usage. Should be returned as is in the status response. - -##### 4.1.1.3. HTTP Response - -```http -HTTP/1.1 200 OK -Content-Type: application/json - -{} -``` - -#### 4.1.2. Status Callback Request (Partner -> Cocoon) - -After executing custom test suites on proprietary hardware, the partner sends the final status back to Cocoon. - -##### 4.1.2.1. HTTP POST Request - -```http -POST /api/partner/callback HTTP/1.1 -Host: cocoon.flutter.dev -Content-Type: application/jwt - -.. -``` - -##### 4.1.2.2. Decoded Payload - -```json -{ - "iss": "3p-partner-ci", - "aud": "https://dashboard.flutter.dev/api/partner/callback", - "iat": 1774963500, - "exp": 1774963800, - "data": { - "uid": "0sAtLhwbmGz2Y0WBDzPL", - "tags": { - "attempt": "1", - "key2": "value2", - "key3": "value3" - }, - "status": "failed", - "message": "Some optional summary if needed", - "details": [ - { - "name": "test_1", - "status": "failed", - "log_url": "https://parnter.com/test_1/log" - }, - { - "name": "test_2", - "status": "success", - "log_url": "https://parnter-ci.com/test_2/log" - }, - { - "name": "test_3", - "status": "skipped", - "log_url": "https://parnter-ci.com/test_3/log" - }, - { - "name": "test_4", - "status": "failed", - "log_url": "https://parnter-ci.com/test_4/log" - }, - { - "name": "test_5", - "status": "cancelled", - "log_url": "https://parnter-ci.com/test_5/log" - } - ] - } -} -``` - -Where: - -* **iss**: Issuer claim identifying the caller that issued the token (e.g., partner). -* **aud**: Audience claim specifying the intended recipient URL of the request. -* **iat**: Issued At timestamp (Unix epoch time) indicating when the token was created. -* **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. -* **data**: Object containing custom payload claims and event details: - * **uid**: The unique correlation identifier to match which CI execution request this status response belongs to. - * **tags**: Optional key-value string pairs, for internal usage by Flutter CI. Should be returned as is if received in request. - * **status**: The terminal status of CI execution (cannot be `in_progress`): - * `cancelled` - Task is cancelled by a Partner CI (failure state). - * `infra_failure` - Infrastructure failure during tests execution (failure state). - * `failed` - Some or all of the tests are failed (failure state). - * `succeeded` - All tests passed (success state). - * `neutral` - Only some flaky tests failed (success state). - * `skipped` - Some or all tests are skipped (success state). - * **message**: Markdown formatted message with a summary of CI run. - * **details**: Granular test execution details: - * **name**: Test name. - * **status**: Terminal test status (cannot be `in_progress`): - * `cancelled` - Test is cancelled by a Partner CI (failure state). - * `infra_failure` - Infrastructure failure during test execution (failure state). - * `failed` - Test failed (failure state). - * `succeeded` - Test passed (success state). - * `neutral` - Test failed but considered flaky (success state). - * `skipped` - Test skipped (success state). - * **log_url**: The publicly accessible task execution log URL. - -##### 4.1.2.3. HTTP Response - -```http -HTTP/1.1 200 OK -Content-Type: application/json - -{} -``` - -#### 4.1.3. Manual On-Demand Status Reconciliation Request - -Cocoon can manually query the partner's status endpoint. - -##### 4.1.3.1. HTTP GET Request - -```http -GET /status?uid=0sAtLhwbmGz2Y0WBDzPL HTTP/1.1 -Host: partner-ci.example.com -Authorization: Bearer -``` - -##### 4.1.3.2. HTTP Response - -```http -HTTP/1.1 200 OK -Content-Type: application/json - -{ - "status": "in_progress", - "message": "Some optional summary if needed", - "details": [ - { - "name": "test_1", - "status": "failed", - "log_url": "https://parnter.com/test_1/log" - }, - { - "name": "test_2", - "status": "success", - "log_url": "https://parnter-ci.com/test_2/log" - }, - { - "name": "test_3", - "status": "skipped", - "log_url": "https://parnter-ci.com/test_3/log" - }, - { - "name": "test_4", - "status": "failed", - "log_url": "https://parnter-ci.com/test_4/log" - }, - { - "name": "test_5", - "status": "in_progress", - "log_url": null - } - ] -} -``` - -Where: - -* **status**: The status of CI execution: - * `in_progress` - Partner CI starts the execution of this task (in progress state). - * `cancelled` - Task is cancelled by a Partner CI (failure state). - * `infra_failure` - Infrastructure failure during test execution (failure state). - * `failed` - Some or all of the tests are failed (failure state). - * `succeeded` - All tests passed (success state). - * `neutral` - Only some flaky tests failed (success state). - * `skipped` - Some or all tests are skipped (success state). -* **message**: Markdown formatted message with a summary of CI run. -* **details**: Granular test execution details: - * **name**: Test name. - * **status**: Current test execution status: - * `in_progress` - Partner CI starts the execution of this task (in progress state). - * `cancelled` - Test is cancelled by a Partner CI (failure state). - * `infra_failure` - Infrastructure failure during test execution (failure state). - * `failed` - Test failed (failure state). - * `succeeded` - Test passed (success state). - * `neutral` - Test failed but considered flaky (success state). - * `skipped` - Test skipped (success state). - * **log_url**: The publicly accessible task execution log URL. - -## 5. Detailed Design +## 4. Detailed Design Integration with third-party CIs requires two main points of discussion: communication and authentication. For external system notification, the recommended approach is a bi-directional webhook server-to-server communication, combined with a manual on-demand status reconciliation mechanism. -### 5.1. Primary Design: Webhook Server-to-Server Communication (Push-Callback) with Manual Status Request +### 4.1. Primary Design: Webhook Server-to-Server Communication (Push-Callback) with Manual Status Request In this architecture, Cocoon uses standard HTTP webhooks to push event notifications to external partner endpoints. External partners process the event asynchronously on their custom infrastructure and send execution status back via Cocoon's callback REST API. Additionally, partners must host an endpoint capable of responding to on-demand status queries from Cocoon to enable manual status checks from the build dashboard, eliminating automated polling loops. -#### 5.1.1. End-to-End Workflow Diagram +#### 4.1.1. End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -384,7 +144,18 @@ sequenceDiagram end ``` -#### 5.1.2. Re-run Failed Task Workflow Diagram +#### 4.1.2. Flow Breakdown + +* **1. Trigger:** GitHub notifies Cocoon via `/api/github/webhook-subscription` when a commit lands on `main`. +* **2. Scheduling:** Cocoon schedules tasks on LUCI and creates dummy tasks in Firestore/Redis for each configured partner. +* **3. Outbound Queueing:** For every configured partner, Cocoon creates a record in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, and publishes a message containing the `uid` and commit context (`commit_sha`, `pr_number`, `repo`, etc.) to an internal GCP Pub/Sub topic for each partner. +* **4. Partner Dispatch:** A Pub/Sub subscriber fetches partner configurations, generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, and issues an HTTP POST webhook request to the partner's configured URL. Cocoon remains stateless and retries the dispatch via Pub/Sub if it timed out waiting for status response, receives 5xx (Server error), or 408 (Request Timeout) response. +* **5. Terminal unsuccessful response:** If a partner webhook responds with client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. +* **6. Execution & Callback:** The partner CI executes tests on proprietary hardware and makes an API callback to Cocoon with the execution status (`in_progress`, `cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), message, and log URLs. Cocoon reads the Key ID from the JWT header and does a lookup for the corresponding cached public key; if the public key is not found it attempts to re-read partner's JWKS and update cache; if the key is not found it responds with 401. If Cocoon successfully validates the payload and sends the partner status to Pub/Sub, it will respond with status code 200. Failure responses indicating that the partner CI system should retry sending the CI execution status are: 5xx and 408. In case of a JWT signature validation failure, Cocoon will respond with 401. If there is a validation error, it responds with 400; if the `uid` is not found, it responds with 404; if already processed, it responds with 409; if body size exceeds 8MB it responds with 413. +* **7. Processing response:** Once Cocoon receives the partner status from Pub/Sub, it updates the partner task state in Firestore/Redis. +* **8. Manual On-Demand Status Reconciliation:** Partners are required to host a status endpoint (e.g., `GET /status?uid=abcd`). In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. Automated polling is strictly rejected to avoid DoS risks and operational overhead. + +#### 4.1.3. Re-run Failed Task Workflow Diagram ```mermaid sequenceDiagram @@ -445,48 +216,47 @@ sequenceDiagram end ``` -#### 5.1.3. Flow Breakdown +#### 4.1.4. Re-run Failed Flow Breakdown -* **1. Trigger:** GitHub notifies Cocoon via `/api/github/webhook-subscription` when a commit lands on `main`. -* **2. Scheduling:** Cocoon schedules tasks on LUCI and creates dummy tasks in Firestore/Redis for each configured partner. -* **3. Outbound Queueing:** For every configured partner, Cocoon creates a record in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, and publishes a message containing the `uid` and commit context (`commit_sha`, `pr_number`, `repo`, etc.) to an internal GCP Pub/Sub topic for each partner. -* **4. Partner Dispatch:** A Pub/Sub subscriber fetches partner configurations, generates a short-lived JWT signed with an Ed25519 private key, and issues an HTTP POST webhook request to the partner's configured URL. Cocoon remains stateless and retries the dispatch via Pub/Sub if it timed out waiting for status response, receives 5xx (Server error), or 408 (Request Timeout) response. -* **5. Terminal unsuccessful response:** If a partner webhook responds with client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. -* **6. Execution & Callback:** The partner CI executes tests on proprietary hardware and makes an API callback to Cocoon with the execution status (`in_progress`, `cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), message, and log URLs. Cocoon reads the Key ID from the JWT header and does a lookup for the corresponding cached public key; if the public key is not found it attempts to re-read partner's JWKS and update cache; if the key is not found it responds with 401. If Cocoon successfully validates the payload and sends the partner status to Pub/Sub, it will respond with status code 200. Failure responses indicating that the partner CI system should retry sending the CI execution status are: 5xx and 408. In case of a JWT signature validation failure, Cocoon will respond with 401. If there is a validation error, it responds with 400; if the `uid` is not found, it responds with 404; if already processed, it responds with 409; if body size exceeds 8MB it responds with 413. -* **7. Processing response:** Once Cocoon receives the partner status from Pub/Sub, it updates the partner task state in Firestore/Redis. -* **8. Manual On-Demand Status Reconciliation:** Partners are required to host a status endpoint (e.g., `GET /status?uid=abcd`). In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. Automated polling is strictly rejected to avoid DoS risks and operational overhead. +* **1. Initial Trigger & Dispatch:** GitHub notifies Cocoon of a post-submit commit (`sha=xyzw`). Cocoon creates an initial task record with a unique correlation identifier (`uid=abcd`) and publishes a notification to GCP Pub/Sub, which dispatches a signed JWT webhook to the Partner CI. +* **2. Initial Execution & Failure Callback:** The Partner CI verifies Cocoon's JWT signature via Cocoon's JWKS endpoint, runs its custom test suite for `sha=xyzw`, and reports a `failed` execution status back to Cocoon's callback endpoint using `uid=abcd`. +* **3. Failure Processing:** Cocoon validates the partner's JWT signature via the partner's JWKS endpoint, queues the status update in Pub/Sub, and marks the task state as `failed` in Firestore for `sha=xyzw` (`uid=abcd`). +* **4. Re-run Scheduling:** When a task re-run is initiated for `sha=xyzw` (either automatically by Cocoon's flake/retry policy or manually from the build dashboard), Cocoon generates a new unique correlation ID (`uid=efgh`) for the new attempt while retaining the original commit context (`sha=xyzw`), and publishes a new dispatch message to Pub/Sub. +* **5. Re-run Partner Dispatch:** The Pub/Sub subscriber sends a new signed JWT HTTP POST notification to the Partner CI webhook containing `sha=xyzw` and `uid=efgh`. Issuing a distinct `uid` ensures the partner treats the request as a new execution rather than rejecting it as a duplicate (`409 Conflict`) of `uid=abcd`. +* **6. Re-run Execution & Callback:** The Partner CI verifies the new JWT signature, re-executes the custom tests for `sha=xyzw`, and sends a signed JWT callback to Cocoon reporting the updated status (e.g., `succeeded`) associated with `uid=efgh`. +* **7. Final Status Update:** Cocoon verifies the callback signature, processes the status update through Pub/Sub, and updates the task status in Firestore to `succeeded` for `sha=xyzw` (`uid=efgh`). -#### 5.1.4. Security & Authentication +#### 4.1.5. Security & Authentication This section details the decentralized security model used to authorize server-to-server bidirectional webhook communication without a centralized authorization server. -##### 5.1.4.1. Asymmetric Cryptography & Key Management +##### 4.1.5.1. Asymmetric Cryptography & Key Management -Both the Cocoon server and the Partner CI server independently generate their own asymmetric key pairs (using Ed25519). Each system stores its private key in a secure key management system and shares its corresponding public key with the other party. +Both the Cocoon server and the Partner CI server independently generate their own asymmetric key pairs (using [Ed25519](https://www.rfc-editor.org/info/rfc8410/)). Each system stores its private key in a secure key management system and shares its corresponding public key with the other party. By leveraging decentralized asymmetric cryptography, this approach intentionally avoids the need for a central Authorization Server or Identity Provider, eliminating a potential single point of failure and reducing infrastructure complexity. -##### 5.1.4.2. Key Distribution Strategy: Public Key Endpoints +##### 4.1.5.2. Key Distribution Strategy: Public Key Endpoints For this authentication model to work, both parties must reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to fetch or refresh this key periodically. This is highly secure because it allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. -Example of JWKS: `https://firebaseappcheck.googleapis.com/v1/jwks` +Example of JWKS: https://firebaseappcheck.googleapis.com/v1/jwks -##### 5.1.4.3. Serialized JWT HTTP Body +##### 4.1.5.3. Serialized JWT HTTP Body When either system sends an HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender constructs a POST request as follows: * **Header**: `Content-Type: application/jwt` * **Body**: A signed JSON Web Token (JWT) -##### 5.1.4.4. JWT Payload Structure +##### 4.1.5.4. JWT Payload Structure The JWT consists of a base64 encoded header, payload, and signature separated by dots. For example: `eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` -##### 5.1.4.5. Decoded Header +##### 4.1.5.5. JWT Header ```json { @@ -498,11 +268,11 @@ For example: Where: -* **alg**: The cryptographic algorithm used to sign the token (e.g., `EdDSA` for Ed25519 signatures). +* **alg**: The cryptographic algorithm used to sign the token (e.g., `EdDSA` for [Ed25519](https://www.rfc-editor.org/info/rfc8410/) signatures). * **typ**: The type of the token, which is set to `JWT`. * **kid**: Key ID that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. -##### 5.1.4.6. Decoded Payload +##### 4.1.5.6. JWT Payload ```jsonc { @@ -521,24 +291,15 @@ Where: } ``` -##### 5.1.4.7. Decoded Signature +##### 4.1.5.7. JWT Signature -```text -Ed25519Sign(privateKeyEd25519, ASCII_OR_UTF8_BYTES(encodedHeader + "." + encodedPayload)) -``` +[Ed25519 signature](https://www.rfc-editor.org/info/rfc8037/#appendix-A.4) as defined in [RFC 8037](https://www.rfc-editor.org/info/rfc8037). -Where: - -* **Ed25519Sign** is an EdDSA/Ed25519 function for deterministic signatures. -* **privateKeyEd25519** is an Ed25519 private key. -* **encodedHeader** is a `Base64UrlEncode(JSON.stringify(header))`. -* **encodedPayload** is a `Base64UrlEncode(JSON.stringify(payload))`. - -##### 5.1.4.8. Validating Incoming Requests +##### 4.1.5.8. Validating Incoming Requests The webhook listener receives the JWT in the body of the request, extracts the Key ID (`kid`), fetches the provider's public key from their JWKS URL, and verifies the signature. This allows each side to validate the caller's identity and ensure request integrity independently without relying on a third-party authority. Cocoon's webhook should also validate that the `uid` provided by the partner exists and has not been processed yet. -#### 5.1.5. Pros & Cons +#### 4.1.6. Pros & Cons * **Pros:** * Industry-standard webhook architecture that is easy for partners to consume. @@ -548,16 +309,14 @@ The webhook listener receives the JWT in the body of the request, extracts the K * Status query endpoints allow robust reconciliation on-demand without automated polling loops. * **Cons:** * Requires partners to maintain publicly accessible HTTP endpoints and handle inbound requests. - * Downtime of partner endpoints requires backoff and retry queues on Cocoon. - * Exposes inbound REST APIs on Cocoon that require JWT validation. -### 5.2. Alternatives Considered +### 4.2. Alternatives Considered -#### 5.2.1. Event-Driven GCP Pub/Sub Integration (Asynchronous Event Bus) +#### 4.2.1. Event-Driven GCP Pub/Sub Integration (Asynchronous Event Bus) This architecture eliminates direct server-to-server HTTP calls by leveraging Google Cloud Pub/Sub as an event bus between Cocoon and partner infrastructure. -##### 5.2.1.1. End-to-End Workflow Diagram +##### 4.2.1.1. End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -577,7 +336,7 @@ sequenceDiagram Note over Cocoon: Update Partner task status ``` -##### 5.2.1.2. Workflow +##### 4.2.1.2. Workflow * **1. Trigger:** GitHub triggers post-commit processing in Cocoon. * **2. Outbound Event:** Cocoon schedules the partner task and publishes an event to an outbound Pub/Sub topic. @@ -586,7 +345,7 @@ sequenceDiagram * **5. Inbound Result:** Partner CI publishes results directly back to Cocoon's inbound Pub/Sub topic. * **6. Dashboard Update:** Cocoon ingests result messages and updates test execution states in the dashboard. -##### 5.2.1.3. Pros & Cons +##### 4.2.1.3. Pros & Cons * **Pros:** * Highly asynchronous and resilient; messages are automatically buffered during outages. @@ -598,11 +357,11 @@ sequenceDiagram * Increases setup and development complexity for partners. * Cannot validate message body at the time of message submission. -#### 5.2.2. Alternative 2: Automated Polling Strategy (Push Trigger, Pull Status) +#### 4.2.2. Alternative 2: Automated Polling Strategy (Push Trigger, Pull Status) This hybrid model retains HTTP push for triggering partner CI jobs, but replaces inbound callbacks with periodic polling by Cocoon to check test completion. -##### 5.2.2.1. End-to-End Workflow Diagram +##### 4.2.2.1. End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -639,14 +398,14 @@ sequenceDiagram Cocoon-->>PubSub: ack ``` -##### 5.2.2.2. Workflow +##### 4.2.2.2. Workflow * **1. Trigger & Dispatch:** GitHub triggers Cocoon, which creates partner tasks and pushes a webhook notification to the partner. * **2. Execution:** The partner CI executes workloads asynchronously. * **3. Polling Loop:** Cocoon automatically queries the partner's status endpoint periodically (e.g., using Pub/Sub retry loop). * **4. Inbound Processing:** Once the status endpoint returns a terminal state (`PASSED`/`FAILED`), Cocoon updates the dashboard. -##### 5.2.2.3. Pros & Cons +##### 4.2.2.3. Pros & Cons * **Pros:** * Prevents partners from needing inbound access or security tokens to write to Cocoon endpoints. @@ -656,7 +415,7 @@ sequenceDiagram * Increases latency in reporting status updates back to the Flutter dashboard due to polling intervals. * Scale limitations and risk of denial-of-service (DoS) on partner endpoints under heavy load. -### 5.3. Comparison of Options +### 4.3. Comparison of Options | Approach Criteria | Primary Design: Webhook Push-Callback | Alternative 1: Event-Driven GCP Pub/Sub | Alternative 2: Automated Polling (Push-Poll) | | --- | --- | --- | --- | @@ -668,13 +427,13 @@ sequenceDiagram | **Latency / Status Reporting** | ✅ **Real-time.** Push-based callback minimizes status reporting latency. | ✅ **Real-time.** Push-based consumption minimizes latency. | ❌ **Higher latency.** Updates are delayed based on Cocoon's polling intervals. | | **Resilience & Overhead** | ✅ **Highly resilient.** Retries are backed by Pub/Sub; Status query endpoint prevents polling loops. | ✅ **Highly resilient.** Messages are automatically buffered during partner infrastructure outages. | ❌ **High overhead.** Constant automated polling creates network load and risks DoS on partner endpoints. | -## 6. Integration with Existing Features +## 5. Integration with Existing Features To support webhook flows, the following must be implemented in Cocoon: -### 6.1. Firestore +### 5.1. Firestore -#### 6.1.1. partner_config collection +#### 5.1.1. partner_config collection For partner configuration, add a `partner_config` collection with Document ID equal to `task_name`. @@ -689,7 +448,7 @@ keys_url string // Url to read partner key id to public key mapping Currently, there will be no UI for editing this table. For every post-submit dashboard checks will be extended with tasks from this list. All tasks from this list would be created suppressed without the ability to unsuppress. Later if a decision will be made to let some 1P partner CI block the tree we may extend this table with a `block_tree` boolean variable. -#### 6.1.2. partner_ci collection +#### 5.1.2. partner_ci collection Add a `partner_ci` collection with Auto-ID Document ID that will be equal to the `uid` of the `data` payload. This is used to match post-submit tasks to corresponding status responses from partner CI systems. @@ -702,29 +461,269 @@ attempt_number int // Number of re-run attempt processed bool // Whether status response from partner was already processed ``` -#### 6.1.3. tasks +#### 5.1.3. tasks Extend `tasks` collection with `message` and `log_urls` fields. -### 6.2. Pub/Sub +### 5.2. Pub/Sub -#### 6.2.1. partner_ci_notification topic +#### 5.2.1. partner_ci_notification topic To guarantee that Cocoon can notify each partner CI even during network outages or temporary partner infrastructure outages, once Cocoon reads the post-submit task configuration for a post-submit event, it will send a message to the `partner_ci_notification` topic for every partner integration. A separate instance of Cocoon will receive that message from `partner_ci_notification_sub` and attempt to post it to the partner webhook. -#### 6.2.2. partner_ci_status topic +#### 5.2.2. partner_ci_status topic To guarantee that Cocoon can properly process status updates once the status webhook is called by a partner and the payload is successfully validated, the payload will be sent to the `partner_ci_status` topic and later received by Cocoon from `partner_ci_status_sub` to update the corresponding task status. -## 7. Open Questions +## 6. Open Questions * What other fields should we provide to partners webhook to request CI run? * Can tasks PK `__` be used as `uid`? In that case we would not need a `partner_ci` collection. -## 8. Testing Plan +## 7. Testing Plan Implement a dummy partner CI to test the end-to-end workflow. -## 9. Documentation Plan +## 8. Documentation Plan Provide documentation for partners detailing integration setup and best practices for retry mechanisms. + +## 9. Appendix A. Usage Examples + +### 9.1. Communication Example: Cocoon and 3rd Party Partner + +#### 9.1.1. Webhook Dispatch Request (Cocoon -> Partner) + +When a post-submit commit is processed, Cocoon posts a signed JWT to the partner's webhook URL. +See: [JSON Web Token (JWT) RFC 7519](https://www.rfc-editor.org/info/rfc7519/) for more details about JWT. + +##### 9.1.1.1. HTTP POST Request + +```http +POST /webhook HTTP/1.1 +Host: partner-ci.example.com +Content-Type: application/jwt + +.. +``` + +See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc7515/) for more details about Signature (JWS). + +##### 9.1.1.2. Decoded JWT Payload + +```json +{ + "iss": "cocoon", + "aud": "https://partner-ci.com/webhook", + "iat": 1774962000, + "exp": 1774962300, + "data": { + "uid": "0sAtLhwbmGz2Y0WBDzPL", + "slug": "flutter/flutter", + "pr": 12345, + "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b", + "task_name": "some_awesome_partners_test", + "tags": { + "attempt": "1", + "key2": "value2", + "key3": "value3" + } + } +} +``` + +Where: + +* **iss**: Issuer claim identifying the caller that issued the token (e.g., `cocoon`). +* **aud**: Audience claim specifying the intended recipient URL of the request. +* **iat**: Issued At timestamp (Unix epoch time) indicating when the token was created. +* **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. +* **data**: Object containing custom payload claims and event details: + * **uid**: The unique correlation identifier to match asynchronous CI execution request to future status response. + * **slug**: The target repository identifier (e.g., `flutter/flutter`). + * **pr**: The Pull Request number associated with the event. + * **sha**: The target git commit SHA hash. + * **task_name**: Name of task to be displayed on dashboard, to potentially support several CI types or platforms per partner. For only one task per partner can be ignored. + * **tags**: Optional key-value string pairs, for future usage. Should be returned as is in the status response. + +##### 9.1.1.3. HTTP Response + +```http +HTTP/1.1 200 OK +Content-Type: application/json + +{} +``` + +#### 9.1.2. Status Callback Request (Partner -> Cocoon) + +After executing custom test suites on proprietary hardware, the partner sends the final status back to Cocoon. + +##### 9.1.2.1. HTTP POST Request + +```http +POST /api/partner/callback HTTP/1.1 +Host: cocoon.flutter.dev +Content-Type: application/jwt + +.. +``` + +##### 9.1.2.2. Decoded Payload + +```json +{ + "iss": "3p-partner-ci", + "aud": "https://dashboard.flutter.dev/api/partner/callback", + "iat": 1774963500, + "exp": 1774963800, + "data": { + "uid": "0sAtLhwbmGz2Y0WBDzPL", + "tags": { + "attempt": "1", + "key2": "value2", + "key3": "value3" + }, + "status": "failed", + "message": "Some optional summary if needed", + "details": [ + { + "name": "test_1", + "status": "failed", + "log_url": "https://parnter.com/test_1/log" + }, + { + "name": "test_2", + "status": "success", + "log_url": "https://parnter-ci.com/test_2/log" + }, + { + "name": "test_3", + "status": "skipped", + "log_url": "https://parnter-ci.com/test_3/log" + }, + { + "name": "test_4", + "status": "failed", + "log_url": "https://parnter-ci.com/test_4/log" + }, + { + "name": "test_5", + "status": "cancelled", + "log_url": "https://parnter-ci.com/test_5/log" + } + ] + } +} +``` + +Where: + +* **iss**: Issuer claim identifying the caller that issued the token (e.g., partner). +* **aud**: Audience claim specifying the intended recipient URL of the request. +* **iat**: Issued At timestamp (Unix epoch time) indicating when the token was created. +* **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. +* **data**: Object containing custom payload claims and event details: + * **uid**: The unique correlation identifier to match which CI execution request this status response belongs to. + * **tags**: Optional key-value string pairs, for internal usage by Flutter CI. Should be returned as is if received in request. + * **status**: The terminal status of CI execution (cannot be `in_progress`): + * `cancelled` - Task is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during tests execution (failure state). + * `failed` - Some or all of the tests are failed (failure state). + * `succeeded` - All tests passed (success state). + * `neutral` - Only some flaky tests failed (success state). + * `skipped` - Some or all tests are skipped (success state). + * **message**: Markdown formatted message with a summary of CI run. + * **details**: Granular test execution details: + * **name**: Test name. + * **status**: Terminal test status (cannot be `in_progress`): + * `cancelled` - Test is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during test execution (failure state). + * `failed` - Test failed (failure state). + * `succeeded` - Test passed (success state). + * `neutral` - Test failed but considered flaky (success state). + * `skipped` - Test skipped (success state). + * **log_url**: The publicly accessible task execution log URL. + +##### 9.1.2.3. HTTP Response + +```http +HTTP/1.1 200 OK +Content-Type: application/json + +{} +``` + +#### 9.1.3. Manual On-Demand Status Reconciliation Request + +For on-demand execution triggered by Tree Gardeners, Cocoon can manually query the partner's status endpoint. + +##### 9.1.3.1. HTTP GET Request + +```http +GET /status?uid=0sAtLhwbmGz2Y0WBDzPL HTTP/1.1 +Host: partner-ci.example.com +Authorization: Bearer +``` + +##### 9.1.3.2. HTTP Response + +```http +HTTP/1.1 200 OK +Content-Type: application/json + +{ + "status": "in_progress", + "message": "Some optional summary if needed", + "details": [ + { + "name": "test_1", + "status": "failed", + "log_url": "https://parnter.com/test_1/log" + }, + { + "name": "test_2", + "status": "success", + "log_url": "https://parnter-ci.com/test_2/log" + }, + { + "name": "test_3", + "status": "skipped", + "log_url": "https://parnter-ci.com/test_3/log" + }, + { + "name": "test_4", + "status": "failed", + "log_url": "https://parnter-ci.com/test_4/log" + }, + { + "name": "test_5", + "status": "in_progress", + "log_url": null + } + ] +} +``` + +Where: + +* **status**: The status of CI execution: + * `in_progress` - Partner CI starts the execution of this task (in progress state). + * `cancelled` - Task is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during test execution (failure state). + * `failed` - Some or all of the tests are failed (failure state). + * `succeeded` - All tests passed (success state). + * `neutral` - Only some flaky tests failed (success state). + * `skipped` - Some or all tests are skipped (success state). +* **message**: Markdown formatted message with a summary of CI run. +* **details**: Granular test execution details: + * **name**: Test name. + * **status**: Current test execution status: + * `in_progress` - Partner CI starts the execution of this task (in progress state). + * `cancelled` - Test is cancelled by a Partner CI (failure state). + * `infra_failure` - Infrastructure failure during test execution (failure state). + * `failed` - Test failed (failure state). + * `succeeded` - Test passed (success state). + * `neutral` - Test failed but considered flaky (success state). + * `skipped` - Test skipped (success state). + * **log_url**: The publicly accessible task execution log URL. From ac3b9c3e75997e5947791326f6ec7669003df44f Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 8 Oct 2026 16:36:54 -0700 Subject: [PATCH 19/35] fix no-bare-urls --- rfc/610.0000-flutter-pluggable-ci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 83e6a08..6662a0c 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -240,7 +240,7 @@ By leveraging decentralized asymmetric cryptography, this approach intentionally For this authentication model to work, both parties must reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to fetch or refresh this key periodically. This is highly secure because it allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. -Example of JWKS: https://firebaseappcheck.googleapis.com/v1/jwks +Example of JWKS: ##### 4.1.5.3. Serialized JWT HTTP Body From 6d6dfbf41be2ec4a4bfd8ff6b649204bdc694998 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Fri, 9 Oct 2026 12:11:12 -0700 Subject: [PATCH 20/35] pointintg to specific rfc's for request validation --- rfc/610.0000-flutter-pluggable-ci.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 6662a0c..c32e43c 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -297,7 +297,13 @@ Where: ##### 4.1.5.8. Validating Incoming Requests -The webhook listener receives the JWT in the body of the request, extracts the Key ID (`kid`), fetches the provider's public key from their JWKS URL, and verifies the signature. This allows each side to validate the caller's identity and ensure request integrity independently without relying on a third-party authority. Cocoon's webhook should also validate that the `uid` provided by the partner exists and has not been processed yet. +Webhook listeners validate incoming requests in accordance with standard JWT/JWS/JWKS specifications and security best practices, specifically: +* [RFC 7519 Section 7.2](https://www.rfc-editor.org/info/rfc7519/#section-7.2) for JWT validation; +* [RFC 7515 Section 5.2](https://www.rfc-editor.org/info/rfc7515/#section-5.2) for signature verification; +* [RFC 7517 Section 4.5](https://www.rfc-editor.org/info/rfc7517/#section-4.5) & [Section 5](https://www.rfc-editor.org/info/rfc7517/#section-5) for `kid` lookup via cached JWKS; +* [RFC 8725 Section 3](https://www.rfc-editor.org/info/rfc8725/#section-3) for algorithm pinning and `iss`/`aud`/`exp` claim checks. + +In addition, Cocoon's callback endpoint validates that `data.uid` exists and has not been processed yet. #### 4.1.6. Pros & Cons From a16daddf5a2d02157ed9747be73e3c828385ebcf Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Mon, 12 Oct 2026 10:46:50 -0700 Subject: [PATCH 21/35] respond to all codefu comments. split flow diagram to multiple logical parts --- rfc/610.0000-flutter-pluggable-ci.md | 218 ++++++++++++--------------- 1 file changed, 99 insertions(+), 119 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index c32e43c..f4a30db 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -91,110 +91,88 @@ In this architecture, Cocoon uses standard HTTP webhooks to push event notificat #### 4.1.1. End-to-End Workflow Diagram +Internal processing of post submit event. ```mermaid sequenceDiagram participant PubSub@{"type":"queue"} actor GitHub as GitHub Repo participant Cocoon as Cocoon Server - participant Partner as Partner CI destroy GitHub GitHub->>+Cocoon: Postsubmit Event - Note over Cocoon: Creates Partner task with uid=abcd - Cocoon->>-PubSub: Notify partner message with uid=abcd - - alt Cocoon calls partner webhook to notify about post submit to run CI, retrying on failures - PubSub->>+Cocoon: Notify partner message with uid=abcd - Cocoon->>+Partner: HTTP POST (signed with cocoon key, JWT body with notification for uid=abcd) - Partner->>Cocoon: HTTP GET of Cocoon JWKS url - Cocoon-->>Partner: JWKS to verify JWT signature - Note over Partner: Internal Failure during processing the request - Partner-->>-Cocoon: Response 500 - Cocoon-->>-PubSub: nack - - PubSub->>+Cocoon: Retry notify partner message with uid=abcd - Cocoon->>+Partner: Retry HTTP POST (signed with cocoon key, JWT body with notification for uid=abcd) - Partner->>Cocoon: HTTP GET of Cocoon JWKS url - Cocoon-->>Partner: JWKS to verify JWT signature - Partner-->>Cocoon: Response 200 - Cocoon-->>-PubSub: ack - end - - opt Manual status request - Note over Cocoon: User requested task status while waiting execution results - Cocoon->>Partner: HTTP GET /status?uid=abcd - Partner-->>Cocoon: Response 202 {"status": "in_progress"} - end - - Note over Partner: Executing custom tests - - alt Partner calls cocoon webhook to provide CI run status update, retrying on failures - Partner->>Cocoon: HTTP POST Callback (signed with partner key, JWT body with status for uid=abcd) - Note over Partner: Response timeout - - Partner->>+Cocoon: Retry HTTP POST (signed with partner key, JWT body with status for uid=abcd) - Cocoon->>Partner: HTTP GET of Partner JWKS url - Partner-->>Cocoon: JWKS to verify JWT signature - Cocoon->>PubSub: Partner test execution status for uid=abcd - Cocoon-->>-Partner: Response 200 - deactivate Partner + Note over Cocoon: Creates task for sha=wxyz with uid=abcd + Cocoon->>-PubSub: Schedule partner notification
event for sha=wxyz with uid=abcd +``` +Cocoon calls partner webhook to notify about post submit to run CI, retrying on failures. +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + participant Cocoon as Cocoon Server + participant Partner as Partner CI - PubSub->>+Cocoon: Partner test execution status for uid=abcd - Note over Cocoon: Update Partner task status for uid=abcd - deactivate Cocoon - end + PubSub->>+Cocoon: Receive partner notification
event for sha=wxyz with uid=abcd + Cocoon->>+Partner: HTTP POST (signed with cocoon key,
JWT body with notification for sha=wxyz with uid=abcd) + Partner->>Cocoon: HTTP GET of Cocoon JWKS url + Cocoon-->>Partner: JWKS to verify JWT signature + Note over Partner: Internal Failure during processing the request + Partner-->>-Cocoon: Response 500 + Cocoon-->>-PubSub: nack + + PubSub->>+Cocoon: Retry partner notification
event for sha=wxyz with uid=abcd + Cocoon->>+Partner: Retry HTTP POST (signed with cocoon key,
JWT body with notification for sha=wxyz with uid=abcd) + Partner->>Cocoon: HTTP GET of Cocoon JWKS url + Cocoon-->>Partner: JWKS to verify JWT signature + Partner-->>Cocoon: Response 200 + Cocoon-->>-PubSub: ack + Note over Partner: Start executing custom tests ``` -#### 4.1.2. Flow Breakdown +Optionally, if supported by Partner, Cocoon can poll for status updates. +```mermaid +sequenceDiagram + participant Cocoon as Cocoon Server + participant Partner as Partner CI -* **1. Trigger:** GitHub notifies Cocoon via `/api/github/webhook-subscription` when a commit lands on `main`. -* **2. Scheduling:** Cocoon schedules tasks on LUCI and creates dummy tasks in Firestore/Redis for each configured partner. -* **3. Outbound Queueing:** For every configured partner, Cocoon creates a record in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, and publishes a message containing the `uid` and commit context (`commit_sha`, `pr_number`, `repo`, etc.) to an internal GCP Pub/Sub topic for each partner. -* **4. Partner Dispatch:** A Pub/Sub subscriber fetches partner configurations, generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, and issues an HTTP POST webhook request to the partner's configured URL. Cocoon remains stateless and retries the dispatch via Pub/Sub if it timed out waiting for status response, receives 5xx (Server error), or 408 (Request Timeout) response. -* **5. Terminal unsuccessful response:** If a partner webhook responds with client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. -* **6. Execution & Callback:** The partner CI executes tests on proprietary hardware and makes an API callback to Cocoon with the execution status (`in_progress`, `cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), message, and log URLs. Cocoon reads the Key ID from the JWT header and does a lookup for the corresponding cached public key; if the public key is not found it attempts to re-read partner's JWKS and update cache; if the key is not found it responds with 401. If Cocoon successfully validates the payload and sends the partner status to Pub/Sub, it will respond with status code 200. Failure responses indicating that the partner CI system should retry sending the CI execution status are: 5xx and 408. In case of a JWT signature validation failure, Cocoon will respond with 401. If there is a validation error, it responds with 400; if the `uid` is not found, it responds with 404; if already processed, it responds with 409; if body size exceeds 8MB it responds with 413. -* **7. Processing response:** Once Cocoon receives the partner status from Pub/Sub, it updates the partner task state in Firestore/Redis. -* **8. Manual On-Demand Status Reconciliation:** Partners are required to host a status endpoint (e.g., `GET /status?uid=abcd`). In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. Automated polling is strictly rejected to avoid DoS risks and operational overhead. + Note over Cocoon: User requested task status while waiting execution results + Cocoon->>Partner: HTTP GET /status?uid=abcd + Partner-->>Cocoon: Response 202 {"status": "in_progress"} -#### 4.1.3. Re-run Failed Task Workflow Diagram + Note over Partner: Continue executing custom tests +``` +Partner calls cocoon webhook to provide CI run status update, retrying on failures. ```mermaid sequenceDiagram participant PubSub@{"type":"queue"} - actor GitHub as GitHub Repo participant Cocoon as Cocoon Server participant Partner as Partner CI - destroy GitHub - GitHub->>+Cocoon: Postsubmit Event - Note over Cocoon: Creates Partner task for sha=xyzw with uid=abcd - Cocoon->>-PubSub: Notify partner message for sha=xyzw with uid=abcd - - alt Cocoon calls partner webhook to notify about post submit to run CI - PubSub->>+Cocoon: Notify partner message for sha=xyzw with uid=abcd - Cocoon->>+Partner: HTTP POST (signed with cocoon key, JWT body with notification for sha=xyzw with uid=abcd) - Partner->>Cocoon: HTTP GET of Cocoon JWKS url - Cocoon-->>Partner: JWKS to verify JWT signature - Cocoon-->>Partner: JWKS to verify JWT signature - Partner-->>Cocoon: Response 200 - Cocoon-->>-PubSub: ack - end - - Note over Partner: Executing custom tests - - alt Partner calls cocoon webhook to provide CI run status update - Partner->>+Cocoon: HTTP POST Callback (signed with partner key, JWT body with failed status for sha=xyzw with uid=abcd) + + Note over Partner: Finishes executing custom tests + Partner->>Cocoon: HTTP POST Callback (signed with partner key,
JWT body with status for sha=wxyz with uid=abcd) + Note over Partner: Response timeout + + Partner->>+Cocoon: Retry HTTP POST (signed with partner key,
JWT body with status for sha=wxyz with uid=abcd) + Cocoon->>Partner: HTTP GET of Partner JWKS url + Partner-->>Cocoon: JWKS to verify JWT signature + Cocoon->>PubSub: Partner test execution status
for sha=wxyz with uid=abcd + Cocoon-->>-Partner: Response 200 + + PubSub->>+Cocoon: Partner test execution status
for sha=wxyz with uid=abcd + Note over Cocoon: Update Partner task status for sha=wxyz with uid=abcd + deactivate Cocoon +``` - Partner-->>Cocoon: JWKS to verify JWT signature - Cocoon->>PubSub: Partner test execution status for sha=xyzw with uid=abcd - Cocoon-->>-Partner: Response 200 - deactivate Partner - PubSub->>+Cocoon: Partner test execution status for sha=xyzw with uid=abcd - Note over Cocoon: Update Partner task status to failed for sha=xyzw with uid=abcd - deactivate Cocoon - end +If test failed, Cocon may request re-run of failed task. +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + participant Cocoon as Cocoon Server + participant Partner as Partner CI opt Cocoon calls partner webhook to re-run failed task - PubSub->>+Cocoon: Notify partner message for sha=xyzw with uid=efgh - Cocoon->>+Partner: HTTP POST (signed with cocoon key, JWT body with notification for sha=xyzw with uid=efgh) + Cocoon->>+Cocoon: Re-run for uid=efgh requested + Cocoon->>PubSub: Schedule partner notification
event for sha=wxyz with uid=efgh + PubSub->>Cocoon: Receive partner notification
event for sha=wxyz with uid=efgh + Cocoon->>+Partner: HTTP POST (signed with cocoon key,
JWT body with notification for sha=wxyz with uid=efgh) Partner->>Cocoon: HTTP GET of Cocoon JWKS url Cocoon-->>Partner: JWKS to verify JWT signature Partner-->>Cocoon: Response 200 @@ -204,59 +182,64 @@ sequenceDiagram Note over Partner: Executing custom tests opt Partner calls cocoon webhook to provide CI run status update - Partner->>+Cocoon: HTTP POST Callback (signed with partner key, JWT body with success status for sha=xyzw with uid=efgh) + Partner->>+Cocoon: HTTP POST Callback (signed with partner key,
JWT body with success status for sha=wxyz with uid=efgh) Partner-->>Cocoon: JWKS to verify JWT signature - Cocoon->>PubSub: Partner test execution status for sha=xyzw with uid=efgh + Cocoon->>PubSub: Partner test execution status
for sha=wxyz with uid=efgh Cocoon-->>-Partner: Response 200 deactivate Partner - PubSub->>+Cocoon: Partner test execution status for sha=xyzw with uid=efgh - Note over Cocoon: Update Partner task status to succeeded for sha=xyzw with uid=efgh + PubSub->>+Cocoon: Partner test execution status
for sha=wxyz with uid=efgh + Note over Cocoon: Update Partner task status to succeeded for sha=wxyz with uid=efgh deactivate Cocoon end ``` -#### 4.1.4. Re-run Failed Flow Breakdown +#### 4.1.2. Flow Breakdown -* **1. Initial Trigger & Dispatch:** GitHub notifies Cocoon of a post-submit commit (`sha=xyzw`). Cocoon creates an initial task record with a unique correlation identifier (`uid=abcd`) and publishes a notification to GCP Pub/Sub, which dispatches a signed JWT webhook to the Partner CI. -* **2. Initial Execution & Failure Callback:** The Partner CI verifies Cocoon's JWT signature via Cocoon's JWKS endpoint, runs its custom test suite for `sha=xyzw`, and reports a `failed` execution status back to Cocoon's callback endpoint using `uid=abcd`. -* **3. Failure Processing:** Cocoon validates the partner's JWT signature via the partner's JWKS endpoint, queues the status update in Pub/Sub, and marks the task state as `failed` in Firestore for `sha=xyzw` (`uid=abcd`). -* **4. Re-run Scheduling:** When a task re-run is initiated for `sha=xyzw` (either automatically by Cocoon's flake/retry policy or manually from the build dashboard), Cocoon generates a new unique correlation ID (`uid=efgh`) for the new attempt while retaining the original commit context (`sha=xyzw`), and publishes a new dispatch message to Pub/Sub. -* **5. Re-run Partner Dispatch:** The Pub/Sub subscriber sends a new signed JWT HTTP POST notification to the Partner CI webhook containing `sha=xyzw` and `uid=efgh`. Issuing a distinct `uid` ensures the partner treats the request as a new execution rather than rejecting it as a duplicate (`409 Conflict`) of `uid=abcd`. -* **6. Re-run Execution & Callback:** The Partner CI verifies the new JWT signature, re-executes the custom tests for `sha=xyzw`, and sends a signed JWT callback to Cocoon reporting the updated status (e.g., `succeeded`) associated with `uid=efgh`. -* **7. Final Status Update:** Cocoon verifies the callback signature, processes the status update through Pub/Sub, and updates the task status in Firestore to `succeeded` for `sha=xyzw` (`uid=efgh`). +* **1. Trigger:** GitHub notifies Cocoon via `/api/github/webhook-subscription` when a commit `sha=wxyz` lands on `main`. +* **2. Scheduling:** Cocoon schedules tasks on LUCI and creates records in `tasks` collection in Firestore for each configured partner. +* **3. Outbound Queueing:** For every configured partner, Cocoon creates a record in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, and publishes a message containing the `uid` and commit context (`sha`, `pr`, `slug`, etc.) to an internal GCP Pub/Sub topic for each partner. +* **4. Partner Dispatch:** A Pub/Sub subscriber fetches partner configurations, generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, and issues an HTTP POST webhook request to the partner's configured URL. Cocoon remains stateless and retries the dispatch via Pub/Sub if it timed out waiting for status response, receives 5xx (Server error), or 408 (Request Timeout) response. +* **5. Terminal unsuccessful response:** If a partner webhook responds with client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. +* **6. Execution & Callback:** The partner CI executes tests on proprietary hardware and makes an API callback to Cocoon with the execution status (`in_progress`, `cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), message, and log URLs. Cocoon reads the Key ID from the JWT header and does a lookup for the corresponding cached public key; if the public key is not found it attempts to re-read partner's JWKS and update cache; if the key is not found it responds with 401. If Cocoon successfully validates the payload and sends the partner status to Pub/Sub, it will respond with status code 200. Failure responses indicating that the partner CI system should retry sending the CI execution status are: 5xx and 408. In case of a JWT signature validation failure, Cocoon will respond with 401. If there is a validation error, it responds with 400; if the `uid` is not found, it responds with 404; if already processed, it responds with 409; if body size exceeds 8MB it responds with 413. +* **7. Processing response:** Once Cocoon receives the partner status from Pub/Sub, it updates the partner task state in Firestore. +* **8. Manual On-Demand Status Reconciliation:** Partners are required to host a status endpoint (e.g., `GET /status?uid=abcd`). In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. Automated polling is strictly rejected to avoid DoS risks and operational overhead. +* **9. Re-run Scheduling:** When a task re-run is initiated for commit `sha=wxyz` (either automatically by Cocoon's flake/retry policy or manually from the build dashboard), Cocoon generates a new unique correlation ID (`uid=efgh`) for the new attempt while retaining the original commit `sha=wxyz`, and publishes a new dispatch message to Pub/Sub. +* **10. Re-run Partner Dispatch:** The Pub/Sub subscriber sends a new signed JWT HTTP POST notification to the Partner CI webhook containing `sha=wxyz` and `uid=efgh`. Issuing a distinct `uid` ensures the partner treats the request as a new execution rather than rejecting it as a duplicate (`409 Conflict`) of `uid=abcd`. +* **11. Re-run Execution & Callback:** The Partner CI verifies the new JWT signature, re-executes the custom tests for `sha=wxyz`, and sends a signed JWT callback to Cocoon reporting the updated status (e.g., `succeeded`) associated with `uid=efgh`. +* **12. Final Status Update:** Cocoon verifies the callback signature, processes the status update through Pub/Sub, and updates the task status in Firestore to `succeeded` for `sha=wxyz` (`uid=efgh`). -#### 4.1.5. Security & Authentication +#### 4.1.3. Security & Authentication This section details the decentralized security model used to authorize server-to-server bidirectional webhook communication without a centralized authorization server. -##### 4.1.5.1. Asymmetric Cryptography & Key Management +##### 4.1.3.1. Asymmetric Cryptography & Key Management Both the Cocoon server and the Partner CI server independently generate their own asymmetric key pairs (using [Ed25519](https://www.rfc-editor.org/info/rfc8410/)). Each system stores its private key in a secure key management system and shares its corresponding public key with the other party. By leveraging decentralized asymmetric cryptography, this approach intentionally avoids the need for a central Authorization Server or Identity Provider, eliminating a potential single point of failure and reducing infrastructure complexity. -##### 4.1.5.2. Key Distribution Strategy: Public Key Endpoints +##### 4.1.3.2. Key Distribution Strategy: Public Key Endpoints For this authentication model to work, both parties must reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to fetch or refresh this key periodically. This is highly secure because it allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. Example of JWKS: -##### 4.1.5.3. Serialized JWT HTTP Body +##### 4.1.3.3. Serialized JWT HTTP Body When either system sends an HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender constructs a POST request as follows: * **Header**: `Content-Type: application/jwt` * **Body**: A signed JSON Web Token (JWT) -##### 4.1.5.4. JWT Payload Structure +##### 4.1.3.4. JWT Payload Structure The JWT consists of a base64 encoded header, payload, and signature separated by dots. For example: `eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` -##### 4.1.5.5. JWT Header +##### 4.1.3.5. JWT Header ```json { @@ -272,7 +255,7 @@ Where: * **typ**: The type of the token, which is set to `JWT`. * **kid**: Key ID that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. -##### 4.1.5.6. JWT Payload +##### 4.1.3.6. JWT Payload ```jsonc { @@ -285,17 +268,16 @@ Where: "data": { "slug": "flutter/flutter", "pr": 12345, - "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b", - "task_name": "some_awesome_test" + "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b" } } ``` -##### 4.1.5.7. JWT Signature +##### 4.1.3.7. JWT Signature [Ed25519 signature](https://www.rfc-editor.org/info/rfc8037/#appendix-A.4) as defined in [RFC 8037](https://www.rfc-editor.org/info/rfc8037). -##### 4.1.5.8. Validating Incoming Requests +##### 4.1.3.8. Validating Incoming Requests Webhook listeners validate incoming requests in accordance with standard JWT/JWS/JWKS specifications and security best practices, specifically: * [RFC 7519 Section 7.2](https://www.rfc-editor.org/info/rfc7519/#section-7.2) for JWT validation; @@ -379,27 +361,27 @@ sequenceDiagram GitHub->>Cocoon: Postsubmit Event Note over Cocoon: Creates Partner task - Cocoon->>PubSub: Notify partner message for uid=abcd - PubSub->>Cocoon: Notify partner message for uid=abcd - Cocoon->>Partner: HTTP POST (signed with cocoon key, JWT body with notification for uid=abcd) + Cocoon->>PubSub: Schedule partner notification
event for sha=wxyz with uid=abcd + PubSub->>Cocoon: Receive partner notification
event for sha=wxyz with uid=abcd + Cocoon->>Partner: HTTP POST (signed with cocoon key,
JWT body with notification for uid=abcd) Partner-->>Cocoon: Response 200 Cocoon-->>PubSub: ack - Cocoon->>PubSub: Pull partner CI status for uid=abcd + Cocoon->>PubSub: Schedule pull partner CI status event
for sha=wxyz with uid=abcd loop Polling by using PubSub retry - PubSub->>Cocoon: Pull partner CI status for uid=abcd + PubSub->>Cocoon: Receive pull partner CI status event
for sha=wxyz with uid=abcd Note over Partner: Executes custom tests Cocoon->>Partner: GET /status?uid=abcd Partner-->>Cocoon: Response 202, Status: RUNNING Cocoon-->>PubSub: nack end - PubSub->>Cocoon: Pull partner CI status for uid=abcd + PubSub->>Cocoon: Receive pull partner CI status event
for sha=wxyz with uid=abcd Cocoon->>Partner: GET /status?uid=abcd Partner-->>Cocoon: Response 200, Status: COMPLETED Cocoon-->>PubSub: ack - Cocoon->>PubSub: Partner test execution status for uid=abcd - PubSub->>Cocoon: Partner test execution status for uid=abcd + Cocoon->>PubSub: Schedule Partner test execution status
for sha=wxyz with uid=abcd + PubSub->>Cocoon: Receive Partner test execution status
for sha=wxyz with uid=abcd Note over Cocoon: Update Partner task status Cocoon-->>PubSub: ack ``` @@ -445,7 +427,7 @@ For partner configuration, add a `partner_config` collection with Document ID eq ```go slug string // github repo (e.g. flutter/flutter or flutter/packages) to trigger on -task_name string // Name of post submit task +task_name string // Name of post submit task to display on dashboard partner string // 3p partner name webhook_url string // Url to call to request CI status_url string // Manual status request url @@ -462,7 +444,7 @@ Add a `partner_ci` collection with Auto-ID Document ID that will be equal to the commit_sha string // Head sha at a moment of check-run creation ie commit sha slug string // repository_owner/repository_name (e.g. flutter/flutter) partner string // 3p partner name -task_name string // Name of post submit task +task_name string // Name of post submit task to display on dashboard attempt_number int // Number of re-run attempt processed bool // Whether status response from partner was already processed ``` @@ -484,7 +466,7 @@ To guarantee that Cocoon can properly process status updates once the status web ## 6. Open Questions * What other fields should we provide to partners webhook to request CI run? -* Can tasks PK `__` be used as `uid`? In that case we would not need a `partner_ci` collection. +* Can `tasks` collection PK `__` be used as `uid`? In that case we would not need a `partner_ci` collection. ## 7. Testing Plan @@ -528,7 +510,6 @@ See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc751 "slug": "flutter/flutter", "pr": 12345, "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b", - "task_name": "some_awesome_partners_test", "tags": { "attempt": "1", "key2": "value2", @@ -549,7 +530,6 @@ Where: * **slug**: The target repository identifier (e.g., `flutter/flutter`). * **pr**: The Pull Request number associated with the event. * **sha**: The target git commit SHA hash. - * **task_name**: Name of task to be displayed on dashboard, to potentially support several CI types or platforms per partner. For only one task per partner can be ignored. * **tags**: Optional key-value string pairs, for future usage. Should be returned as is in the status response. ##### 9.1.1.3. HTTP Response From 6c54bc4102e3023b755c5f1ae33d15de37ea3c6a Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 12:24:27 -0700 Subject: [PATCH 22/35] fixed comments --- rfc/610.0000-flutter-pluggable-ci.md | 50 +++++++++++----------------- 1 file changed, 19 insertions(+), 31 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index f4a30db..9209765 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -111,7 +111,7 @@ sequenceDiagram PubSub->>+Cocoon: Receive partner notification
event for sha=wxyz with uid=abcd Cocoon->>+Partner: HTTP POST (signed with cocoon key,
JWT body with notification for sha=wxyz with uid=abcd) - Partner->>Cocoon: HTTP GET of Cocoon JWKS url + Partner->>Cocoon: HTTP GET to Cocoon JWKS url if missed in cache Cocoon-->>Partner: JWKS to verify JWT signature Note over Partner: Internal Failure during processing the request Partner-->>-Cocoon: Response 500 @@ -119,7 +119,7 @@ sequenceDiagram PubSub->>+Cocoon: Retry partner notification
event for sha=wxyz with uid=abcd Cocoon->>+Partner: Retry HTTP POST (signed with cocoon key,
JWT body with notification for sha=wxyz with uid=abcd) - Partner->>Cocoon: HTTP GET of Cocoon JWKS url + Partner->>Cocoon: HTTP GET to Cocoon JWKS url if missed in cache Cocoon-->>Partner: JWKS to verify JWT signature Partner-->>Cocoon: Response 200 Cocoon-->>-PubSub: ack @@ -151,7 +151,7 @@ sequenceDiagram Note over Partner: Response timeout Partner->>+Cocoon: Retry HTTP POST (signed with partner key,
JWT body with status for sha=wxyz with uid=abcd) - Cocoon->>Partner: HTTP GET of Partner JWKS url + Cocoon->>Partner: HTTP GET to Partner JWKS url if missed in cache Partner-->>Cocoon: JWKS to verify JWT signature Cocoon->>PubSub: Partner test execution status
for sha=wxyz with uid=abcd Cocoon-->>-Partner: Response 200 @@ -173,7 +173,7 @@ sequenceDiagram Cocoon->>PubSub: Schedule partner notification
event for sha=wxyz with uid=efgh PubSub->>Cocoon: Receive partner notification
event for sha=wxyz with uid=efgh Cocoon->>+Partner: HTTP POST (signed with cocoon key,
JWT body with notification for sha=wxyz with uid=efgh) - Partner->>Cocoon: HTTP GET of Cocoon JWKS url + Partner->>Cocoon: HTTP GET to Cocoon JWKS url if missed in cache Cocoon-->>Partner: JWKS to verify JWT signature Partner-->>Cocoon: Response 200 Cocoon-->>-PubSub: ack @@ -182,26 +182,26 @@ sequenceDiagram Note over Partner: Executing custom tests opt Partner calls cocoon webhook to provide CI run status update - Partner->>+Cocoon: HTTP POST Callback (signed with partner key,
JWT body with success status for sha=wxyz with uid=efgh) + Partner->>+Cocoon: HTTP POST Callback (signed with partner key,
JWT body with status for sha=wxyz with uid=efgh) Partner-->>Cocoon: JWKS to verify JWT signature Cocoon->>PubSub: Partner test execution status
for sha=wxyz with uid=efgh Cocoon-->>-Partner: Response 200 deactivate Partner PubSub->>+Cocoon: Partner test execution status
for sha=wxyz with uid=efgh - Note over Cocoon: Update Partner task status to succeeded for sha=wxyz with uid=efgh + Note over Cocoon: Update Partner task status for sha=wxyz with uid=efgh deactivate Cocoon end ``` #### 4.1.2. Flow Breakdown -* **1. Trigger:** GitHub notifies Cocoon via `/api/github/webhook-subscription` when a commit `sha=wxyz` lands on `main`. +* **1. Trigger:** GitHub notifies Cocoon via `/api/github/webhook-subscription` when a PR with `sha=wxyz` merged. * **2. Scheduling:** Cocoon schedules tasks on LUCI and creates records in `tasks` collection in Firestore for each configured partner. * **3. Outbound Queueing:** For every configured partner, Cocoon creates a record in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, and publishes a message containing the `uid` and commit context (`sha`, `pr`, `slug`, etc.) to an internal GCP Pub/Sub topic for each partner. * **4. Partner Dispatch:** A Pub/Sub subscriber fetches partner configurations, generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, and issues an HTTP POST webhook request to the partner's configured URL. Cocoon remains stateless and retries the dispatch via Pub/Sub if it timed out waiting for status response, receives 5xx (Server error), or 408 (Request Timeout) response. * **5. Terminal unsuccessful response:** If a partner webhook responds with client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. -* **6. Execution & Callback:** The partner CI executes tests on proprietary hardware and makes an API callback to Cocoon with the execution status (`in_progress`, `cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), message, and log URLs. Cocoon reads the Key ID from the JWT header and does a lookup for the corresponding cached public key; if the public key is not found it attempts to re-read partner's JWKS and update cache; if the key is not found it responds with 401. If Cocoon successfully validates the payload and sends the partner status to Pub/Sub, it will respond with status code 200. Failure responses indicating that the partner CI system should retry sending the CI execution status are: 5xx and 408. In case of a JWT signature validation failure, Cocoon will respond with 401. If there is a validation error, it responds with 400; if the `uid` is not found, it responds with 404; if already processed, it responds with 409; if body size exceeds 8MB it responds with 413. +* **6. Execution & Callback:** The partner CI executes tests on proprietary hardware and makes an API callback to Cocoon with the execution status (`cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), message, and log URLs. Cocoon reads the Key ID from the JWT header and does a lookup for the corresponding cached public key; if the public key is not found it attempts to re-read partner's JWKS and update cache; if the key is not found it responds with 401. If Cocoon successfully validates the payload and sends the partner status to Pub/Sub, it will respond with status code 200. Failure responses indicating that the partner CI system should retry sending the CI execution status are: 5xx and 408. In case of a JWT signature validation failure, Cocoon will respond with 401. If there is a validation error, it responds with 400; if the `uid` is not found, it responds with 404; if already processed, it responds with 409; if body size exceeds 8MB it responds with 413. * **7. Processing response:** Once Cocoon receives the partner status from Pub/Sub, it updates the partner task state in Firestore. * **8. Manual On-Demand Status Reconciliation:** Partners are required to host a status endpoint (e.g., `GET /status?uid=abcd`). In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. Automated polling is strictly rejected to avoid DoS risks and operational overhead. * **9. Re-run Scheduling:** When a task re-run is initiated for commit `sha=wxyz` (either automatically by Cocoon's flake/retry policy or manually from the build dashboard), Cocoon generates a new unique correlation ID (`uid=efgh`) for the new attempt while retaining the original commit `sha=wxyz`, and publishes a new dispatch message to Pub/Sub. @@ -245,7 +245,7 @@ For example: { "alg": "EdDSA", "typ": "JWT", - "kid": 12345 + "kid": "12345" } ``` @@ -285,7 +285,7 @@ Webhook listeners validate incoming requests in accordance with standard JWT/JWS * [RFC 7517 Section 4.5](https://www.rfc-editor.org/info/rfc7517/#section-4.5) & [Section 5](https://www.rfc-editor.org/info/rfc7517/#section-5) for `kid` lookup via cached JWKS; * [RFC 8725 Section 3](https://www.rfc-editor.org/info/rfc8725/#section-3) for algorithm pinning and `iss`/`aud`/`exp` claim checks. -In addition, Cocoon's callback endpoint validates that `data.uid` exists and has not been processed yet. +In addition, Cocoon's callback endpoint validates that `data.uid` exists, has not been processed yet and belongs to a partner that is listed in `iss` claim. #### 4.1.6. Pros & Cons @@ -509,12 +509,7 @@ See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc751 "uid": "0sAtLhwbmGz2Y0WBDzPL", "slug": "flutter/flutter", "pr": 12345, - "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b", - "tags": { - "attempt": "1", - "key2": "value2", - "key3": "value3" - } + "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b" } } ``` @@ -530,7 +525,6 @@ Where: * **slug**: The target repository identifier (e.g., `flutter/flutter`). * **pr**: The Pull Request number associated with the event. * **sha**: The target git commit SHA hash. - * **tags**: Optional key-value string pairs, for future usage. Should be returned as is in the status response. ##### 9.1.1.3. HTTP Response @@ -565,38 +559,33 @@ Content-Type: application/jwt "exp": 1774963800, "data": { "uid": "0sAtLhwbmGz2Y0WBDzPL", - "tags": { - "attempt": "1", - "key2": "value2", - "key3": "value3" - }, "status": "failed", "message": "Some optional summary if needed", "details": [ { "name": "test_1", "status": "failed", - "log_url": "https://parnter.com/test_1/log" + "url": "https://parnter.com/test_1/log" }, { "name": "test_2", - "status": "success", - "log_url": "https://parnter-ci.com/test_2/log" + "status": "succeeded", + "url": "https://parnter-ci.com/test_2/log" }, { "name": "test_3", "status": "skipped", - "log_url": "https://parnter-ci.com/test_3/log" + "url": "https://parnter-ci.com/test_3/log" }, { "name": "test_4", "status": "failed", - "log_url": "https://parnter-ci.com/test_4/log" + "url": "https://parnter-ci.com/test_4/log" }, { "name": "test_5", "status": "cancelled", - "log_url": "https://parnter-ci.com/test_5/log" + "url": "https://parnter-ci.com/test_5/log" } ] } @@ -611,7 +600,6 @@ Where: * **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. * **data**: Object containing custom payload claims and event details: * **uid**: The unique correlation identifier to match which CI execution request this status response belongs to. - * **tags**: Optional key-value string pairs, for internal usage by Flutter CI. Should be returned as is if received in request. * **status**: The terminal status of CI execution (cannot be `in_progress`): * `cancelled` - Task is cancelled by a Partner CI (failure state). * `infra_failure` - Infrastructure failure during tests execution (failure state). @@ -629,7 +617,7 @@ Where: * `succeeded` - Test passed (success state). * `neutral` - Test failed but considered flaky (success state). * `skipped` - Test skipped (success state). - * **log_url**: The publicly accessible task execution log URL. + * **url**: The publicly accessible task execution details or logs URL. ##### 9.1.2.3. HTTP Response @@ -669,7 +657,7 @@ Content-Type: application/json }, { "name": "test_2", - "status": "success", + "status": "succeeded", "log_url": "https://parnter-ci.com/test_2/log" }, { From 67b5393536d7c6a22a16884b6853c3c535b57723 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 12:27:33 -0700 Subject: [PATCH 23/35] add some open questions --- rfc/610.0000-flutter-pluggable-ci.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 9209765..30e0617 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -467,6 +467,8 @@ To guarantee that Cocoon can properly process status updates once the status web * What other fields should we provide to partners webhook to request CI run? * Can `tasks` collection PK `__` be used as `uid`? In that case we would not need a `partner_ci` collection. +* Should we allow partner to call us back with "in_progress" status? It can lead to race conditions but might provide better visibility of CI progress. +* How long should we keep webhook data in firestore? It can grow large and we should probably remove old data. ## 7. Testing Plan From ae86db70aa342183090c769ad623b71f213a8df1 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 12:43:39 -0700 Subject: [PATCH 24/35] markdown lint --- rfc/610.0000-flutter-pluggable-ci.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 30e0617..61aa841 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -92,6 +92,7 @@ In this architecture, Cocoon uses standard HTTP webhooks to push event notificat #### 4.1.1. End-to-End Workflow Diagram Internal processing of post submit event. + ```mermaid sequenceDiagram participant PubSub@{"type":"queue"} @@ -102,7 +103,9 @@ sequenceDiagram Note over Cocoon: Creates task for sha=wxyz with uid=abcd Cocoon->>-PubSub: Schedule partner notification
event for sha=wxyz with uid=abcd ``` + Cocoon calls partner webhook to notify about post submit to run CI, retrying on failures. + ```mermaid sequenceDiagram participant PubSub@{"type":"queue"} @@ -127,6 +130,7 @@ sequenceDiagram ``` Optionally, if supported by Partner, Cocoon can poll for status updates. + ```mermaid sequenceDiagram participant Cocoon as Cocoon Server @@ -140,6 +144,7 @@ sequenceDiagram ``` Partner calls cocoon webhook to provide CI run status update, retrying on failures. + ```mermaid sequenceDiagram participant PubSub@{"type":"queue"} @@ -162,6 +167,7 @@ sequenceDiagram ``` If test failed, Cocon may request re-run of failed task. + ```mermaid sequenceDiagram participant PubSub@{"type":"queue"} @@ -280,6 +286,7 @@ Where: ##### 4.1.3.8. Validating Incoming Requests Webhook listeners validate incoming requests in accordance with standard JWT/JWS/JWKS specifications and security best practices, specifically: + * [RFC 7519 Section 7.2](https://www.rfc-editor.org/info/rfc7519/#section-7.2) for JWT validation; * [RFC 7515 Section 5.2](https://www.rfc-editor.org/info/rfc7515/#section-5.2) for signature verification; * [RFC 7517 Section 4.5](https://www.rfc-editor.org/info/rfc7517/#section-4.5) & [Section 5](https://www.rfc-editor.org/info/rfc7517/#section-5) for `kid` lookup via cached JWKS; From 2cb1a669b55f0445735b2ab9dbae3205453f4aef Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 12:45:25 -0700 Subject: [PATCH 25/35] markdown lint --- rfc/610.0000-flutter-pluggable-ci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 61aa841..1de66a8 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -287,7 +287,7 @@ Where: Webhook listeners validate incoming requests in accordance with standard JWT/JWS/JWKS specifications and security best practices, specifically: -* [RFC 7519 Section 7.2](https://www.rfc-editor.org/info/rfc7519/#section-7.2) for JWT validation; +* [RFC 7519 Section 7.2](https://www.rfc-editor.org/info/rfc7519/#section-7.2) for JWT validation; * [RFC 7515 Section 5.2](https://www.rfc-editor.org/info/rfc7515/#section-5.2) for signature verification; * [RFC 7517 Section 4.5](https://www.rfc-editor.org/info/rfc7517/#section-4.5) & [Section 5](https://www.rfc-editor.org/info/rfc7517/#section-5) for `kid` lookup via cached JWKS; * [RFC 8725 Section 3](https://www.rfc-editor.org/info/rfc8725/#section-3) for algorithm pinning and `iss`/`aud`/`exp` claim checks. From 70353a3621338f2e545971a16cd7762d72bd4207 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 14:08:46 -0700 Subject: [PATCH 26/35] added optional `in_progress` notification flow support. --- rfc/610.0000-flutter-pluggable-ci.md | 116 +++++++++++++++++++-------- 1 file changed, 81 insertions(+), 35 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 1de66a8..c5f1bd9 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -91,20 +91,35 @@ In this architecture, Cocoon uses standard HTTP webhooks to push event notificat #### 4.1.1. End-to-End Workflow Diagram -Internal processing of post submit event. +##### 4.1.1.1. Internal processing of post submit event + +1. GitHub notifies Cocoon via `/api/github/webhook-subscription` when a PR with `sha=wxyz` merged. +2. Cocoon schedules tasks on LUCI and creates records in `tasks` collection in Firestore for each configured partner. +3. For every configured partner, Cocoon: + * Creates a record in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, + * Publishes a message containing the `uid` and commit context (`sha`, `pr`, `slug`, etc.) to an internal GCP Pub/Sub topic. ```mermaid sequenceDiagram participant PubSub@{"type":"queue"} actor GitHub as GitHub Repo participant Cocoon as Cocoon Server + participant Firestore@{"type":"database"} destroy GitHub GitHub->>+Cocoon: Postsubmit Event - Note over Cocoon: Creates task for sha=wxyz with uid=abcd + Cocoon->>Firestore: Creates record in `tasks` collection
for sha=wxyz with uid=abcd,
partner name, attempt number, etc. Cocoon->>-PubSub: Schedule partner notification
event for sha=wxyz with uid=abcd ``` -Cocoon calls partner webhook to notify about post submit to run CI, retrying on failures. +##### 4.1.1.2. Cocoon calls partner webhook to notify about post submit to run CI, retrying on failures + +1. A Pub/Sub subscriber: + * receives partner notification event, + * fetches partner configurations, + * generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, + * issues an HTTP POST webhook request to the partner's configured URL. +2. If webhook invocation times out or partner server returns 5xx (Server error) or 408 (Request Timeout) response, Cocoon will not acknowledge the Pub/Sub message and will retry the dispatch. +3. If a partner webhook responds with success code 200 or client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. In both cases, Cocoon will acknowledge the message and will not retry the dispatch. ```mermaid sequenceDiagram @@ -129,7 +144,39 @@ sequenceDiagram Note over Partner: Start executing custom tests ``` -Optionally, if supported by Partner, Cocoon can poll for status updates. +##### 4.1.1.3. Optionally, Partner may notify cocoon it started test execution and provide test details + +Optionally, when the partner CI begins test execution for `sha=wxyz` (`uid=abcd`), + +1. It may send a signed JWT callback to Cocoon with `in_progress` status along with scheduled test names and links to execution details. +2. Cocoon verifies the JWT signature (fetching the partner's JWKS on cache miss), publishes the status to Pub/Sub, responds with `200`, and updates the task status in Firestore to `in_progress`. +3. That will lead to showing test names and links to execution details in the Cocoon build dashboard. + +```mermaid +sequenceDiagram + participant PubSub@{"type":"queue"} + participant Cocoon as Cocoon Server + participant Partner as Partner CI + + Partner->>+Cocoon: HTTP POST Callback (signed with partner key, JWT body
with `in_progress` status for sha=wxyz with uid=abcd) + Cocoon->>Partner: HTTP GET to Partner JWKS url if missed in cache + Partner-->>Cocoon: JWKS to verify JWT signature + Cocoon->>PubSub: Partner test execution status
for sha=wxyz with uid=abcd + Cocoon-->>-Partner: Response 200 + + PubSub->>+Cocoon: Partner test execution status
for sha=wxyz with uid=abcd + Note over Cocoon: Update Partner task status to `in_progress`. If provided,
display scheduled test names and links to execution details. + deactivate Cocoon +``` + +##### 4.1.1.4. Optionally, if supported by Partner, Cocoon can poll for status updates + +Partners may host a status endpoint (e.g., `GET /status?uid=abcd`). + +1. In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard while waiting for execution results. +2. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. + +**Note:** Automated polling is strictly rejected to avoid DoS risks and operational overhead. ```mermaid sequenceDiagram @@ -143,7 +190,14 @@ sequenceDiagram Note over Partner: Continue executing custom tests ``` -Partner calls cocoon webhook to provide CI run status update, retrying on failures. +##### 4.1.1.5. Partner calls cocoon webhook to provide CI run status update, retrying on failures + +1. Once the partner CI finishes executing tests on proprietary hardware for `sha=wxyz` (`uid=abcd`), it makes an HTTP POST callback to Cocoon signed with the partner's key containing the terminal execution status (`cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), summary message, and test URLs. + * If the callback request experiences a response timeout or receives a retryable failure response (`5xx` or `408 Request Timeout`), the partner CI retries the HTTP POST callback for `uid=abcd`. + * Upon receiving the callback, Cocoon reads the Key ID (`kid`) from the JWT header and looks up the corresponding cached public key. If the public key is not found in cache, it fetches the partner's JWKS to update the cache, or responds with `401` if the key is not found or JWT signature validation fails. + * If Cocoon successfully validates the payload and publishes the partner test execution status to Pub/Sub, it responds with status code `200`. + * If Cocoon fails to process the request, it responds with non-retryable error status code, such as `400` for a validation error, `404` if the `uid` is not found, `409` if already processed, or `413` if the body size exceeds 8MB. +2. Once Cocoon receives the partner test execution status for `sha=wxyz` (`uid=abcd`) from Pub/Sub, it updates the partner task state in Firestore. ```mermaid sequenceDiagram @@ -166,7 +220,12 @@ sequenceDiagram deactivate Cocoon ``` -If test failed, Cocon may request re-run of failed task. +##### 4.1.1.6. If test failed, Cocon may request re-run of failed task + +1. When a task re-run is initiated for commit `sha=wxyz` (either automatically by Cocoon's flake/retry policy or manually from the build dashboard), Cocoon generates a new unique correlation ID (`uid=efgh`) for the new attempt while retaining the original commit `sha=wxyz`, and publishes a new dispatch message to Pub/Sub. +2. The Pub/Sub subscriber sends a new signed JWT HTTP POST notification to the Partner CI webhook containing `sha=wxyz` and `uid=efgh`. Issuing a distinct `uid` ensures the partner treats the request as a new execution rather than rejecting it as a duplicate (`409 Conflict`) of `uid=abcd`. +3. The Partner CI verifies the new JWT signature, re-executes the custom tests for `sha=wxyz`, and sends a signed JWT callback to Cocoon reporting the updated status (e.g., `succeeded`) associated with `uid=efgh`. +4. Cocoon verifies the callback signature, processes the status update through Pub/Sub, and updates the task status in Firestore to `succeeded` for `sha=wxyz` (`uid=efgh`). ```mermaid sequenceDiagram @@ -200,52 +259,37 @@ sequenceDiagram end ``` -#### 4.1.2. Flow Breakdown - -* **1. Trigger:** GitHub notifies Cocoon via `/api/github/webhook-subscription` when a PR with `sha=wxyz` merged. -* **2. Scheduling:** Cocoon schedules tasks on LUCI and creates records in `tasks` collection in Firestore for each configured partner. -* **3. Outbound Queueing:** For every configured partner, Cocoon creates a record in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, and publishes a message containing the `uid` and commit context (`sha`, `pr`, `slug`, etc.) to an internal GCP Pub/Sub topic for each partner. -* **4. Partner Dispatch:** A Pub/Sub subscriber fetches partner configurations, generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, and issues an HTTP POST webhook request to the partner's configured URL. Cocoon remains stateless and retries the dispatch via Pub/Sub if it timed out waiting for status response, receives 5xx (Server error), or 408 (Request Timeout) response. -* **5. Terminal unsuccessful response:** If a partner webhook responds with client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. -* **6. Execution & Callback:** The partner CI executes tests on proprietary hardware and makes an API callback to Cocoon with the execution status (`cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), message, and log URLs. Cocoon reads the Key ID from the JWT header and does a lookup for the corresponding cached public key; if the public key is not found it attempts to re-read partner's JWKS and update cache; if the key is not found it responds with 401. If Cocoon successfully validates the payload and sends the partner status to Pub/Sub, it will respond with status code 200. Failure responses indicating that the partner CI system should retry sending the CI execution status are: 5xx and 408. In case of a JWT signature validation failure, Cocoon will respond with 401. If there is a validation error, it responds with 400; if the `uid` is not found, it responds with 404; if already processed, it responds with 409; if body size exceeds 8MB it responds with 413. -* **7. Processing response:** Once Cocoon receives the partner status from Pub/Sub, it updates the partner task state in Firestore. -* **8. Manual On-Demand Status Reconciliation:** Partners are required to host a status endpoint (e.g., `GET /status?uid=abcd`). In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. Automated polling is strictly rejected to avoid DoS risks and operational overhead. -* **9. Re-run Scheduling:** When a task re-run is initiated for commit `sha=wxyz` (either automatically by Cocoon's flake/retry policy or manually from the build dashboard), Cocoon generates a new unique correlation ID (`uid=efgh`) for the new attempt while retaining the original commit `sha=wxyz`, and publishes a new dispatch message to Pub/Sub. -* **10. Re-run Partner Dispatch:** The Pub/Sub subscriber sends a new signed JWT HTTP POST notification to the Partner CI webhook containing `sha=wxyz` and `uid=efgh`. Issuing a distinct `uid` ensures the partner treats the request as a new execution rather than rejecting it as a duplicate (`409 Conflict`) of `uid=abcd`. -* **11. Re-run Execution & Callback:** The Partner CI verifies the new JWT signature, re-executes the custom tests for `sha=wxyz`, and sends a signed JWT callback to Cocoon reporting the updated status (e.g., `succeeded`) associated with `uid=efgh`. -* **12. Final Status Update:** Cocoon verifies the callback signature, processes the status update through Pub/Sub, and updates the task status in Firestore to `succeeded` for `sha=wxyz` (`uid=efgh`). - -#### 4.1.3. Security & Authentication +#### 4.1.2. Security & Authentication This section details the decentralized security model used to authorize server-to-server bidirectional webhook communication without a centralized authorization server. -##### 4.1.3.1. Asymmetric Cryptography & Key Management +##### 4.1.2.1. Asymmetric Cryptography & Key Management Both the Cocoon server and the Partner CI server independently generate their own asymmetric key pairs (using [Ed25519](https://www.rfc-editor.org/info/rfc8410/)). Each system stores its private key in a secure key management system and shares its corresponding public key with the other party. By leveraging decentralized asymmetric cryptography, this approach intentionally avoids the need for a central Authorization Server or Identity Provider, eliminating a potential single point of failure and reducing infrastructure complexity. -##### 4.1.3.2. Key Distribution Strategy: Public Key Endpoints +##### 4.1.2.2. Key Distribution Strategy: Public Key Endpoints For this authentication model to work, both parties must reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to fetch or refresh this key periodically. This is highly secure because it allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. Example of JWKS: -##### 4.1.3.3. Serialized JWT HTTP Body +##### 4.1.2.3. Serialized JWT HTTP Body When either system sends an HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender constructs a POST request as follows: * **Header**: `Content-Type: application/jwt` * **Body**: A signed JSON Web Token (JWT) -##### 4.1.3.4. JWT Payload Structure +##### 4.1.2.4. JWT Payload Structure The JWT consists of a base64 encoded header, payload, and signature separated by dots. For example: `eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` -##### 4.1.3.5. JWT Header +##### 4.1.2.5. JWT Header ```json { @@ -261,7 +305,7 @@ Where: * **typ**: The type of the token, which is set to `JWT`. * **kid**: Key ID that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. -##### 4.1.3.6. JWT Payload +##### 4.1.2.6. JWT Payload ```jsonc { @@ -279,11 +323,11 @@ Where: } ``` -##### 4.1.3.7. JWT Signature +##### 4.1.2.7. JWT Signature [Ed25519 signature](https://www.rfc-editor.org/info/rfc8037/#appendix-A.4) as defined in [RFC 8037](https://www.rfc-editor.org/info/rfc8037). -##### 4.1.3.8. Validating Incoming Requests +##### 4.1.2.8. Validating Incoming Requests Webhook listeners validate incoming requests in accordance with standard JWT/JWS/JWKS specifications and security best practices, specifically: @@ -294,7 +338,7 @@ Webhook listeners validate incoming requests in accordance with standard JWT/JWS In addition, Cocoon's callback endpoint validates that `data.uid` exists, has not been processed yet and belongs to a partner that is listed in `iss` claim. -#### 4.1.6. Pros & Cons +#### 4.1.3. Pros & Cons * **Pros:** * Industry-standard webhook architecture that is easy for partners to consume. @@ -609,17 +653,19 @@ Where: * **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. * **data**: Object containing custom payload claims and event details: * **uid**: The unique correlation identifier to match which CI execution request this status response belongs to. - * **status**: The terminal status of CI execution (cannot be `in_progress`): + * **status**: + * `in_progress` - Task is in progress. Used only for optional `in_progress` status notification. Can not be in terminal status notification. * `cancelled` - Task is cancelled by a Partner CI (failure state). * `infra_failure` - Infrastructure failure during tests execution (failure state). * `failed` - Some or all of the tests are failed (failure state). * `succeeded` - All tests passed (success state). * `neutral` - Only some flaky tests failed (success state). * `skipped` - Some or all tests are skipped (success state). - * **message**: Markdown formatted message with a summary of CI run. - * **details**: Granular test execution details: + * **message**: Optional Markdown formatted message with a summary of CI run. Expected to be present when status is `infra_failure` or `failed`. + * **details**: Granular test execution details. * **name**: Test name. - * **status**: Terminal test status (cannot be `in_progress`): + * **status**: + * `in_progress` - Test is in progress. Used only for optional `in_progress` status notification. Can not be in terminal status notification. * `cancelled` - Test is cancelled by a Partner CI (failure state). * `infra_failure` - Infrastructure failure during test execution (failure state). * `failed` - Test failed (failure state). From 6baaf23c63026be8e69791406f48d4ccfffe90af Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 14:10:21 -0700 Subject: [PATCH 27/35] removed open question as it already addressed --- rfc/610.0000-flutter-pluggable-ci.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index c5f1bd9..dd178e6 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -518,8 +518,6 @@ To guarantee that Cocoon can properly process status updates once the status web * What other fields should we provide to partners webhook to request CI run? * Can `tasks` collection PK `__` be used as `uid`? In that case we would not need a `partner_ci` collection. -* Should we allow partner to call us back with "in_progress" status? It can lead to race conditions but might provide better visibility of CI progress. -* How long should we keep webhook data in firestore? It can grow large and we should probably remove old data. ## 7. Testing Plan From 3c426b0d4f18628058be4bb5220723b9bdeda1da Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 14:39:45 -0700 Subject: [PATCH 28/35] removed trailing spaces --- rfc/610.0000-flutter-pluggable-ci.md | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index dd178e6..9a0a45d 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -65,7 +65,7 @@ sequenceDiagram Partner-->>Cocoon: Response 200 deactivate Cocoon Note over Partner: Executes custom tests - Partner->>Cocoon: HTTP POST Callback (signed with partner key
JWT body with status for uid=abcd) + Partner->>Cocoon: HTTP POST Callback (signed with partner key
JWT body with status for uid=abcd) activate Cocoon Cocoon->>PubSub: Partner test execution status for uid=abcd Cocoon-->>Partner: Response 200 @@ -146,10 +146,8 @@ sequenceDiagram ##### 4.1.1.3. Optionally, Partner may notify cocoon it started test execution and provide test details -Optionally, when the partner CI begins test execution for `sha=wxyz` (`uid=abcd`), - -1. It may send a signed JWT callback to Cocoon with `in_progress` status along with scheduled test names and links to execution details. -2. Cocoon verifies the JWT signature (fetching the partner's JWKS on cache miss), publishes the status to Pub/Sub, responds with `200`, and updates the task status in Firestore to `in_progress`. +1. When the partner CI begins test execution for `sha=wxyz` (`uid=abcd`), it may send a signed JWT callback to Cocoon with `in_progress` status along with scheduled test names and links to execution details. +2. Cocoon verifies the JWT signature (fetching the partner's JWKS on cache miss), publishes the status to Pub/Sub, responds with `200`, and updates the task status in Firestore to `in_progress`. 3. That will lead to showing test names and links to execution details in the Cocoon build dashboard. ```mermaid @@ -157,7 +155,7 @@ sequenceDiagram participant PubSub@{"type":"queue"} participant Cocoon as Cocoon Server participant Partner as Partner CI - + Partner->>+Cocoon: HTTP POST Callback (signed with partner key, JWT body
with `in_progress` status for sha=wxyz with uid=abcd) Cocoon->>Partner: HTTP GET to Partner JWKS url if missed in cache Partner-->>Cocoon: JWKS to verify JWT signature @@ -171,10 +169,10 @@ sequenceDiagram ##### 4.1.1.4. Optionally, if supported by Partner, Cocoon can poll for status updates -Partners may host a status endpoint (e.g., `GET /status?uid=abcd`). +Partners may host a status endpoint (e.g., `GET /status?uid=abcd`). -1. In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard while waiting for execution results. -2. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. +1. In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard while waiting for execution results. +2. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. **Note:** Automated polling is strictly rejected to avoid DoS risks and operational overhead. @@ -193,9 +191,9 @@ sequenceDiagram ##### 4.1.1.5. Partner calls cocoon webhook to provide CI run status update, retrying on failures 1. Once the partner CI finishes executing tests on proprietary hardware for `sha=wxyz` (`uid=abcd`), it makes an HTTP POST callback to Cocoon signed with the partner's key containing the terminal execution status (`cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), summary message, and test URLs. - * If the callback request experiences a response timeout or receives a retryable failure response (`5xx` or `408 Request Timeout`), the partner CI retries the HTTP POST callback for `uid=abcd`. + * If the callback request experiences a response timeout or receives a retryable failure response (`5xx` or `408 Request Timeout`), the partner CI retries the HTTP POST callback for `uid=abcd`. * Upon receiving the callback, Cocoon reads the Key ID (`kid`) from the JWT header and looks up the corresponding cached public key. If the public key is not found in cache, it fetches the partner's JWKS to update the cache, or responds with `401` if the key is not found or JWT signature validation fails. - * If Cocoon successfully validates the payload and publishes the partner test execution status to Pub/Sub, it responds with status code `200`. + * If Cocoon successfully validates the payload and publishes the partner test execution status to Pub/Sub, it responds with status code `200`. * If Cocoon fails to process the request, it responds with non-retryable error status code, such as `400` for a validation error, `404` if the `uid` is not found, `409` if already processed, or `413` if the body size exceeds 8MB. 2. Once Cocoon receives the partner test execution status for `sha=wxyz` (`uid=abcd`) from Pub/Sub, it updates the partner task state in Firestore. @@ -204,7 +202,7 @@ sequenceDiagram participant PubSub@{"type":"queue"} participant Cocoon as Cocoon Server participant Partner as Partner CI - + Note over Partner: Finishes executing custom tests Partner->>Cocoon: HTTP POST Callback (signed with partner key,
JWT body with status for sha=wxyz with uid=abcd) Note over Partner: Response timeout @@ -651,7 +649,7 @@ Where: * **exp**: Expiration timestamp (Unix epoch time) after which the token is considered invalid. * **data**: Object containing custom payload claims and event details: * **uid**: The unique correlation identifier to match which CI execution request this status response belongs to. - * **status**: + * **status**: * `in_progress` - Task is in progress. Used only for optional `in_progress` status notification. Can not be in terminal status notification. * `cancelled` - Task is cancelled by a Partner CI (failure state). * `infra_failure` - Infrastructure failure during tests execution (failure state). @@ -662,7 +660,7 @@ Where: * **message**: Optional Markdown formatted message with a summary of CI run. Expected to be present when status is `infra_failure` or `failed`. * **details**: Granular test execution details. * **name**: Test name. - * **status**: + * **status**: * `in_progress` - Test is in progress. Used only for optional `in_progress` status notification. Can not be in terminal status notification. * `cancelled` - Test is cancelled by a Partner CI (failure state). * `infra_failure` - Infrastructure failure during test execution (failure state). From 60d02416a6b5e802d326952cce61b1aa15ae69cb Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 16:09:37 -0700 Subject: [PATCH 29/35] fixed Logical & Technical Bugs --- rfc/610.0000-flutter-pluggable-ci.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 9a0a45d..3dc5a8a 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -119,7 +119,8 @@ sequenceDiagram * generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, * issues an HTTP POST webhook request to the partner's configured URL. 2. If webhook invocation times out or partner server returns 5xx (Server error) or 408 (Request Timeout) response, Cocoon will not acknowledge the Pub/Sub message and will retry the dispatch. -3. If a partner webhook responds with success code 200 or client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. In both cases, Cocoon will acknowledge the message and will not retry the dispatch. +3. If a partner webhook responds with success code 200, Cocoon will acknowledge the message. +4. If a partner webhook responds with client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. In both cases, Cocoon will acknowledge the message and will not retry the dispatch. ```mermaid sequenceDiagram @@ -147,7 +148,7 @@ sequenceDiagram ##### 4.1.1.3. Optionally, Partner may notify cocoon it started test execution and provide test details 1. When the partner CI begins test execution for `sha=wxyz` (`uid=abcd`), it may send a signed JWT callback to Cocoon with `in_progress` status along with scheduled test names and links to execution details. -2. Cocoon verifies the JWT signature (fetching the partner's JWKS on cache miss), publishes the status to Pub/Sub, responds with `200`, and updates the task status in Firestore to `in_progress`. +2. Cocoon verifies the JWT signature (fetching the partner's JWKS on cache miss), publishes the status to Pub/Sub, responds with `200`, and updates the task status in Firestore to `in_progress` (if the task has not already reached a terminal state, guarding against out-of-order Pub/Sub delivery). 3. That will lead to showing test names and links to execution details in the Cocoon build dashboard. ```mermaid @@ -194,7 +195,7 @@ sequenceDiagram * If the callback request experiences a response timeout or receives a retryable failure response (`5xx` or `408 Request Timeout`), the partner CI retries the HTTP POST callback for `uid=abcd`. * Upon receiving the callback, Cocoon reads the Key ID (`kid`) from the JWT header and looks up the corresponding cached public key. If the public key is not found in cache, it fetches the partner's JWKS to update the cache, or responds with `401` if the key is not found or JWT signature validation fails. * If Cocoon successfully validates the payload and publishes the partner test execution status to Pub/Sub, it responds with status code `200`. - * If Cocoon fails to process the request, it responds with non-retryable error status code, such as `400` for a validation error, `404` if the `uid` is not found, `409` if already processed, or `413` if the body size exceeds 8MB. + * If Cocoon fails to process the request, it responds with non-retryable error status code, such as `400` for a validation error, `404` if the `uid` is not found, `409` if a terminal status was already processed, or `413` if the body size exceeds 768KB (to stay within Firestore's 1 MiB document size limit). 2. Once Cocoon receives the partner test execution status for `sha=wxyz` (`uid=abcd`) from Pub/Sub, it updates the partner task state in Firestore. ```mermaid @@ -334,7 +335,7 @@ Webhook listeners validate incoming requests in accordance with standard JWT/JWS * [RFC 7517 Section 4.5](https://www.rfc-editor.org/info/rfc7517/#section-4.5) & [Section 5](https://www.rfc-editor.org/info/rfc7517/#section-5) for `kid` lookup via cached JWKS; * [RFC 8725 Section 3](https://www.rfc-editor.org/info/rfc8725/#section-3) for algorithm pinning and `iss`/`aud`/`exp` claim checks. -In addition, Cocoon's callback endpoint validates that `data.uid` exists, has not been processed yet and belongs to a partner that is listed in `iss` claim. +In addition, Cocoon's callback endpoint validates that `data.uid` exists, has not processed a terminal status yet, and belongs to a partner that is listed in `iss` claim. #### 4.1.3. Pros & Cons @@ -460,7 +461,7 @@ sequenceDiagram | **Partner Setup Complexity** | ⚠️ **Low to moderate.** Webhooks are easy to consume, but partners must implement retry logic for outbound webhook calls. | ❌ **High.** Requires managing cross-project GCP permissions or GCP service account keys. | ✅ **Low.** Simpler for partners to only set up webhook and status endpoints. | | **Cloud Vendor Lock-in** | ✅ **Low.** Built on standard REST and HTTP/JWT protocols. | ❌ **High.** Introduces GCP cloud vendor lock-in for third-party partners. | ✅ **Low.** Built on standard REST and HTTP/JWT protocols. | | **Security & Authentication** | ✅ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | ⚠️ **Secure.** Managed via GCP IAM and service accounts. | ✅ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | -| **Status response validation** | ✅ Can validate payload schema, unknown/duplicated `uid` and notify partner with HTTP response codes. | ⚠️ Can only validate and notify partner about payload schema violation. | ✅ Cannot notify partner of any payload schema or data validation issues. | +| **Status response validation** | ✅ Can validate payload schema, unknown/duplicated `uid` and notify partner with HTTP response codes. | ⚠️ Can only validate and notify partner about payload schema violation. | ❌ Cannot notify partner of any payload schema or data validation issues. | | **Latency / Status Reporting** | ✅ **Real-time.** Push-based callback minimizes status reporting latency. | ✅ **Real-time.** Push-based consumption minimizes latency. | ❌ **Higher latency.** Updates are delayed based on Cocoon's polling intervals. | | **Resilience & Overhead** | ✅ **Highly resilient.** Retries are backed by Pub/Sub; Status query endpoint prevents polling loops. | ✅ **Highly resilient.** Messages are automatically buffered during partner infrastructure outages. | ❌ **High overhead.** Constant automated polling creates network load and risks DoS on partner endpoints. | @@ -495,7 +496,7 @@ slug string // repository_owner/repository_name (e.g. flutter/flutter) partner string // 3p partner name task_name string // Name of post submit task to display on dashboard attempt_number int // Number of re-run attempt -processed bool // Whether status response from partner was already processed +processed bool // Whether terminal status response from partner was already processed ``` #### 5.1.3. tasks From 44a22865523dd5065242fdcc93e842231e909b75 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 16:22:08 -0700 Subject: [PATCH 30/35] Fixed Cross-Paragraph Inconsistencies --- rfc/610.0000-flutter-pluggable-ci.md | 40 +++++++++++++++------------- 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 3dc5a8a..be47e7f 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -87,7 +87,7 @@ Integration with third-party CIs requires two main points of discussion: communi ### 4.1. Primary Design: Webhook Server-to-Server Communication (Push-Callback) with Manual Status Request -In this architecture, Cocoon uses standard HTTP webhooks to push event notifications to external partner endpoints. External partners process the event asynchronously on their custom infrastructure and send execution status back via Cocoon's callback REST API. Additionally, partners must host an endpoint capable of responding to on-demand status queries from Cocoon to enable manual status checks from the build dashboard, eliminating automated polling loops. +In this architecture, Cocoon uses standard HTTP webhooks to push event notifications to external partner endpoints. External partners process the event asynchronously on their custom infrastructure and send execution status back via Cocoon's callback REST API. Additionally, partners may optionally host an endpoint capable of responding to on-demand status queries from Cocoon to enable manual status checks from the build dashboard, eliminating automated polling loops. #### 4.1.1. End-to-End Workflow Diagram @@ -96,7 +96,7 @@ In this architecture, Cocoon uses standard HTTP webhooks to push event notificat 1. GitHub notifies Cocoon via `/api/github/webhook-subscription` when a PR with `sha=wxyz` merged. 2. Cocoon schedules tasks on LUCI and creates records in `tasks` collection in Firestore for each configured partner. 3. For every configured partner, Cocoon: - * Creates a record in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, + * Creates a record in `partner_ci` collection in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, * Publishes a message containing the `uid` and commit context (`sha`, `pr`, `slug`, etc.) to an internal GCP Pub/Sub topic. ```mermaid @@ -107,7 +107,7 @@ sequenceDiagram participant Firestore@{"type":"database"} destroy GitHub GitHub->>+Cocoon: Postsubmit Event - Cocoon->>Firestore: Creates record in `tasks` collection
for sha=wxyz with uid=abcd,
partner name, attempt number, etc. + Cocoon->>Firestore: Creates records in `tasks` and `partner_ci` collections
for sha=wxyz with uid=abcd,
partner name, attempt number, etc. Cocoon->>-PubSub: Schedule partner notification
event for sha=wxyz with uid=abcd ``` @@ -184,7 +184,7 @@ sequenceDiagram Note over Cocoon: User requested task status while waiting execution results Cocoon->>Partner: HTTP GET /status?uid=abcd - Partner-->>Cocoon: Response 202 {"status": "in_progress"} + Partner-->>Cocoon: Response 200 {"status": "in_progress"} Note over Partner: Continue executing custom tests ``` @@ -203,7 +203,7 @@ sequenceDiagram participant PubSub@{"type":"queue"} participant Cocoon as Cocoon Server participant Partner as Partner CI - + Note over Partner: Finishes executing custom tests Partner->>Cocoon: HTTP POST Callback (signed with partner key,
JWT body with status for sha=wxyz with uid=abcd) Note over Partner: Response timeout @@ -247,6 +247,7 @@ sequenceDiagram opt Partner calls cocoon webhook to provide CI run status update Partner->>+Cocoon: HTTP POST Callback (signed with partner key,
JWT body with status for sha=wxyz with uid=efgh) + Cocoon->>Partner: HTTP GET to Partner JWKS url if missed in cache Partner-->>Cocoon: JWKS to verify JWT signature Cocoon->>PubSub: Partner test execution status
for sha=wxyz with uid=efgh Cocoon-->>-Partner: Response 200 @@ -276,17 +277,19 @@ Example of JWKS: ##### 4.1.2.3. Serialized JWT HTTP Body -When either system sends an HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender constructs a POST request as follows: +When either system sends a webhook or callback HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender constructs a `POST` request as follows: * **Header**: `Content-Type: application/jwt` * **Body**: A signed JSON Web Token (JWT) +For manual on-demand status queries (`GET /status?uid=`), Cocoon passes the signed JWT in the `Authorization: Bearer ` header. + ##### 4.1.2.4. JWT Payload Structure The JWT consists of a base64 encoded header, payload, and signature separated by dots. For example: -`eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6MTIzNDV9.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovLzNwLXBhcnRuZXItY2kuY29tIiwiaWF0IjoxNzc0OTYyMDAwLCJleHAiOjE3NzQ5NjIzMDAsImRhdGEiOnsic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` +`eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6IjEyMzQ1In0.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovL3BhcnRuZXItY2kuZXhhbXBsZS5jb20vd2ViaG9vayIsImlhdCI6MTc3NDk2MjAwMCwiZXhwIjoxNzc0OTYyMzAwLCJkYXRhIjp7InVpZCI6IjBzQXBMaHdibUd6Mlkwd0JEelBMIiwic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` ##### 4.1.2.5. JWT Header @@ -310,11 +313,12 @@ Where: { // --- Standard Security Metadata --- "iss": "cocoon", - "aud": "https://3p-partner-ci.com/webhook", + "aud": "https://partner-ci.example.com/webhook", "iat": 1774962000, "exp": 1774962300, // --- The Actual Message (Custom Claims) --- "data": { + "uid": "0sAtLhwbmGz2Y0WBDzPL", "slug": "flutter/flutter", "pr": 12345, "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b" @@ -343,7 +347,7 @@ In addition, Cocoon's callback endpoint validates that `data.uid` exists, has no * Industry-standard webhook architecture that is easy for partners to consume. * Push-based trigger is real-time and minimizes latency. * Decouples Flutter core infrastructure from partner device management. - * Stateless design on Cocoon via Firestore/Redis and GCP Pub/Sub. + * Stateless design on Cocoon via Firestore and GCP Pub/Sub. * Status query endpoints allow robust reconciliation on-demand without automated polling loops. * **Cons:** * Requires partners to maintain publicly accessible HTTP endpoints and handle inbound requests. @@ -501,7 +505,7 @@ processed bool // Whether terminal status response from partner was alrea #### 5.1.3. tasks -Extend `tasks` collection with `message` and `log_urls` fields. +Extend `tasks` collection with `message` and `details` fields. ### 5.2. Pub/Sub @@ -552,7 +556,7 @@ See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc751 ```json { "iss": "cocoon", - "aud": "https://partner-ci.com/webhook", + "aud": "https://partner-ci.example.com/webhook", "iat": 1774962000, "exp": 1774962300, "data": { @@ -604,7 +608,7 @@ Content-Type: application/jwt ```json { "iss": "3p-partner-ci", - "aud": "https://dashboard.flutter.dev/api/partner/callback", + "aud": "https://cocoon.flutter.dev/api/partner/callback", "iat": 1774963500, "exp": 1774963800, "data": { @@ -705,27 +709,27 @@ Content-Type: application/json { "name": "test_1", "status": "failed", - "log_url": "https://parnter.com/test_1/log" + "url": "https://parnter.com/test_1/log" }, { "name": "test_2", "status": "succeeded", - "log_url": "https://parnter-ci.com/test_2/log" + "url": "https://parnter-ci.com/test_2/log" }, { "name": "test_3", "status": "skipped", - "log_url": "https://parnter-ci.com/test_3/log" + "url": "https://parnter-ci.com/test_3/log" }, { "name": "test_4", "status": "failed", - "log_url": "https://parnter-ci.com/test_4/log" + "url": "https://parnter-ci.com/test_4/log" }, { "name": "test_5", "status": "in_progress", - "log_url": null + "url": null } ] } @@ -752,4 +756,4 @@ Where: * `succeeded` - Test passed (success state). * `neutral` - Test failed but considered flaky (success state). * `skipped` - Test skipped (success state). - * **log_url**: The publicly accessible task execution log URL. + * **url**: The publicly accessible task execution details or logs URL. From eab17a45ec22612d6b2a5d5074011b87390d9aa6 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 16:25:56 -0700 Subject: [PATCH 31/35] Fixed Typos & Formatting Issues --- rfc/610.0000-flutter-pluggable-ci.md | 30 ++++++++++++++-------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index be47e7f..01be3aa 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -17,7 +17,7 @@ authors: ## 1. Problem Statement -To allow partners who maintain off-tree embedders to detect regression immediately after a commit lands on the flutter main branch by running their CI on every flutter PR post-submit. +To allow partners who maintain out-of-tree embedders to detect regression immediately after a commit lands on the Flutter main branch by running their CI on every Flutter PR post-submit. ## 2. Background @@ -145,7 +145,7 @@ sequenceDiagram Note over Partner: Start executing custom tests ``` -##### 4.1.1.3. Optionally, Partner may notify cocoon it started test execution and provide test details +##### 4.1.1.3. Optionally, Partner may notify Cocoon it started test execution and provide test details 1. When the partner CI begins test execution for `sha=wxyz` (`uid=abcd`), it may send a signed JWT callback to Cocoon with `in_progress` status along with scheduled test names and links to execution details. 2. Cocoon verifies the JWT signature (fetching the partner's JWKS on cache miss), publishes the status to Pub/Sub, responds with `200`, and updates the task status in Firestore to `in_progress` (if the task has not already reached a terminal state, guarding against out-of-order Pub/Sub delivery). @@ -189,7 +189,7 @@ sequenceDiagram Note over Partner: Continue executing custom tests ``` -##### 4.1.1.5. Partner calls cocoon webhook to provide CI run status update, retrying on failures +##### 4.1.1.5. Partner calls Cocoon webhook to provide CI run status update, retrying on failures 1. Once the partner CI finishes executing tests on proprietary hardware for `sha=wxyz` (`uid=abcd`), it makes an HTTP POST callback to Cocoon signed with the partner's key containing the terminal execution status (`cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), summary message, and test URLs. * If the callback request experiences a response timeout or receives a retryable failure response (`5xx` or `408 Request Timeout`), the partner CI retries the HTTP POST callback for `uid=abcd`. @@ -219,7 +219,7 @@ sequenceDiagram deactivate Cocoon ``` -##### 4.1.1.6. If test failed, Cocon may request re-run of failed task +##### 4.1.1.6. If test failed, Cocoon may request re-run of failed task 1. When a task re-run is initiated for commit `sha=wxyz` (either automatically by Cocoon's flake/retry policy or manually from the build dashboard), Cocoon generates a new unique correlation ID (`uid=efgh`) for the new attempt while retaining the original commit `sha=wxyz`, and publishes a new dispatch message to Pub/Sub. 2. The Pub/Sub subscriber sends a new signed JWT HTTP POST notification to the Partner CI webhook containing `sha=wxyz` and `uid=efgh`. Issuing a distinct `uid` ensures the partner treats the request as a new execution rather than rejecting it as a duplicate (`409 Conflict`) of `uid=abcd`. @@ -619,27 +619,27 @@ Content-Type: application/jwt { "name": "test_1", "status": "failed", - "url": "https://parnter.com/test_1/log" + "url": "https://partner-ci.example.com/test_1/log" }, { "name": "test_2", "status": "succeeded", - "url": "https://parnter-ci.com/test_2/log" + "url": "https://partner-ci.example.com/test_2/log" }, { "name": "test_3", "status": "skipped", - "url": "https://parnter-ci.com/test_3/log" + "url": "https://partner-ci.example.com/test_3/log" }, { "name": "test_4", "status": "failed", - "url": "https://parnter-ci.com/test_4/log" + "url": "https://partner-ci.example.com/test_4/log" }, { "name": "test_5", "status": "cancelled", - "url": "https://parnter-ci.com/test_5/log" + "url": "https://partner-ci.example.com/test_5/log" } ] } @@ -655,7 +655,7 @@ Where: * **data**: Object containing custom payload claims and event details: * **uid**: The unique correlation identifier to match which CI execution request this status response belongs to. * **status**: - * `in_progress` - Task is in progress. Used only for optional `in_progress` status notification. Can not be in terminal status notification. + * `in_progress` - Task is in progress. Used only for optional `in_progress` status notification. Cannot be in terminal status notification. * `cancelled` - Task is cancelled by a Partner CI (failure state). * `infra_failure` - Infrastructure failure during tests execution (failure state). * `failed` - Some or all of the tests are failed (failure state). @@ -666,7 +666,7 @@ Where: * **details**: Granular test execution details. * **name**: Test name. * **status**: - * `in_progress` - Test is in progress. Used only for optional `in_progress` status notification. Can not be in terminal status notification. + * `in_progress` - Test is in progress. Used only for optional `in_progress` status notification. Cannot be in terminal status notification. * `cancelled` - Test is cancelled by a Partner CI (failure state). * `infra_failure` - Infrastructure failure during test execution (failure state). * `failed` - Test failed (failure state). @@ -709,22 +709,22 @@ Content-Type: application/json { "name": "test_1", "status": "failed", - "url": "https://parnter.com/test_1/log" + "url": "https://partner-ci.example.com/test_1/log" }, { "name": "test_2", "status": "succeeded", - "url": "https://parnter-ci.com/test_2/log" + "url": "https://partner-ci.example.com/test_2/log" }, { "name": "test_3", "status": "skipped", - "url": "https://parnter-ci.com/test_3/log" + "url": "https://partner-ci.example.com/test_3/log" }, { "name": "test_4", "status": "failed", - "url": "https://parnter-ci.com/test_4/log" + "url": "https://partner-ci.example.com/test_4/log" }, { "name": "test_5", From d6d79e29f5d7bc016e0b8f8b9497cb19d4ab5702 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Thu, 15 Oct 2026 16:34:48 -0700 Subject: [PATCH 32/35] Gaps to Consider --- rfc/610.0000-flutter-pluggable-ci.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 01be3aa..a8d5578 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -27,6 +27,7 @@ Multiple Flutter partners adopt custom embedders to support specific platforms n * Flutter TLs * First-party partners +* Third-party partners ### 2.2. Glossary @@ -118,9 +119,9 @@ sequenceDiagram * fetches partner configurations, * generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, * issues an HTTP POST webhook request to the partner's configured URL. -2. If webhook invocation times out or partner server returns 5xx (Server error) or 408 (Request Timeout) response, Cocoon will not acknowledge the Pub/Sub message and will retry the dispatch. +2. If webhook invocation times out or partner server returns 5xx (Server error), 408 (Request Timeout), or 429 (Too Many Requests) response, Cocoon will not acknowledge the Pub/Sub message and will retry the dispatch. 3. If a partner webhook responds with success code 200, Cocoon will acknowledge the message. -4. If a partner webhook responds with client error code 4xx (except 408 Request Timeout), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. In both cases, Cocoon will acknowledge the message and will not retry the dispatch. +4. If a partner webhook responds with client error code 4xx (except 408 Request Timeout and 429 Too Many Requests), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. In both cases, Cocoon will acknowledge the message and will not retry the dispatch. ```mermaid sequenceDiagram @@ -192,7 +193,7 @@ sequenceDiagram ##### 4.1.1.5. Partner calls Cocoon webhook to provide CI run status update, retrying on failures 1. Once the partner CI finishes executing tests on proprietary hardware for `sha=wxyz` (`uid=abcd`), it makes an HTTP POST callback to Cocoon signed with the partner's key containing the terminal execution status (`cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), summary message, and test URLs. - * If the callback request experiences a response timeout or receives a retryable failure response (`5xx` or `408 Request Timeout`), the partner CI retries the HTTP POST callback for `uid=abcd`. + * If the callback request experiences a response timeout or receives a retryable failure response (`5xx`, `408 Request Timeout`, or `429 Too Many Requests`), the partner CI retries the HTTP POST callback for `uid=abcd`. * Upon receiving the callback, Cocoon reads the Key ID (`kid`) from the JWT header and looks up the corresponding cached public key. If the public key is not found in cache, it fetches the partner's JWKS to update the cache, or responds with `401` if the key is not found or JWT signature validation fails. * If Cocoon successfully validates the payload and publishes the partner test execution status to Pub/Sub, it responds with status code `200`. * If Cocoon fails to process the request, it responds with non-retryable error status code, such as `400` for a validation error, `404` if the `uid` is not found, `409` if a terminal status was already processed, or `413` if the body size exceeds 768KB (to stay within Firestore's 1 MiB document size limit). @@ -521,6 +522,7 @@ To guarantee that Cocoon can properly process status updates once the status web * What other fields should we provide to partners webhook to request CI run? * Can `tasks` collection PK `__` be used as `uid`? In that case we would not need a `partner_ci` collection. +* How should Cocoon handle partner tasks that never receive a terminal status callback (e.g., marking tasks stuck in `New` or `in_progress` as `infra_failure` after a configurable timeout)? ## 7. Testing Plan From fe56ffc4a8de96503b70825d0472a018bf6c9453 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Fri, 16 Oct 2026 15:33:50 -0700 Subject: [PATCH 33/35] restructured --- rfc/610.0000-flutter-pluggable-ci.md | 352 +++++++++++++++------------ 1 file changed, 203 insertions(+), 149 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index a8d5578..a08fd0e 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -5,45 +5,74 @@ title: Flutter Pluggable CI Design description: Describes the integration with partner CI systems to trigger partner CI execution on post-submit PRs and display execution details on the post-submit dashboard. status: draft created: 2026-10-06T09:00:00Z -updated: 2026-10-08T00:00:00Z +updated: 2026-10-15T00:00:00Z tags: - 600-infrastructure - 610-cicd + - 450-custom-embedders authors: - '"Dmitry Grand" ' --- # RFC 610.0000: Flutter Pluggable CI Design -## 1. Problem Statement +## 1. Overview -To allow partners who maintain out-of-tree embedders to detect regression immediately after a commit lands on the Flutter main branch by running their CI on every Flutter PR post-submit. +To enable Flutter to scale across external platforms and out-of-tree embedder maintainers, Flutter infrastructure (Cocoon) requires a robust, secure, and decoupled communication model. This system notifies partner CI pipelines of post-submit events on `flutter/flutter` so partners can execute custom test suites on proprietary hardware and report execution status back to the Flutter build dashboard. -## 2. Background +### 1.1. Problem Statement + +To allow partners who maintain out-of-tree embedders to detect regressions immediately after PR merged in Flutter repo by running their CI on every Flutter post-submit event. + +### 1.2. Background & Audience Multiple Flutter partners adopt custom embedders to support specific platforms not officially supported by Flutter. Changes in the upstream Flutter repository often cause build failures for custom embedders. Depending on how frequently a partner syncs their fork, isolating the exact breaking change is difficult to manage. This solution mitigates the issue by allowing partners to run their CI pipelines against every Flutter PR post-submit, catching regressions early. We cannot allow partner CI execution to block the tree, at least for now, because gardeners responsible for maintaining tree health lack the knowledge or access to partner codebases and infrastructure necessary to resolve failures effectively. -### 2.1. Audience +**Target Audience:** * Flutter TLs * First-party partners * Third-party partners -### 2.2. Glossary +### 1.3. Terminology + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [BCP 14](https://www.rfc-editor.org/info/bcp14) ([RFC 2119](https://www.rfc-editor.org/info/rfc2119/)). + +### 1.4. Glossary -* **In-Tree Platforms:** These are core Flutter platforms that are stored, updated, and managed directly inside the main Flutter repository. This includes the official ports for iOS, Android, macOS, Linux, Windows and Web. -* **Out-of-Tree Platforms:** These are external Flutter platforms developed and maintained by third parties in separate, independent repositories. Because they live outside the main codebase, today developers must fork the Flutter tool and reverse-engineer undocumented system channels (like text input or app lifecycles) to make them work. -* **Custom Embedders:** These are platform-specific codebases that host the Flutter engine and Dart runtime on non-standard operating systems. They act as out-of-tree ports for custom hardware. +* **In-Tree Platforms:** Core Flutter platforms that are stored, updated, and managed directly inside the main Flutter repository. This includes the official ports for iOS, Android, macOS, Linux, Windows, and Web. +* **Out-of-Tree Platforms:** External Flutter platforms developed and maintained by third parties in separate, independent repositories. Because they live outside the main codebase, today developers must fork the Flutter tool and reverse-engineer undocumented system channels (like text input or app lifecycles) to make them work. +* **Custom Embedders:** Platform-specific codebases that host the Flutter engine and Dart runtime on non-standard operating systems. They act as out-of-tree ports for custom hardware. * **Cocoon:** Flutter's custom Continuous Integration (CI) orchestration system that integrates underlying CI services like LUCI with GitHub to schedule tests and report build statuses. It also provides developer tooling such as auto-submit mechanisms, test flakiness tracking, and powers the public Flutter Pre-submit/Build Dashboards. * **LUCI (Layered Universal Continuous Integration):** The underlying execution engine that runs Flutter's testing and build workloads across diverse platforms like macOS, Windows, and Linux. Orchestrated by Cocoon, LUCI executes specific Python-based "recipes" to securely validate code changes for the Flutter framework, engine, and packages during both pre-submit and post-submit pipelines. +* **Post-submit Event:** An event in Cocoon that is triggered after a PR is merged into the `flutter/flutter` repository. This may be a merge to `master`, `beta: *`, `staging: *` or release candidate branches. + + +--- + +## 2. Goals & Non-Goals -## 3. Overview +### 2.1. Goals -To enable Flutter to scale across external platforms and out-of-tree embedder maintainers, Flutter infrastructure (Cocoon) requires a robust, secure, and decoupled communication model. This system notifies partner CI pipelines of post-submit commits on `flutter/flutter` so partners can execute custom test suites on proprietary hardware and report execution status back to the Flutter build dashboard. +* **Non-Blocking Post-Submit Execution:** Trigger partner CI runs on post-submit events and display results on the Flutter build dashboard without blocking the tree. +* **Decentralized Security:** Authenticate bidirectional server-to-server requests using [Ed25519](https://www.rfc-editor.org/info/rfc8410/) asymmetric key pairs and public JWKS endpoints without a central authorization server. +* **High Resilience:** Tolerate temporary network and partner infrastructure outages via asynchronous GCP Pub/Sub buffering and exponential backoff retries. +* **Minimal Operational Overhead:** Avoid automated status polling loops by combining push callbacks with optional manual on-demand status reconciliation. -Key architectural goals include non-blocking test execution, strict security via [Ed25519](https://www.rfc-editor.org/info/rfc8410/) public/private key authentication, high resilience against network outages, and minimal operational overhead for Flutter infrastructure maintainers. +### 2.2. Non-Goals -The proposed solution is to use webhooks for bidirectional asynchronous communication backed internally by GCP Pub/Sub for fault tolerance and retries. +* Defining what kind of CI system out-of-tree platform maintainers must use or what tests they run to validate custom embedders. +* Guaranteeing the delivery of CI execution status from partner CI systems back to Cocoon if the partner infrastructure permanently fails to invoke the callback endpoint and does not support the manual status endpoint. + +--- + +## 3. System Architecture + +Integration with third-party CIs requires two main components: communication and authentication. The recommended architecture is **bi-directional webhook server-to-server communication (Push-Callback)** backed internally by GCP Pub/Sub for fault tolerance and retries, combined with an optional **manual on-demand status reconciliation** mechanism. + +### 3.1. High-Level Architecture + +In this architecture, Cocoon uses standard HTTP webhooks to push event notifications to external partner endpoints. External partners process the event asynchronously on their custom infrastructure and send execution status back via Cocoon's callback REST API. Additionally, partners MAY host an endpoint capable of responding to on-demand status queries from Cocoon to enable manual status checks from the build dashboard, eliminating automated polling loops. Below is a high-level flow diagram of the proposed communication between Cocoon and Partner CI: @@ -78,26 +107,14 @@ sequenceDiagram deactivate Cocoon ``` -### 3.1 Non-goals - -This design does not cover what kind of CI for out-of-tree platforms to use and what tests to run to validate custom embedders. Nor does it cover how to guarantee the delivery of CI execution status from partner CI systems back to Cocoon. +### 3.2. End-to-End Workflows -## 4. Detailed Design +#### 3.2.1. Internal Processing of Post-Submit Event -Integration with third-party CIs requires two main points of discussion: communication and authentication. For external system notification, the recommended approach is a bi-directional webhook server-to-server communication, combined with a manual on-demand status reconciliation mechanism. - -### 4.1. Primary Design: Webhook Server-to-Server Communication (Push-Callback) with Manual Status Request - -In this architecture, Cocoon uses standard HTTP webhooks to push event notifications to external partner endpoints. External partners process the event asynchronously on their custom infrastructure and send execution status back via Cocoon's callback REST API. Additionally, partners may optionally host an endpoint capable of responding to on-demand status queries from Cocoon to enable manual status checks from the build dashboard, eliminating automated polling loops. - -#### 4.1.1. End-to-End Workflow Diagram - -##### 4.1.1.1. Internal processing of post submit event - -1. GitHub notifies Cocoon via `/api/github/webhook-subscription` when a PR with `sha=wxyz` merged. -2. Cocoon schedules tasks on LUCI and creates records in `tasks` collection in Firestore for each configured partner. +1. GitHub notifies Cocoon via `/api/github/webhook-subscription` when a PR with `sha=wxyz` is merged. +2. Cocoon schedules tasks on LUCI and creates records in the `tasks` collection in Firestore for each configured partner. 3. For every configured partner, Cocoon: - * Creates a record in `partner_ci` collection in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, + * Creates a record in the `partner_ci` collection in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, * Publishes a message containing the `uid` and commit context (`sha`, `pr`, `slug`, etc.) to an internal GCP Pub/Sub topic. ```mermaid @@ -112,16 +129,16 @@ sequenceDiagram Cocoon->>-PubSub: Schedule partner notification
event for sha=wxyz with uid=abcd ``` -##### 4.1.1.2. Cocoon calls partner webhook to notify about post submit to run CI, retrying on failures +#### 3.2.2. Cocoon Calls Partner Webhook to Notify About Post-Submit to Run CI, Retrying on Failures 1. A Pub/Sub subscriber: - * receives partner notification event, + * receives the partner notification event, * fetches partner configurations, * generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, * issues an HTTP POST webhook request to the partner's configured URL. -2. If webhook invocation times out or partner server returns 5xx (Server error), 408 (Request Timeout), or 429 (Too Many Requests) response, Cocoon will not acknowledge the Pub/Sub message and will retry the dispatch. -3. If a partner webhook responds with success code 200, Cocoon will acknowledge the message. -4. If a partner webhook responds with client error code 4xx (except 408 Request Timeout and 429 Too Many Requests), Cocoon will log that as an error. Status code 409 (Conflict) will be considered an already processed notification and will be logged as a warning. In both cases, Cocoon will acknowledge the message and will not retry the dispatch. +2. If webhook invocation times out or the partner server returns `5xx` (Server Error), `408` (Request Timeout), or `429` (Too Many Requests), Cocoon MUST NOT acknowledge the Pub/Sub message and will retry the dispatch. +3. If a partner webhook responds with success code `200`, Cocoon acknowledges the message. +4. If a partner webhook responds with client error code `4xx` (except `408 Request Timeout` and `429 Too Many Requests`), Cocoon logs it as an error. Status code `409` (Conflict) is treated as an already processed notification and logged as a warning. In both cases, Cocoon acknowledges the message and does not retry the dispatch. ```mermaid sequenceDiagram @@ -146,18 +163,18 @@ sequenceDiagram Note over Partner: Start executing custom tests ``` -##### 4.1.1.3. Optionally, Partner may notify Cocoon it started test execution and provide test details +#### 3.2.3. Optionally, Partner May Notify Cocoon It Started Test Execution and Provide Test Details -1. When the partner CI begins test execution for `sha=wxyz` (`uid=abcd`), it may send a signed JWT callback to Cocoon with `in_progress` status along with scheduled test names and links to execution details. +1. When the partner CI begins test execution for `sha=wxyz` (`uid=abcd`), it MAY send a signed JWT callback to Cocoon with `in_progress` status along with scheduled test names and links to execution details. 2. Cocoon verifies the JWT signature (fetching the partner's JWKS on cache miss), publishes the status to Pub/Sub, responds with `200`, and updates the task status in Firestore to `in_progress` (if the task has not already reached a terminal state, guarding against out-of-order Pub/Sub delivery). -3. That will lead to showing test names and links to execution details in the Cocoon build dashboard. +3. This displays the scheduled test names and links to execution details in the Cocoon build dashboard. ```mermaid sequenceDiagram participant PubSub@{"type":"queue"} participant Cocoon as Cocoon Server participant Partner as Partner CI - + Partner->>+Cocoon: HTTP POST Callback (signed with partner key, JWT body
with `in_progress` status for sha=wxyz with uid=abcd) Cocoon->>Partner: HTTP GET to Partner JWKS url if missed in cache Partner-->>Cocoon: JWKS to verify JWT signature @@ -169,12 +186,12 @@ sequenceDiagram deactivate Cocoon ``` -##### 4.1.1.4. Optionally, if supported by Partner, Cocoon can poll for status updates +#### 3.2.4. Optionally, If Supported by Partner, Cocoon Can Poll for Status Updates -Partners may host a status endpoint (e.g., `GET /status?uid=abcd`). +Partners MAY host a status endpoint (e.g., `GET /status?uid=abcd`). 1. In the event that a webhook callback fails to arrive or a build hangs, authorized users can manually trigger a status query from the Cocoon build dashboard while waiting for execution results. -2. Cocoon will make a GET request to the partner's status endpoint, fetch the latest status, and update the task state accordingly. +2. Cocoon makes a `GET` request to the partner's status endpoint, fetches the latest status, and updates the task state accordingly. **Note:** Automated polling is strictly rejected to avoid DoS risks and operational overhead. @@ -190,13 +207,13 @@ sequenceDiagram Note over Partner: Continue executing custom tests ``` -##### 4.1.1.5. Partner calls Cocoon webhook to provide CI run status update, retrying on failures +#### 3.2.5. Partner Calls Cocoon Webhook to Provide CI Run Status Update, Retrying on Failures 1. Once the partner CI finishes executing tests on proprietary hardware for `sha=wxyz` (`uid=abcd`), it makes an HTTP POST callback to Cocoon signed with the partner's key containing the terminal execution status (`cancelled`, `infra_failure`, `failed`, `succeeded`, `neutral`, `skipped`), summary message, and test URLs. - * If the callback request experiences a response timeout or receives a retryable failure response (`5xx`, `408 Request Timeout`, or `429 Too Many Requests`), the partner CI retries the HTTP POST callback for `uid=abcd`. - * Upon receiving the callback, Cocoon reads the Key ID (`kid`) from the JWT header and looks up the corresponding cached public key. If the public key is not found in cache, it fetches the partner's JWKS to update the cache, or responds with `401` if the key is not found or JWT signature validation fails. - * If Cocoon successfully validates the payload and publishes the partner test execution status to Pub/Sub, it responds with status code `200`. - * If Cocoon fails to process the request, it responds with non-retryable error status code, such as `400` for a validation error, `404` if the `uid` is not found, `409` if a terminal status was already processed, or `413` if the body size exceeds 768KB (to stay within Firestore's 1 MiB document size limit). + * If the callback request experiences a response timeout or receives a retryable failure response (`5xx`, `408 Request Timeout`, or `429 Too Many Requests`), the partner CI SHOULD retry the HTTP POST callback for `uid=abcd`. + * Upon receiving the callback, Cocoon reads the Key ID (`kid`) from the JWT header and looks up the corresponding cached public key. If the public key is not found in cache, it fetches the partner's JWKS to update the cache, or responds with `401` if the key is not found or JWT signature validation fails. + * If Cocoon successfully validates the payload and publishes the partner test execution status to Pub/Sub, it responds with status code `200`. + * If Cocoon fails to process the request, it responds with a non-retryable error status code, such as `400` for a validation error, `404` if the `uid` is not found, `409` if a terminal status was already processed, or `413` if the body size exceeds 768KB (to stay within Firestore's 1 MiB document size limit). 2. Once Cocoon receives the partner test execution status for `sha=wxyz` (`uid=abcd`) from Pub/Sub, it updates the partner task state in Firestore. ```mermaid @@ -220,7 +237,7 @@ sequenceDiagram deactivate Cocoon ``` -##### 4.1.1.6. If test failed, Cocoon may request re-run of failed task +#### 3.2.6. If Test Failed, Cocoon May Request Re-run of Failed Task 1. When a task re-run is initiated for commit `sha=wxyz` (either automatically by Cocoon's flake/retry policy or manually from the build dashboard), Cocoon generates a new unique correlation ID (`uid=efgh`) for the new attempt while retaining the original commit `sha=wxyz`, and publishes a new dispatch message to Pub/Sub. 2. The Pub/Sub subscriber sends a new signed JWT HTTP POST notification to the Partner CI webhook containing `sha=wxyz` and `uid=efgh`. Issuing a distinct `uid` ensures the partner treats the request as a new execution rather than rejecting it as a duplicate (`409 Conflict`) of `uid=abcd`. @@ -260,39 +277,54 @@ sequenceDiagram end ``` -#### 4.1.2. Security & Authentication +### 3.3. Pros & Cons + +* **Pros:** + * Industry-standard webhook architecture that is easy for partners to consume. + * Push-based trigger is real-time and minimizes latency. + * Decouples Flutter core infrastructure from partner device management. + * Stateless design on Cocoon via Firestore and GCP Pub/Sub. + * Status query endpoints allow robust reconciliation on-demand without automated polling loops. +* **Cons:** + * Requires partners to maintain publicly accessible HTTP endpoints and handle inbound requests. + +--- + +## 4. Design + +### 4.1. Security & Authentication This section details the decentralized security model used to authorize server-to-server bidirectional webhook communication without a centralized authorization server. -##### 4.1.2.1. Asymmetric Cryptography & Key Management +#### 4.1.1. Asymmetric Cryptography & Key Management Both the Cocoon server and the Partner CI server independently generate their own asymmetric key pairs (using [Ed25519](https://www.rfc-editor.org/info/rfc8410/)). Each system stores its private key in a secure key management system and shares its corresponding public key with the other party. By leveraging decentralized asymmetric cryptography, this approach intentionally avoids the need for a central Authorization Server or Identity Provider, eliminating a potential single point of failure and reducing infrastructure complexity. -##### 4.1.2.2. Key Distribution Strategy: Public Key Endpoints +#### 4.1.2. Key Distribution Strategy: Public Key Endpoints -For this authentication model to work, both parties must reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to fetch or refresh this key periodically. This is highly secure because it allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. +For this authentication model to work, both parties MUST reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of the JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to cache and refresh this key set periodically. On a `kid` cache miss, the verifier MAY trigger an on-demand JWKS refresh subject to a per-partner cooldown (e.g., at most once every 5 minutes) and a strict HTTP timeout to prevent cache-miss DoS amplification. This allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. Example of JWKS: -##### 4.1.2.3. Serialized JWT HTTP Body +#### 4.1.3. Serialized JWT HTTP Request Format -When either system sends a webhook or callback HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender constructs a `POST` request as follows: +When either system sends a webhook or callback HTTP request (Cocoon dispatching a webhook or a Partner sending status callbacks), the sender MUST construct a `POST` request as follows: * **Header**: `Content-Type: application/jwt` * **Body**: A signed JSON Web Token (JWT) -For manual on-demand status queries (`GET /status?uid=`), Cocoon passes the signed JWT in the `Authorization: Bearer ` header. +For manual on-demand status queries (`GET /status?uid=`), Cocoon passes the signed JWT in the `Authorization: Bearer ` header with `"aud"` set to the partner's configured `status_url` and `"data": {"uid": ""}`. -##### 4.1.2.4. JWT Payload Structure +#### 4.1.4. JWT Payload Structure -The JWT consists of a base64 encoded header, payload, and signature separated by dots. +The JWT consists of a base64url-encoded header, payload, and signature separated by dots. For example: `eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6IjEyMzQ1In0.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovL3BhcnRuZXItY2kuZXhhbXBsZS5jb20vd2ViaG9vayIsImlhdCI6MTc3NDk2MjAwMCwiZXhwIjoxNzc0OTYyMzAwLCJkYXRhIjp7InVpZCI6IjBzQXBMaHdibUd6Mlkwd0JEelBMIiwic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` -##### 4.1.2.5. JWT Header +#### 4.1.5. JWT Header ```json { @@ -304,11 +336,13 @@ For example: Where: -* **alg**: The cryptographic algorithm used to sign the token (e.g., `EdDSA` for [Ed25519](https://www.rfc-editor.org/info/rfc8410/) signatures). -* **typ**: The type of the token, which is set to `JWT`. -* **kid**: Key ID that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. +* **alg**: The cryptographic algorithm used to sign the token (`EdDSA` for [Ed25519](https://www.rfc-editor.org/info/rfc8410/) signatures). +* **typ**: The type of the token, which MUST be set to `JWT`. +* **kid**: Key ID string that identifies the specific public key needed to verify the token's signature from the JWKS endpoint. + +#### 4.1.6. JWT Payload -##### 4.1.2.6. JWT Payload +Tokens are generated and signed per HTTP request attempt with a short-lived expiration window (e.g., 5 minutes): ```jsonc { @@ -327,39 +361,92 @@ Where: } ``` -##### 4.1.2.7. JWT Signature +#### 4.1.7. JWT Signature [Ed25519 signature](https://www.rfc-editor.org/info/rfc8037/#appendix-A.4) as defined in [RFC 8037](https://www.rfc-editor.org/info/rfc8037). -##### 4.1.2.8. Validating Incoming Requests +#### 4.1.8. Validating Incoming Requests -Webhook listeners validate incoming requests in accordance with standard JWT/JWS/JWKS specifications and security best practices, specifically: +Webhook listeners MUST validate incoming requests in accordance with standard JWT/JWS/JWKS specifications and security best practices, specifically: * [RFC 7519 Section 7.2](https://www.rfc-editor.org/info/rfc7519/#section-7.2) for JWT validation; * [RFC 7515 Section 5.2](https://www.rfc-editor.org/info/rfc7515/#section-5.2) for signature verification; * [RFC 7517 Section 4.5](https://www.rfc-editor.org/info/rfc7517/#section-4.5) & [Section 5](https://www.rfc-editor.org/info/rfc7517/#section-5) for `kid` lookup via cached JWKS; -* [RFC 8725 Section 3](https://www.rfc-editor.org/info/rfc8725/#section-3) for algorithm pinning and `iss`/`aud`/`exp` claim checks. +* [RFC 8725 Section 3](https://www.rfc-editor.org/info/rfc8725/#section-3) for algorithm pinning (`EdDSA`) and `iss`/`aud`/`exp` claim checks. -In addition, Cocoon's callback endpoint validates that `data.uid` exists, has not processed a terminal status yet, and belongs to a partner that is listed in `iss` claim. +In addition, Cocoon's callback endpoint validates that `data.uid` exists, has not processed a terminal status yet, and belongs to a partner that is listed in the `iss` claim. -#### 4.1.3. Pros & Cons +### 4.2. API Contracts & Status State Machine -* **Pros:** - * Industry-standard webhook architecture that is easy for partners to consume. - * Push-based trigger is real-time and minimizes latency. - * Decouples Flutter core infrastructure from partner device management. - * Stateless design on Cocoon via Firestore and GCP Pub/Sub. - * Status query endpoints allow robust reconciliation on-demand without automated polling loops. -* **Cons:** - * Requires partners to maintain publicly accessible HTTP endpoints and handle inbound requests. +See [Section 8 (Appendix A)](#8-appendix-a-usage-examples) for full HTTP wire examples of each endpoint. + +* **Outbound Webhook Dispatch (`POST `):** Dispatched by Cocoon with `data` claims `{uid, slug, pr, sha}`. +* **Inbound Status Callback (`POST /api/partner/callback`):** Called by Partner CI with `data` claims `{uid, status, message, details}`. Maximum request body size is 768KB. +* **Manual On-Demand Status Query (`GET ?uid=`):** Called by Cocoon with `Authorization: Bearer ` when a user manually requests status reconciliation from the dashboard; returns a JSON object `{status, message, details}`. + +**Allowed Status Values (`status` & `details[].status`):** + +* `in_progress` — Task or test is currently executing. Used only in optional start-execution callbacks ([Section 3.2.3](#323-optionally-partner-may-notify-cocoon-it-started-test-execution-and-provide-test-details)) and manual status responses ([Section 3.2.4](#324-optionally-if-supported-by-partner-cocoon-can-poll-for-status-updates)). MUST NOT be sent in terminal status callbacks. +* **Terminal Failure States:** `cancelled`, `infra_failure`, `failed`. +* **Terminal Success States:** `succeeded`, `neutral`, `skipped`. + +Once a task transitions to any terminal state, `partner_ci.processed` is set to `true` and subsequent callbacks for the same `uid` are rejected with `409 Conflict`. + +### 4.3. Data Model (Firestore) + +To support webhook flows, the following Firestore collections and schema extensions are implemented in Cocoon: -### 4.2. Alternatives Considered +#### 4.3.1. partner_config Collection -#### 4.2.1. Event-Driven GCP Pub/Sub Integration (Asynchronous Event Bus) +For partner configuration, add a `partner_config` collection with Document ID equal to `task_name`: + +```go +slug string // github repo (e.g. flutter/flutter or flutter/packages) to trigger on +task_name string // Name of post submit task to display on dashboard +partner string // 3p partner name +webhook_url string // Url to call to request CI +status_url string // Manual status request url +keys_url string // Url to read partner key id to public key mapping +``` + +Currently, there will be no UI for editing this table. For every post-submit event, dashboard checks will be extended with tasks from this list. All tasks from this list will be created suppressed without the ability to unsuppress. Later, if a decision is made to let some 1P partner CI block the tree, we may extend this table with a `block_tree` boolean field. + +#### 4.3.2. partner_ci Collection + +Add a `partner_ci` collection with Auto-ID Document ID equal to the `uid` of the `data` payload. This is used to match post-submit tasks to corresponding status responses from partner CI systems: + +```go +commit_sha string // Head sha at a moment of check-run creation ie commit sha +slug string // repository_owner/repository_name (e.g. flutter/flutter) +partner string // 3p partner name +task_name string // Name of post submit task to display on dashboard +attempt_number int // Number of re-run attempt +processed bool // Whether terminal status response from partner was already processed +``` + +#### 4.3.3. tasks Collection + +Extend the `tasks` collection with `message` and `details` fields. + +### 4.4. Asynchronous Messaging & Retries (Pub/Sub) + +#### 4.4.1. partner_ci_notification Topic + +To guarantee that Cocoon can notify each partner CI even during network outages or temporary partner infrastructure outages, once Cocoon reads the post-submit task configuration for a post-submit event, it publishes a message to the `partner_ci_notification` topic for every partner integration. A separate instance of Cocoon receives that message from `partner_ci_notification_sub` and attempts to POST it to the partner webhook using Pub/Sub exponential backoff on retryable failures (`5xx`, `408`, `429`, or network timeouts). + +#### 4.4.2. partner_ci_status Topic + +To guarantee that Cocoon can reliably process status updates once the status webhook is called by a partner and the payload is successfully validated, the payload is published to the `partner_ci_status` topic and consumed by Cocoon via `partner_ci_status_sub` to update the corresponding task status in Firestore. + +--- + +## 5. Alternatives Considered + +### 5.1. Alternative 1: Event-Driven GCP Pub/Sub Integration (Asynchronous Event Bus) This architecture eliminates direct server-to-server HTTP calls by leveraging Google Cloud Pub/Sub as an event bus between Cocoon and partner infrastructure. -##### 4.2.1.1. End-to-End Workflow Diagram +#### 5.1.1. End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -379,7 +466,7 @@ sequenceDiagram Note over Cocoon: Update Partner task status ``` -##### 4.2.1.2. Workflow +#### 5.1.2. Workflow * **1. Trigger:** GitHub triggers post-commit processing in Cocoon. * **2. Outbound Event:** Cocoon schedules the partner task and publishes an event to an outbound Pub/Sub topic. @@ -388,7 +475,7 @@ sequenceDiagram * **5. Inbound Result:** Partner CI publishes results directly back to Cocoon's inbound Pub/Sub topic. * **6. Dashboard Update:** Cocoon ingests result messages and updates test execution states in the dashboard. -##### 4.2.1.3. Pros & Cons +#### 5.1.3. Pros & Cons * **Pros:** * Highly asynchronous and resilient; messages are automatically buffered during outages. @@ -400,11 +487,11 @@ sequenceDiagram * Increases setup and development complexity for partners. * Cannot validate message body at the time of message submission. -#### 4.2.2. Alternative 2: Automated Polling Strategy (Push Trigger, Pull Status) +### 5.2. Alternative 2: Automated Polling Strategy (Push Trigger, Pull Status) This hybrid model retains HTTP push for triggering partner CI jobs, but replaces inbound callbacks with periodic polling by Cocoon to check test completion. -##### 4.2.2.1. End-to-End Workflow Diagram +#### 5.2.1. End-to-End Workflow Diagram ```mermaid sequenceDiagram @@ -441,14 +528,14 @@ sequenceDiagram Cocoon-->>PubSub: ack ``` -##### 4.2.2.2. Workflow +#### 5.2.2. Workflow * **1. Trigger & Dispatch:** GitHub triggers Cocoon, which creates partner tasks and pushes a webhook notification to the partner. * **2. Execution:** The partner CI executes workloads asynchronously. * **3. Polling Loop:** Cocoon automatically queries the partner's status endpoint periodically (e.g., using Pub/Sub retry loop). * **4. Inbound Processing:** Once the status endpoint returns a terminal state (`PASSED`/`FAILED`), Cocoon updates the dashboard. -##### 4.2.2.3. Pros & Cons +#### 5.2.3. Pros & Cons * **Pros:** * Prevents partners from needing inbound access or security tokens to write to Cocoon endpoints. @@ -458,90 +545,57 @@ sequenceDiagram * Increases latency in reporting status updates back to the Flutter dashboard due to polling intervals. * Scale limitations and risk of denial-of-service (DoS) on partner endpoints under heavy load. -### 4.3. Comparison of Options +### 5.3. Comparison of Options | Approach Criteria | Primary Design: Webhook Push-Callback | Alternative 1: Event-Driven GCP Pub/Sub | Alternative 2: Automated Polling (Push-Poll) | | --- | --- | --- | --- | | **Communication Model** | ✅ Bidirectional HTTP webhooks with manual query backup. | ✅ Asynchronous Event Bus (Pub/Sub). | ✅ HTTP push trigger, automated polling by Cocoon. | | **Partner Setup Complexity** | ⚠️ **Low to moderate.** Webhooks are easy to consume, but partners must implement retry logic for outbound webhook calls. | ❌ **High.** Requires managing cross-project GCP permissions or GCP service account keys. | ✅ **Low.** Simpler for partners to only set up webhook and status endpoints. | | **Cloud Vendor Lock-in** | ✅ **Low.** Built on standard REST and HTTP/JWT protocols. | ❌ **High.** Introduces GCP cloud vendor lock-in for third-party partners. | ✅ **Low.** Built on standard REST and HTTP/JWT protocols. | -| **Security & Authentication** | ✅ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | ⚠️ **Secure.** Managed via GCP IAM and service accounts. | ✅ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | +| **Security & Authentication** | ✅ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | ⚠️ **Secure.** Managed via GCP IAM and service accounts. | ✅ **Highly secure.** Uses decentralized asymmetric cryptography (Ed25519 JWT) with no central Auth Server. | | **Status response validation** | ✅ Can validate payload schema, unknown/duplicated `uid` and notify partner with HTTP response codes. | ⚠️ Can only validate and notify partner about payload schema violation. | ❌ Cannot notify partner of any payload schema or data validation issues. | | **Latency / Status Reporting** | ✅ **Real-time.** Push-based callback minimizes status reporting latency. | ✅ **Real-time.** Push-based consumption minimizes latency. | ❌ **Higher latency.** Updates are delayed based on Cocoon's polling intervals. | | **Resilience & Overhead** | ✅ **Highly resilient.** Retries are backed by Pub/Sub; Status query endpoint prevents polling loops. | ✅ **Highly resilient.** Messages are automatically buffered during partner infrastructure outages. | ❌ **High overhead.** Constant automated polling creates network load and risks DoS on partner endpoints. | -## 5. Integration with Existing Features - -To support webhook flows, the following must be implemented in Cocoon: - -### 5.1. Firestore - -#### 5.1.1. partner_config collection - -For partner configuration, add a `partner_config` collection with Document ID equal to `task_name`. - -```go -slug string // github repo (e.g. flutter/flutter or flutter/packages) to trigger on -task_name string // Name of post submit task to display on dashboard -partner string // 3p partner name -webhook_url string // Url to call to request CI -status_url string // Manual status request url -keys_url string // Url to read partner key id to public key mapping -``` - -Currently, there will be no UI for editing this table. For every post-submit dashboard checks will be extended with tasks from this list. All tasks from this list would be created suppressed without the ability to unsuppress. Later if a decision will be made to let some 1P partner CI block the tree we may extend this table with a `block_tree` boolean variable. - -#### 5.1.2. partner_ci collection - -Add a `partner_ci` collection with Auto-ID Document ID that will be equal to the `uid` of the `data` payload. This is used to match post-submit tasks to corresponding status responses from partner CI systems. - -```go -commit_sha string // Head sha at a moment of check-run creation ie commit sha -slug string // repository_owner/repository_name (e.g. flutter/flutter) -partner string // 3p partner name -task_name string // Name of post submit task to display on dashboard -attempt_number int // Number of re-run attempt -processed bool // Whether terminal status response from partner was already processed -``` +--- -#### 5.1.3. tasks +## 6. Testing, Rollout & Documentation Plan -Extend `tasks` collection with `message` and `details` fields. +### 6.1. Testing Plan -### 5.2. Pub/Sub +* **Unit Tests:** Validate Ed25519 JWT generation and signature verification, JWKS `kid` lookup and cache-miss refresh logic, payload schema validation, and task status state transitions (including idempotency and out-of-order `in_progress` vs. terminal status handling). +* **Integration Tests:** Verify Firestore (`partner_config`, `partner_ci`, `tasks`) and Pub/Sub (`partner_ci_notification`, `partner_ci_status`) message publishing, acknowledgment, and retry behavior. +* **End-to-End Tests:** Implement a dummy partner CI service to test the complete post-submit dispatch, `in_progress` callback, manual status reconciliation, terminal callback, and task re-run workflows. -#### 5.2.1. partner_ci_notification topic +### 6.2. Partner Onboarding & Documentation Plan -To guarantee that Cocoon can notify each partner CI even during network outages or temporary partner infrastructure outages, once Cocoon reads the post-submit task configuration for a post-submit event, it will send a message to the `partner_ci_notification` topic for every partner integration. A separate instance of Cocoon will receive that message from `partner_ci_notification_sub` and attempt to post it to the partner webhook. +Provide partner integration documentation detailing: -#### 5.2.2. partner_ci_status topic +* Generating Ed25519 key pairs and hosting a public JWKS endpoint. +* Implementing the inbound webhook listener and JWT verification. +* Implementing the outbound status callback with exponential backoff retries on `5xx`, `408`, and `429` responses. +* Optionally hosting the manual on-demand `GET /status` endpoint. -To guarantee that Cocoon can properly process status updates once the status webhook is called by a partner and the payload is successfully validated, the payload will be sent to the `partner_ci_status` topic and later received by Cocoon from `partner_ci_status_sub` to update the corresponding task status. +--- -## 6. Open Questions +## 7. Open Questions * What other fields should we provide to partners webhook to request CI run? * Can `tasks` collection PK `__` be used as `uid`? In that case we would not need a `partner_ci` collection. * How should Cocoon handle partner tasks that never receive a terminal status callback (e.g., marking tasks stuck in `New` or `in_progress` as `infra_failure` after a configurable timeout)? -## 7. Testing Plan - -Implement a dummy partner CI to test the end-to-end workflow. - -## 8. Documentation Plan - -Provide documentation for partners detailing integration setup and best practices for retry mechanisms. +--- -## 9. Appendix A. Usage Examples +## 8. Appendix A. Usage Examples -### 9.1. Communication Example: Cocoon and 3rd Party Partner +### 8.1. Communication Example: Cocoon and 3rd Party Partner -#### 9.1.1. Webhook Dispatch Request (Cocoon -> Partner) +#### 8.1.1. Webhook Dispatch Request (Cocoon -> Partner) -When a post-submit commit is processed, Cocoon posts a signed JWT to the partner's webhook URL. +When a post-submit event is processed, Cocoon posts a signed JWT to the partner's webhook URL. See: [JSON Web Token (JWT) RFC 7519](https://www.rfc-editor.org/info/rfc7519/) for more details about JWT. -##### 9.1.1.1. HTTP POST Request +##### 8.1.1.1. HTTP POST Request ```http POST /webhook HTTP/1.1 @@ -553,7 +607,7 @@ Content-Type: application/jwt See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc7515/) for more details about Signature (JWS). -##### 9.1.1.2. Decoded JWT Payload +##### 8.1.1.2. Decoded JWT Payload ```json { @@ -582,7 +636,7 @@ Where: * **pr**: The Pull Request number associated with the event. * **sha**: The target git commit SHA hash. -##### 9.1.1.3. HTTP Response +##### 8.1.1.3. HTTP Response ```http HTTP/1.1 200 OK @@ -591,11 +645,11 @@ Content-Type: application/json {} ``` -#### 9.1.2. Status Callback Request (Partner -> Cocoon) +#### 8.1.2. Status Callback Request (Partner -> Cocoon) After executing custom test suites on proprietary hardware, the partner sends the final status back to Cocoon. -##### 9.1.2.1. HTTP POST Request +##### 8.1.2.1. HTTP POST Request ```http POST /api/partner/callback HTTP/1.1 @@ -605,7 +659,7 @@ Content-Type: application/jwt .. ``` -##### 9.1.2.2. Decoded Payload +##### 8.1.2.2. Decoded Payload ```json { @@ -677,7 +731,7 @@ Where: * `skipped` - Test skipped (success state). * **url**: The publicly accessible task execution details or logs URL. -##### 9.1.2.3. HTTP Response +##### 8.1.2.3. HTTP Response ```http HTTP/1.1 200 OK @@ -686,11 +740,11 @@ Content-Type: application/json {} ``` -#### 9.1.3. Manual On-Demand Status Reconciliation Request +#### 8.1.3. Manual On-Demand Status Reconciliation Request For on-demand execution triggered by Tree Gardeners, Cocoon can manually query the partner's status endpoint. -##### 9.1.3.1. HTTP GET Request +##### 8.1.3.1. HTTP GET Request ```http GET /status?uid=0sAtLhwbmGz2Y0WBDzPL HTTP/1.1 @@ -698,7 +752,7 @@ Host: partner-ci.example.com Authorization: Bearer ``` -##### 9.1.3.2. HTTP Response +##### 8.1.3.2. HTTP Response ```http HTTP/1.1 200 OK From f3b0f6e6682e3b07de0cb3e112f34e616a808ac9 Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Fri, 16 Oct 2026 22:05:30 -0700 Subject: [PATCH 34/35] Adresed AI review concerns. --- rfc/610.0000-flutter-pluggable-ci.md | 101 ++++++++++++++++++++------- 1 file changed, 74 insertions(+), 27 deletions(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index a08fd0e..1c328a7 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -22,11 +22,17 @@ To enable Flutter to scale across external platforms and out-of-tree embedder ma ### 1.1. Problem Statement -To allow partners who maintain out-of-tree embedders to detect regressions immediately after PR merged in Flutter repo by running their CI on every Flutter post-submit event. +To allow partners who maintain out-of-tree embedders to detect regressions immediately after a PR is merged into a Flutter repository by running their CI on every Flutter post-submit event. ### 1.2. Background & Audience -Multiple Flutter partners adopt custom embedders to support specific platforms not officially supported by Flutter. Changes in the upstream Flutter repository often cause build failures for custom embedders. Depending on how frequently a partner syncs their fork, isolating the exact breaking change is difficult to manage. This solution mitigates the issue by allowing partners to run their CI pipelines against every Flutter PR post-submit, catching regressions early. We cannot allow partner CI execution to block the tree, at least for now, because gardeners responsible for maintaining tree health lack the knowledge or access to partner codebases and infrastructure necessary to resolve failures effectively. +Multiple Flutter partners adopt custom embedders to support specific platforms not officially supported by Flutter. Changes in the upstream Flutter repository often cause build failures for custom embedders. Depending on how frequently a partner syncs their fork, isolating the exact breaking change is difficult to manage. This solution mitigates the issue by allowing partners to run their CI pipelines against every Flutter PR post-submit, catching regressions early. Flutter infrastructure cannot allow partner CI execution to block the tree, at least for now, because gardeners responsible for maintaining tree health lack the knowledge or access to partner codebases and infrastructure necessary to resolve failures effectively. + +**Expected Scale:** + +* **Post-submit Event Volume:** ~20-30 post-submit events per day across all target branches on `flutter/flutter`. +* **Partner Integration Volume:** 5–20 configured partner tasks per post-submit event over the next 1–2 years. +* **Partner Execution Duration:** Typically 15 minutes to 1 hour per run, depending on custom hardware availability and test suite scope. **Target Audience:** @@ -45,7 +51,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S * **Custom Embedders:** Platform-specific codebases that host the Flutter engine and Dart runtime on non-standard operating systems. They act as out-of-tree ports for custom hardware. * **Cocoon:** Flutter's custom Continuous Integration (CI) orchestration system that integrates underlying CI services like LUCI with GitHub to schedule tests and report build statuses. It also provides developer tooling such as auto-submit mechanisms, test flakiness tracking, and powers the public Flutter Pre-submit/Build Dashboards. * **LUCI (Layered Universal Continuous Integration):** The underlying execution engine that runs Flutter's testing and build workloads across diverse platforms like macOS, Windows, and Linux. Orchestrated by Cocoon, LUCI executes specific Python-based "recipes" to securely validate code changes for the Flutter framework, engine, and packages during both pre-submit and post-submit pipelines. -* **Post-submit Event:** An event in Cocoon that is triggered after a PR is merged into the `flutter/flutter` repository. This may be a merge to `master`, `beta: *`, `staging: *` or release candidate branches. +* **Post-submit Event:** An event in Cocoon that is triggered after a PR is merged into the `flutter/flutter` repository. This may be a merge to `master`, `beta`, `stable` or release candidate branches. --- @@ -112,10 +118,10 @@ sequenceDiagram #### 3.2.1. Internal Processing of Post-Submit Event 1. GitHub notifies Cocoon via `/api/github/webhook-subscription` when a PR with `sha=wxyz` is merged. -2. Cocoon schedules tasks on LUCI and creates records in the `tasks` collection in Firestore for each configured partner. -3. For every configured partner, Cocoon: +2. Cocoon schedules tasks on LUCI and creates records in the `tasks` collection in Firestore for each active (`enabled == true`) partner configuration matching the repository `slug` and target `branch`. +3. For every matching partner configuration, Cocoon: * Creates a record in the `partner_ci` collection in Firestore with a `uid` (unique ID) to map partner notifications to Cocoon tasks, - * Publishes a message containing the `uid` and commit context (`sha`, `pr`, `slug`, etc.) to an internal GCP Pub/Sub topic. + * Publishes a message containing the `uid` and commit context (`sha`, `branch`, `pr`, `slug`, etc.) to an internal GCP Pub/Sub topic. ```mermaid sequenceDiagram @@ -131,11 +137,11 @@ sequenceDiagram #### 3.2.2. Cocoon Calls Partner Webhook to Notify About Post-Submit to Run CI, Retrying on Failures -1. A Pub/Sub subscriber: +1. A Pub/Sub push subscriber endpoint on Cocoon: * receives the partner notification event, * fetches partner configurations, * generates a short-lived JWT signed with an [Ed25519](https://www.rfc-editor.org/info/rfc8410/) private key, - * issues an HTTP POST webhook request to the partner's configured URL. + * issues an HTTP POST webhook request to the partner's configured URL with a strict **30-second HTTP timeout** (partners MUST acknowledge receipt asynchronously before running tests). 2. If webhook invocation times out or the partner server returns `5xx` (Server Error), `408` (Request Timeout), or `429` (Too Many Requests), Cocoon MUST NOT acknowledge the Pub/Sub message and will retry the dispatch. 3. If a partner webhook responds with success code `200`, Cocoon acknowledges the message. 4. If a partner webhook responds with client error code `4xx` (except `408 Request Timeout` and `429 Too Many Requests`), Cocoon logs it as an error. Status code `409` (Conflict) is treated as an already processed notification and logged as a warning. In both cases, Cocoon acknowledges the message and does not retry the dispatch. @@ -214,7 +220,7 @@ sequenceDiagram * Upon receiving the callback, Cocoon reads the Key ID (`kid`) from the JWT header and looks up the corresponding cached public key. If the public key is not found in cache, it fetches the partner's JWKS to update the cache, or responds with `401` if the key is not found or JWT signature validation fails. * If Cocoon successfully validates the payload and publishes the partner test execution status to Pub/Sub, it responds with status code `200`. * If Cocoon fails to process the request, it responds with a non-retryable error status code, such as `400` for a validation error, `404` if the `uid` is not found, `409` if a terminal status was already processed, or `413` if the body size exceeds 768KB (to stay within Firestore's 1 MiB document size limit). -2. Once Cocoon receives the partner test execution status for `sha=wxyz` (`uid=abcd`) from Pub/Sub, it updates the partner task state in Firestore. +2. Once Cocoon receives the partner test execution status for `sha=wxyz` (`uid=abcd`) from Pub/Sub, it executes an atomic Firestore transaction that re-verifies `partner_ci.processed == false` before setting `partner_ci.processed = true` and updating the partner task state in `tasks` (idempotently discarding and acknowledging duplicate concurrent messages). ```mermaid sequenceDiagram @@ -304,7 +310,7 @@ By leveraging decentralized asymmetric cryptography, this approach intentionally #### 4.1.2. Key Distribution Strategy: Public Key Endpoints -For this authentication model to work, both parties MUST reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of the JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to cache and refresh this key set periodically. On a `kid` cache miss, the verifier MAY trigger an on-demand JWKS refresh subject to a per-partner cooldown (e.g., at most once every 5 minutes) and a strict HTTP timeout to prevent cache-miss DoS amplification. This allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. +For this authentication model to work, both parties MUST reliably obtain public keys. Each system exposes a dedicated, unauthenticated public JWKS endpoint. See [RFC 7517: JSON Web Key (JWK)](https://www.rfc-editor.org/info/rfc7517) for details of the JWKS format. This endpoint serves the public key(s) and key ID(s) (`kid`). Both parties configure their systems to cache and refresh this key set periodically. This allows Cocoon and Partners to rotate their keys seamlessly without manual intervention. Example of JWKS: @@ -322,7 +328,7 @@ For manual on-demand status queries (`GET /status?uid=`), Cocoon passes the The JWT consists of a base64url-encoded header, payload, and signature separated by dots. For example: -`eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6IjEyMzQ1In0.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovL3BhcnRuZXItY2kuZXhhbXBsZS5jb20vd2ViaG9vayIsImlhdCI6MTc3NDk2MjAwMCwiZXhwIjoxNzc0OTYyMzAwLCJkYXRhIjp7InVpZCI6IjBzQXBMaHdibUd6Mlkwd0JEelBMIiwic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` +`eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCIsImtpZCI6IjEyMzQ1In0.eyJpc3MiOiJjb2Nvb24iLCJhdWQiOiJodHRwczovL3BhcnRuZXItY2kuZXhhbXBsZS5jb20vd2ViaG9vayIsImlhdCI6MTc3NDk2MjAwMCwiZXhwIjoxNzc0OTYyMzAwLCJkYXRhIjp7InVpZCI6IjBzQXBMaHdibUd6Mlkwd0JEelBMIiwic2x1ZyI6ImZsdXR0ZXIvZmx1dHRlciIsImJyYW5jaCI6Im1hc3RlciIsInByIjoxMjM0NSwic2hhIjoiNmYwZjFjMmQzZTRmNWE2YjdjOGQ5ZTBmMWEyYjNjNGQ1ZTZmN2E4YiJ9fQ.GnIgnoVR7LeMbs4we6Q39Aw7S7f994Ng_TJomRuROoqiN3HUd8o5BqwSzoeo94jkHYpTuAdCxO60fx6G69zVCA` #### 4.1.5. JWT Header @@ -355,6 +361,7 @@ Tokens are generated and signed per HTTP request attempt with a short-lived expi "data": { "uid": "0sAtLhwbmGz2Y0WBDzPL", "slug": "flutter/flutter", + "branch": "master", "pr": 12345, "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b" } @@ -376,11 +383,19 @@ Webhook listeners MUST validate incoming requests in accordance with standard JW In addition, Cocoon's callback endpoint validates that `data.uid` exists, has not processed a terminal status yet, and belongs to a partner that is listed in the `iss` claim. +#### 4.1.9. Outbound Request Security & SSRF Prevention + +Because Cocoon makes outbound HTTP requests to partner-supplied endpoints and renders partner-supplied log URLs on the public Flutter build dashboard, Cocoon MUST enforce the following security controls: + +* **HTTPS-Only Scheme Validation:** All URLs configured in `partner_config` (`webhook_url`, `status_url`, `keys_url`) and all `details[].url` entries submitted in status callbacks MUST use the `https://` scheme. Plaintext `http://`, `javascript:`, `data:`, and other non-HTTPS schemes MUST be rejected with `400 Bad Request`. +* **SSRF Protection:** Cocoon's outbound HTTP client MUST validate resolved IP addresses upon DNS resolution and block connections to loopback (`127.0.0.0/8`, `::1/128`), private (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), and link-local (`169.254.0.0/16`, `fe80::/10`, including the GCP metadata server `169.254.169.254`) address ranges, and MUST NOT follow HTTP redirects to disallowed hosts or schemes. +* **Outbound Response Size & Timeout Limits:** Outbound HTTP responses from partner endpoints MUST be capped at 64 KB for webhook acknowledgments and JWKS responses (with a 30-second timeout for webhooks and 3-second timeout for JWKS) and 768 KB for manual status query responses (with a 30-second timeout). + ### 4.2. API Contracts & Status State Machine See [Section 8 (Appendix A)](#8-appendix-a-usage-examples) for full HTTP wire examples of each endpoint. -* **Outbound Webhook Dispatch (`POST `):** Dispatched by Cocoon with `data` claims `{uid, slug, pr, sha}`. +* **Outbound Webhook Dispatch (`POST `):** Dispatched by Cocoon with `data` claims `{uid, slug, branch, pr, sha}`. * **Inbound Status Callback (`POST /api/partner/callback`):** Called by Partner CI with `data` claims `{uid, status, message, details}`. Maximum request body size is 768KB. * **Manual On-Demand Status Query (`GET ?uid=`):** Called by Cocoon with `Authorization: Bearer ` when a user manually requests status reconciliation from the dashboard; returns a JSON object `{status, message, details}`. @@ -390,7 +405,7 @@ See [Section 8 (Appendix A)](#8-appendix-a-usage-examples) for full HTTP wire ex * **Terminal Failure States:** `cancelled`, `infra_failure`, `failed`. * **Terminal Success States:** `succeeded`, `neutral`, `skipped`. -Once a task transitions to any terminal state, `partner_ci.processed` is set to `true` and subsequent callbacks for the same `uid` are rejected with `409 Conflict`. +Once a task transitions to any terminal state, the `partner_ci_status_sub` handler atomically sets `partner_ci.processed = true` and updates `tasks` inside a single Firestore transaction (rejecting or discarding concurrent duplicate deliveries), and subsequent HTTP callbacks for the same `uid` are rejected with `409 Conflict`. ### 4.3. Data Model (Firestore) @@ -401,15 +416,17 @@ To support webhook flows, the following Firestore collections and schema extensi For partner configuration, add a `partner_config` collection with Document ID equal to `task_name`: ```go -slug string // github repo (e.g. flutter/flutter or flutter/packages) to trigger on -task_name string // Name of post submit task to display on dashboard -partner string // 3p partner name -webhook_url string // Url to call to request CI -status_url string // Manual status request url -keys_url string // Url to read partner key id to public key mapping +slug string // github repo (e.g. flutter/flutter) to trigger on +branches []string // target branch patterns (e.g. ["master", "beta", "stable"]) to trigger on +task_name string // Name of post submit task to display on dashboard +partner string // 3p partner name (must match JWT iss claim on callbacks) +enabled bool // Whether this partner integration is active (emergency kill-switch) +webhook_url string // HTTPS url to call to request CI +status_url string // Optional manual HTTPS status request url +keys_url string // HTTPS url to read partner key id to public key mapping (JWKS) ``` -Currently, there will be no UI for editing this table. For every post-submit event, dashboard checks will be extended with tasks from this list. All tasks from this list will be created suppressed without the ability to unsuppress. Later, if a decision is made to let some 1P partner CI block the tree, we may extend this table with a `block_tree` boolean field. +Currently, there will be no UI for editing this collection. For every post-submit event, dashboard checks will be extended with active (`enabled == true`) tasks from this collection matching the event's `slug` and `branch`. All tasks from this collection will be created suppressed without the ability to unsuppress. Later, if a decision is made to let some 1P partner CI block the tree, the `partner_config` collection MAY be extended with a `block_tree` boolean field. #### 4.3.2. partner_ci Collection @@ -418,6 +435,7 @@ Add a `partner_ci` collection with Auto-ID Document ID equal to the `uid` of the ```go commit_sha string // Head sha at a moment of check-run creation ie commit sha slug string // repository_owner/repository_name (e.g. flutter/flutter) +branch string // Target branch name (e.g. master) partner string // 3p partner name task_name string // Name of post submit task to display on dashboard attempt_number int // Number of re-run attempt @@ -432,11 +450,16 @@ Extend the `tasks` collection with `message` and `details` fields. #### 4.4.1. partner_ci_notification Topic -To guarantee that Cocoon can notify each partner CI even during network outages or temporary partner infrastructure outages, once Cocoon reads the post-submit task configuration for a post-submit event, it publishes a message to the `partner_ci_notification` topic for every partner integration. A separate instance of Cocoon receives that message from `partner_ci_notification_sub` and attempts to POST it to the partner webhook using Pub/Sub exponential backoff on retryable failures (`5xx`, `408`, `429`, or network timeouts). +To guarantee that Cocoon can notify each partner CI even during network outages or temporary partner infrastructure outages, once Cocoon reads the active post-submit task configurations for a post-submit event, it publishes a message to the `partner_ci_notification` topic for every matching partner integration. Cocoon consumes messages from `partner_ci_notification_sub` via a **GCP Pub/Sub Push subscription** (invoking an internal Cocoon push handler endpoint) and attempts to `POST` each notification to the partner webhook (with a 30-second HTTP timeout) using Pub/Sub exponential backoff on retryable failures (`5xx`, `408`, `429`, or network timeouts). + +To prevent head-of-line blocking ("noisy neighbor" exhaustion) when a single partner endpoint is degraded or timing out: + +* Cocoon enforces a per-partner circuit breaker / concurrency cap so a failing partner endpoint cannot exhaust push worker capacity for healthy partners. +* `partner_ci_notification_sub` is configured with exponential backoff (minimum 60 seconds, maximum 600 seconds), `max_delivery_attempts = 10`, and a **Dead-Letter Topic (`partner_ci_notification_dlq`)**. If a notification exhausts all delivery attempts and is routed to `partner_ci_notification_dlq`, a dead-letter handler marks the corresponding task in Firestore as `infra_failure` with a summary message indicating webhook delivery failure. #### 4.4.2. partner_ci_status Topic -To guarantee that Cocoon can reliably process status updates once the status webhook is called by a partner and the payload is successfully validated, the payload is published to the `partner_ci_status` topic and consumed by Cocoon via `partner_ci_status_sub` to update the corresponding task status in Firestore. +To guarantee that Cocoon can reliably process status updates once the status callback webhook (`/api/partner/callback`) is called by a partner and the payload is successfully validated, the payload is published to the `partner_ci_status` topic and consumed by Cocoon via a **GCP Pub/Sub Push subscription (`partner_ci_status_sub`)**. The push handler executes an idempotent Firestore transaction to update `partner_ci` and the corresponding task status in `tasks`. Unrecoverable messages after `max_delivery_attempts = 10` are routed to a **Dead-Letter Topic (`partner_ci_status_dlq`)** for alerting and inspection. --- @@ -563,11 +586,32 @@ sequenceDiagram ### 6.1. Testing Plan -* **Unit Tests:** Validate Ed25519 JWT generation and signature verification, JWKS `kid` lookup and cache-miss refresh logic, payload schema validation, and task status state transitions (including idempotency and out-of-order `in_progress` vs. terminal status handling). -* **Integration Tests:** Verify Firestore (`partner_config`, `partner_ci`, `tasks`) and Pub/Sub (`partner_ci_notification`, `partner_ci_status`) message publishing, acknowledgment, and retry behavior. +* **Unit Tests:** Validate Ed25519 JWT generation and signature verification, JWKS `kid` lookup and cache-miss refresh cooldown logic, SSRF/HTTPS URL validation, payload schema validation, and task status state transitions (including idempotency and out-of-order `in_progress` vs. terminal status handling). +* **Integration Tests:** Verify Firestore (`partner_config`, `partner_ci`, `tasks`) atomic transactions and Pub/Sub (`partner_ci_notification`, `partner_ci_status`, and dead-letter topics) message publishing, acknowledgment, and retry behavior. * **End-to-End Tests:** Implement a dummy partner CI service to test the complete post-submit dispatch, `in_progress` callback, manual status reconciliation, terminal callback, and task re-run workflows. -### 6.2. Partner Onboarding & Documentation Plan +### 6.2. Rollout & Rollback Plan + +* **Phased Rollout:** + 1. **Dummy partner CI validation:** Deploy the Pub/Sub topics, Firestore schema changes, and Cocoon endpoints using the dummy partner CI service. + 2. **Pilot Onboarding:** Onboard a single first-party or pilot partner in production (`enabled = true` in `partner_config`) and monitor webhook delivery latency, callback success rates, and dashboard rendering for 2 weeks. + 3. **General Availability:** Gradually onboard additional partners by adding documents to `partner_config`. +* **Emergency Kill-Switch & Rollback:** + * Setting `enabled = false` on any document in `partner_config` immediately stops Cocoon from scheduling new tasks or dispatching webhooks for that partner without requiring a code deployment or deleting configuration state. + * Because all partner tasks are non-blocking (created suppressed), disabling or rolling back the feature has zero impact on Flutter tree health or PR merge velocity. + +### 6.3. Observability, Privacy & Data Retention + +* **Metrics & Alerting:** Cocoon will emit Cloud Monitoring metrics partitioned by `partner` and `task_name`: + * Outbound webhook dispatch count, HTTP status code distribution, and latency. + * Inbound callback request count, HTTP status code distribution, and JWT verification failure count. + * On-demand JWKS refresh count, latency, and failure rate. + * Dead-letter queue (`partner_ci_notification_dlq` and `partner_ci_status_dlq`) message counts. +* **Operational Ownership:** Flutter EngProd monitors Cocoon pipeline health and Pub/Sub DLQs; individual partner teams are responsible for monitoring and triaging their own CI infrastructure and test failures. +* **Privacy & Data Retention:** + * Because the Flutter build dashboard is public, partner callback payloads (`message` and `details[].url`) MUST NOT contain Personally Identifiable Information (PII), internal hostnames, or confidential partner secrets. + +### 6.4. Partner Onboarding & Documentation Plan Provide partner integration documentation detailing: @@ -580,9 +624,10 @@ Provide partner integration documentation detailing: ## 7. Open Questions -* What other fields should we provide to partners webhook to request CI run? -* Can `tasks` collection PK `__` be used as `uid`? In that case we would not need a `partner_ci` collection. +* What other fields should Cocoon provide to the partner webhook to request a CI run? +* Can the `tasks` collection primary key `__` be used as `uid`? In that case Cocoon would not need a separate `partner_ci` collection. * How should Cocoon handle partner tasks that never receive a terminal status callback (e.g., marking tasks stuck in `New` or `in_progress` as `infra_failure` after a configurable timeout)? +* Do we need to provide branch name (e.g. stable or master) to the partner webhook? Or just the SHA and task name? Technically the branch name could be derived from the sha. --- @@ -618,6 +663,7 @@ See: [RFC 7515: JSON Web Signature (JWS)](https://www.rfc-editor.org/info/rfc751 "data": { "uid": "0sAtLhwbmGz2Y0WBDzPL", "slug": "flutter/flutter", + "branch": "master", "pr": 12345, "sha": "6f0f1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b" } @@ -633,6 +679,7 @@ Where: * **data**: Object containing custom payload claims and event details: * **uid**: The unique correlation identifier to match asynchronous CI execution request to future status response. * **slug**: The target repository identifier (e.g., `flutter/flutter`). + * **branch**: The target git branch name (e.g., `master`). * **pr**: The Pull Request number associated with the event. * **sha**: The target git commit SHA hash. From 7bd7c1dde4cf7ccb2a8f61fdf716eb77a834434c Mon Sep 17 00:00:00 2001 From: "Dmitry Grand (dmgr)" Date: Fri, 16 Oct 2026 22:08:06 -0700 Subject: [PATCH 35/35] fixed markdown lint --- rfc/610.0000-flutter-pluggable-ci.md | 1 - 1 file changed, 1 deletion(-) diff --git a/rfc/610.0000-flutter-pluggable-ci.md b/rfc/610.0000-flutter-pluggable-ci.md index 1c328a7..d66fde8 100644 --- a/rfc/610.0000-flutter-pluggable-ci.md +++ b/rfc/610.0000-flutter-pluggable-ci.md @@ -53,7 +53,6 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S * **LUCI (Layered Universal Continuous Integration):** The underlying execution engine that runs Flutter's testing and build workloads across diverse platforms like macOS, Windows, and Linux. Orchestrated by Cocoon, LUCI executes specific Python-based "recipes" to securely validate code changes for the Flutter framework, engine, and packages during both pre-submit and post-submit pipelines. * **Post-submit Event:** An event in Cocoon that is triggered after a PR is merged into the `flutter/flutter` repository. This may be a merge to `master`, `beta`, `stable` or release candidate branches. - --- ## 2. Goals & Non-Goals