Propose opt-in feature recovery and reconciliation protocol

connection-coordinator/docs/Protocols.md

@@ -374,6 +374,13 @@ to fulfill the connection. Each **Channel** is assigned an owner that is
 responsible for ensuring that any missing features are added to eventually
 converge on a full set of features.
 
+### **Feature Recovery Proposal**
+
+The [draft Feature Recovery and Reconciliation Protocol](FeatureRecovery.md)
+proposes explicit ownership, bounded retries, authoritative failure reporting,
+and safe replacement/cleanup rules. It requires bilateral opt-in and resolution
+of its review gates; it is not an enabled extension of the workflows above.
+
 ## **Migration Workflow \[In Progress\]**
 
 We have the ability to Migrate a Connection from one of the Interconnects

connection-coordinator/parameters/_index.yaml

@@ -82,6 +82,14 @@ parameters:
   x-request-id:
     description: |-
       Unique identifier for the request, used for tracing and idempotency.
+
+      Draft Feature Recovery and Reconciliation profile (docs/FeatureRecovery.md):
+      recovery mutations must supply this existing query parameter after bilateral
+      opt-in. Identical uncertain retries retain it and their resource identity
+      and body. Scope records by calling/receiving provider, operation, and full
+      target identity; reject differing content within that scope with 409.
+      Agree retention and durable retirement fences before enablement; replay
+      must not resurrect deleted resources. A new logical operation uses a new ID.
     in: query
     name: requestId
     required: false

connection-coordinator/paths/connections.yaml

@@ -223,6 +223,13 @@ NotifyConnectionStatus:
       The receiving provider must refresh the specified connection resource
       immediately after receiving this notification to ensure their local state
       is consistent with the server provider.
+
+      Draft opt-in recovery profile (docs/FeatureRecovery.md): also notify on
+      feature lifecycle changes and refresh GetFeature/ListFeatures views, not
+      only the parent. Persist notification intent and retry delivery with bounded
+      backoff and escalation; periodic reconciliation covers lost or reordered
+      notifications. Acknowledgment confirms receipt, not provisioning success.
+      Existing connection verification requirements remain unchanged.
     operationId: NotifyConnectionStatus
     parameters:
     - $ref: "../parameters/_index.yaml#/parameters/x-request-id"

connection-coordinator/paths/features.yaml

@@ -65,6 +65,15 @@ AllFeatures:
       request is in flight, retrying the provisioning flow with new parameters,
       or cleaning up resources and rejecting the customer's provisioning request.
 
+      Draft opt-in recovery profile (docs/FeatureRecovery.md): reconcile unknown
+      outcomes with GetFeature and ListFeatures before retrying. Matching or
+      one-sided generations retain their existing identity and configuration;
+      neither a timeout nor stalled provisioning authorizes new parameters or
+      deletion. Serialize by logical connection/channel/feature type as well as
+      resource ID. Replacement and cleanup require coordinated authorization and
+      cross-provider fencing. Without bilateral agreement, this profile is not
+      enabled. L3 configuration remains immutable.
+
       Errors:
         409: CONFLICT. Returned when a server detect concurrent updates to the
         resource.

connection-coordinator/schemas/channel.yaml

@@ -12,6 +12,9 @@ Channel:
 
     3. cross_connect_manager_provider_id
 
+     4. featureRecoveryManagerProvider (only for the bilaterally agreed draft
+       Feature Recovery and Reconciliation profile)
+
     All other fields within the channel are populated by the server provider.
   example:
     demarc: demarc
@@ -48,6 +51,19 @@ Channel:
         Format: providers/{provider}
       readOnly: true
       type: string
+    featureRecoveryManagerProvider:
+      description: |-
+        Output only. Draft proposal: URI of the provider coordinating recovery
+        of implicit features on this channel. Optional and exposed only after
+        bilateral agreement to the Feature Recovery and Reconciliation profile
+        in docs/FeatureRecovery.md. Static onboarding must agree the same owner
+        on both sides; absence or disagreement disables new autonomous recovery
+        mutations. Do not infer ownership from MACsec or cross-connect managers.
+        Transfer requires agreed quiescence and fencing, not just a field change.
+
+        Format: providers/{provider}
+      readOnly: true
+      type: string
     aggregateChannelSizeGbps:
       description: Output only. Aggregate channel size in gigabits per second.
       format: int32

connection-coordinator/schemas/connection.yaml

@@ -14,6 +14,11 @@ Connection:
     4. Connections present on only one provider may be deleted after 7 days.
     5. When deleted, the server provider is responsible for invalidating the key
         material associated with the connection to avoid accidental reuse.
+
+    Draft opt-in recovery profile (docs/FeatureRecovery.md): the seven-day
+    permissions above are subject to bilateral reconciliation and authorization;
+    age alone does not permit deleting unknown in-flight or working resources.
+    This stricter cleanup rule requires explicit bilateral agreement before use.
   example:
     name: providers/my-provider/environments/my-environment/interconnects/my-interconnect/connections/my-connection
     adminState: ""

connection-coordinator/schemas/feature.yaml

@@ -32,9 +32,16 @@ Feature:
       description: All configuration associated with the feature.
     provisioningState:
       allOf:
-      - $ref: "common.yaml#/ProvisioningState"
+      - $ref: "#/FeatureProvisioningState"
       description: Output only. The provisioning state of the feature.
       readOnly: true
+    failureInfo:
+      allOf:
+      - $ref: "#/FeatureFailureInfo"
+      description: |-
+        Output only. Draft recovery profile: required when this feature is FAILED
+        and omitted otherwise. Expose only after bilateral profile agreement.
+      readOnly: true
     updatable:
       description: |-
         Output only. Indicates whether the feature supports updates via the
@@ -47,6 +54,50 @@ Feature:
       readOnly: true
   type: object
 
+FeatureProvisioningState:
+  description: |-
+    Feature lifecycle. PENDING and FINAL retain their existing meanings from
+    common.yaml ProvisioningState: PENDING means created with parameters reserved;
+    FINAL means configuration fully propagated and ready for production traffic.
+    Draft proposal: FAILED is terminal for this feature generation at the reporting
+    provider and requires failureInfo. Never emit FAILED without bilateral support
+    for docs/FeatureRecovery.md. Local repair occurs before declaring FAILED;
+    afterward recovery requires coordinated replacement with a new feature ID,
+    not a transition back to PENDING or FINAL. Other resource enums are unchanged.
+  enum:
+  - PROVISIONING_STATE_PENDING
+  - PROVISIONING_STATE_FINAL
+  - PROVISIONING_STATE_FAILED
+  type: string
+
+FeatureFailureInfo:
+  description: |-
+    Draft recovery profile: structured failure of the reporting provider's current
+    feature generation. Contains no secrets, customer configuration, or private
+    diagnostics. Failure code vocabulary requires bilateral agreement.
+  properties:
+    code:
+      description: Stable machine-readable failure code.
+      type: string
+    message:
+      description: Sanitized human-readable failure summary.
+      type: string
+    retryable:
+      description: |-
+        Whether a new coordinated replacement attempt may succeed. This does not
+        authorize automatic replacement, replay, or revival of the failed feature.
+      type: boolean
+    observedAt:
+      description: Time the reporting provider observed the terminal failure.
+      format: date-time
+      type: string
+  required:
+  - code
+  - message
+  - retryable
+  - observedAt
+  type: object
+
 FeatureType:
   description: |-
     The type of feature.