Skip to content

Retry transient ARM failures during cert-management provisioning - #454

Merged
Ewerton Scaboro da Silva (ewertons) merged 1 commit into
masterfrom
fix/adr-transient-retries
Sep 30, 2026
Merged

Ewerton Scaboro da Silva (ewertons) merged 1 commit into
masterfrom
fix/adr-transient-retries

Conversation

@ewertons

Copy link
Copy Markdown
Contributor

Cert-management provisioning fails on transient ARM errors that later succeed. Two failures in one ci-c-e2e-csr run (Azure/azure-iot-sdk):

  • DPS create PUT: Internal Server Error / GatewayAuthenticationFailed from the ARM gateway. Not retried, and not matched by ArmTransientPattern (InternalServerError only).
  • Link re-submit after dps-1=Failed (LinkableResourceNotReady): the namespace was still Accepted, so the PUT got 409 ResourceProvisioningInProgress, which was not retryable.

Changes

  • Hub and DPS create PUTs are wrapped in Invoke-WithRetry on ArmTransientPattern (PUT is idempotent).
  • ArmTransientPattern adds Internal ?Server ?Error, Bad ?Gateway, GatewayAuthenticationFailed.
  • Connect-AdrNamespace: before each re-submit, wait up to 300 s for a terminal namespace provisioningState (skipped if the namespace cannot be read); ResourceProvisioningInProgress is retryable on the link PUT.

Validation

  • tests/Validate-Module.ps1: OK.
  • Flow test with az stubbed (17 checks): pattern matches the CI error text and not 400/401/409; re-submit never happens while Accepted; 409 retried; permanent rejection still fails at once; DPS create retried once then succeeds. On master the settle/409/pattern checks fail and the create case aborts the run.
  • Merges cleanly with Say when the ADR link timed out because the namespace could not be read #452.

- Hub and DPS create PUTs retry on transient ARM errors.
- ArmTransientPattern also matches "Internal Server Error", "Bad Gateway"
  and GatewayAuthenticationFailed (a 500 from the ARM-to-RP gateway).
- Before re-submitting a failed link, wait up to 300 s for the namespace
  to reach a terminal provisioningState; a link PUT answered with 409
  ResourceProvisioningInProgress is retried.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The retry behavior is narrowly scoped, preserves permanent-failure handling, and addresses the documented transient ARM cases.

Review effort: Balanced
Findings: None

What changed in this PR

Adds resilience to certificate-management provisioning when Azure Resource Manager returns transient failures.

Changes:

  • Retries transient IoT Hub and DPS creation failures.
  • Expands transient ARM error matching.
  • Waits for ADR namespace settlement and retries link conflicts.
File Description
scripts/​AzIotSdkTest/​parts/​Provisioning.ps1 Retries transient hub and DPS create failures.
scripts/​AzIotSdkTest/​parts/​Dps.ps1 Handles additional transient errors and namespace link conflicts.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@ewertons
Ewerton Scaboro da Silva (ewertons) merged commit 12e4e67 into master Sep 30, 2026
6 checks passed
Ewerton Scaboro da Silva (ewertons) added a commit to Azure/azure-iot-sdk that referenced this pull request Sep 30, 2026
…y fix

Moves ci-c-e2e-csr's provision/destroy actions to Azure/iot-sdks-e2e-fx#454,
which retries transient ARM failures (gateway errors on hub/DPS create;
409 ResourceProvisioningInProgress on ADR link re-submit).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ewerton Scaboro da Silva (ewertons) added a commit to Azure/azure-iot-sdk that referenced this pull request Oct 2, 2026
Pins e2e provisioning in `ci-c-e2e`, `ci-c-e2e-csr` and `ci-dotnet` to
Azure/iot-sdks-e2e-fx **v1.0.1** (`b96d2aa`), replacing v1.0.0.

Since v1.0.0:
- Azure/iot-sdks-e2e-fx#454: retries transient ARM failures during
certificate-management provisioning (gateway errors on hub/DPS create;
`409 ResourceProvisioningInProgress` on ADR link re-submit). Fixes the
`setup (provision)` failures in `ci-c-e2e-csr` run 36681736303.
- Azure/iot-sdks-e2e-fx#455: sets file upload at hub creation when
certificate management is on (needed by `ci-dotnet` with certificate
management, #297).
- Azure/iot-sdks-e2e-fx#453: default location `eastus2euap` (already set
explicitly here).

**Validation**
- actionlint 1.7.12: clean. zizmor 1.30.1 (online): no findings; a wrong
version comment is reported, so `ref-version-mismatch` is checked.
- v1.0.1 commit: iot-sdks-e2e-fx module checks and `horton-gate-build`
pass.
- #454 content verified end to end earlier (runs 36683896318,
36689775583).
- PR runs of `ci-c-e2e` / `ci-c-e2e-csr` use the shared environments and
skip provisioning; full-provisioning runs are dispatched on this branch
for both. `ci-dotnet` provisions in its PR run.

Co-authored-by: ewertons <12401690+ewertons@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants