Move the IoT Edge runtime images from the EOL 1.4 line to 1.5 - #435
Ewerton Scaboro da Silva (ewertons) wants to merge 3 commits into
Conversation
The horton deployments pin edgeHub and edgeAgent to
mcr.microsoft.com/azureiotedge-{hub,agent}:1.4, which resolves to 1.4.43.
That was the final 1.4 release, published 2024-10-10, and the 1.4 line is
out of support. The current lines are 1.6.0 LTS (2026-07-16) and 1.5.43
LTS (2026-07-21).
Worse, scripts/new/install-iotedge.sh installs the daemon unpinned
(apt-get install -y aziot-edge), which today gives aziot-edge 1.6.0-1. So
every gate run has been pairing a 1.6.0 daemon with 1.4.43 runtime
modules, two LTS lines apart. Moving the images to :1.6 lines them up
with the daemon we already install.
This surfaced while investigating flaky horton-java-gate failures where
twin operations against edgeHub never receive a response and the client
times out after 60 seconds. edgeHub 1.4.43 logs its own
IotHubCommunicationException "Amqp resource is disconnected" inside
CloudProxy.SetupDesiredPropertyUpdatesAsync just before the stall, and in
the MQTT case applies a reported properties patch without ever delivering
the response. That may or may not still reproduce on 1.6, but there is no
point reporting it against an out-of-support build.
Also re-enable the IOTHUB_E2E_EDGE_PRIVATE_AGENTIMAGE and
IOTHUB_E2E_EDGE_PRIVATE_HUBIMAGE overrides. They are documented in
test-runner/README.md but were disabled with "if (False and ...)", so the
documented behaviour did not work. With them live, a consuming pipeline
can pin or roll back the image itself without another change here. The
default is unchanged when the variables are unset.
The image_edgeHub and image_edgeAgent parameters in
vsts/templates/steps-pre-test.yaml are moved to 1.6 as well. They are
dead today, nothing references them, but leaving them saying 1.4 would be
misleading.
There was a problem hiding this comment.
Pull request overview
Updates Horton’s IoT Edge runtime from unsupported 1.4 images to supported 1.6 images and restores documented image overrides.
Changes:
- Updates default edgeAgent and edgeHub images to 1.6.
- Re-enables environment-variable image overrides.
- Aligns pipeline template defaults with runtime configuration.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
vsts/templates/steps-pre-test.yaml |
Updates template image defaults to 1.6. |
bin/deploy/edge_configuration.py |
Updates runtime defaults and enables image overrides. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The first attempt at this used :1.6, to match the aziot-edge 1.6.0 daemon
that install-iotedge.sh already installs unpinned. The gate run on that
(build 161703) failed deterministically: all six edgehub_module jobs, in
every language, failed the same four tests.
test_method_call_invoked_on_friend FAILED
test_inputoutput_module_to_friend_routing FAILED
test_inputoutput_friend_to_module_routing ERROR
test_inputoutput_module_test_to_friend_and_back ERROR
All four involve friendMod. The cause is that friendMod's ModuleClient
cannot connect to edgeHub 1.6 over mqtts:
18:28:22.707 MqttBase uri: mqtts://runnervmtroe5
18:29:22.717 UnauthorizedError: mqtt.js returned Failure on first
connection (Not authorized): Unable to establish a connection
That is a 60 second connect timeout that mqtt.js reports as "Not
authorized", and edgeHub logs nothing at all for it. Things that are ruled
out: the MQTT head is up ("Initializing TLS endpoint on port 8883 for MQTT
head", "Started MQTT head"); 8883 is published identically on 1.4 and 1.6
(0.0.0.0:8883->8883/tcp in both); and edgeHub accepted the deployment
config identically on both ("Set the following 4 route(s) in edge hub").
The same friendMod container connects fine to the same edgeHub over
wss://runnervmtroe5:443 as a leaf device moments earlier, so this is
specific to the 1.6 MQTT head rather than the container, the certificates
or the network.
1.5.43 is the current release of the other supported LTS line and is a
much smaller step from 1.4.43, so it is the better first move. It still
gets us off the EOL line, which is the point of this change.
Whatever 1.6 needs (most likely a rebuild of default-friend-module against
a newer SDK) is worth doing, but it should not block getting off 1.4.
|
Updated to target 1.5 rather than 1.6. Recording why, since the 1.6 attempt is a useful negative result. The gate run on 1.6 (build 161703) failed deterministically -- all six All four involve That is a 60 second connect timeout that mqtt.js surfaces as "Not authorized", and edgeHub logs nothing at all for the attempt. What I ruled out:
So it is specific to the 1.6 MQTT head. My guess is 1.5.43 is the current release of the other supported LTS line and a much smaller step from 1.4.43. It still achieves the point of this PR, which is getting off the EOL 1.4 line. Whatever 1.6 needs is worth doing separately and should not block that. Note this leaves a daemon/runtime skew in the other direction: |
The previous run (161749) was green on all six edgehub_module jobs, which is the point of this change. It failed only on py311_mqtt_iothub_module_and_device_async :: test_twin_desired_props, an iothub scenario that deploys no edge runtime at all and so cannot be affected by the image version. Re-running to confirm it was a flake.
Ewerton Scaboro da Silva (ewertons)
changed the title
|
Closing this as superseded by #436, which is now merged as Both substantive changes here are already on
On the third one, #436 took the other option discussed here. This PR bumps the two parameters to 1.5 so they are not misleading, while noting they are "dead today (declared, never referenced anywhere in the repo)". #436 deleted them instead. They now appear nowhere in the repo, so there is nothing left to keep accurate. Merging this as-is would now be a regression rather than a no-op. Diffed against current
and it re-adds the two dead YAML parameters while replacing the derived tags with hardcoded What is not supersededThe 1.6 analysis in the description above is not captured in #436 and is still the best writeup of that blocker:
That work stays tracked in ADO #39226664, with the suggested next step being a 1.6 run with For anyone picking that up later: |
IOTHUB_E2E_EDGE_RELEASE_LINE has defaulted to 1.5 since #436. 1.6 is the current LTS (1.6.0 daemon, 1.6.1 module images) and is serviced through 2028-11-14; 1.5 is the previous LTS. Move the single knob to 1.6, which moves both the edgeAgent/edgeHub image tags and the pinned host aziot-edge/aziot-identity-service packages together. aziot-edge 1.6.0-1 and aziot-identity-service 1.6.0-1 are both present in the Microsoft feed for ubuntu/22.04 (jammy) and ubuntu/24.04 (noble), which are the only Linux agent images these pipelines use, so the pin in install-iotedge.sh resolves. This retries the 1.6 move that #435 abandoned. Every one of the 16 failures recorded there (build 161703) was the same thing in a different language: friendMod's node wrapper on localhost:8098 stopped answering, producing "too many 500 error responses" on connect2 and then ConnectionRefused for the rest of the suite. That was the missing 'error' listener in the node wrapper, since fixed by #439 and #440 -- the same UnauthorizedError ("mqtt.js returned Failure on first connection (Not authorized)") that killed the wrapper on 1.6 was also observed on 1.5 in build 161989, so it was never a 1.6 regression. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
IOTHUB_E2E_EDGE_RELEASE_LINE has defaulted to 1.5 since #436. 1.6 is the current LTS (1.6.0 daemon, 1.6.1 module images) and is serviced through 2028-11-14; 1.5 is the previous LTS. Move the single knob to 1.6, which moves both the edgeAgent/edgeHub image tags and the pinned host aziot-edge/aziot-identity-service packages together. aziot-edge 1.6.0-1 and aziot-identity-service 1.6.0-1 are both present in the Microsoft feed for ubuntu/22.04 (jammy) and ubuntu/24.04 (noble), which are the only Linux agent images these pipelines use, so the pin in install-iotedge.sh resolves. This retries the 1.6 move that #435 abandoned. Every one of the 16 failures recorded there (build 161703) was the same thing in a different language: friendMod's node wrapper on localhost:8098 stopped answering, producing "too many 500 error responses" on connect2 and then ConnectionRefused for the rest of the suite. That was the missing 'error' listener in the node wrapper, since fixed by #439 and #440 -- the same UnauthorizedError ("mqtt.js returned Failure on first connection (Not authorized)") that killed the wrapper on 1.6 was also observed on 1.5 in build 161989, so it was never a 1.6 regression. Co-authored-by: Copilot CLI <agent@local> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
What
Move the horton IoT Edge runtime images from
:1.4to:1.5, and re-enable the two image override environment variables that are already documented intest-runner/README.md.Why
1.4 is out of support.
:1.4resolves to 1.4.43, the final 1.4 release, published 2024-10-10. The supported lines are 1.5.43 (2026-07-21) and 1.6.0 (2026-07-16).We are also running a mismatched pair today.
scripts/new/install-iotedge.shinstalls the daemon unpinned:sudo apt-get install -y aziot-edge # no version pinwhich currently gives
aziot-edge 1.6.0-1, while the runtime modules were hardcoded to 1.4.43. Confirmed in the gate logs:Why 1.5 and not 1.6
This PR originally targeted 1.6. That does not work, and the reason is worth recording.
On 1.6 (build 161703) all six
edgehub_modulejobs failed deterministically, in every language, on the same four tests -- all of which involvefriendMod:friendMod's ModuleClient cannot reach edgeHub 1.6 over mqtts:A 60 second connect timeout that mqtt.js reports as "Not authorized", with no corresponding edgeHub log entry at all.
Ruled out so far:
Initializing TLS endpoint on port 8883 for MQTT head/Started MQTT head0.0.0.0:8883->8883/tcpSet the following 4 route(s) in edge hubwss://<host>:443as a leaf device seconds earlierSslProtocolsdefault changed fromtls1.2in 1.4 totls1.2,tls1.3)Tls12, Tls13and friendMod is fine; and pinningSslProtocols=tls1.2on 1.6 (build 161754) did not help and made the run worseSo the 1.6 blocker is real but not yet root-caused. It is tracked in ADO #39226664; the next step there is running 1.6 with
RuntimeLogLevel: debugon the edgeHub module to see what the MQTT head does with that connection. 1.5 gets us onto a supported line in the meantime without waiting for that.The override change
IOTHUB_E2E_EDGE_PRIVATE_AGENTIMAGEandIOTHUB_E2E_EDGE_PRIVATE_HUBIMAGEare documented intest-runner/README.md, but the code had them switched off:They now work as documented. Behaviour is unchanged when the variables are unset, and since ADO exports pipeline variables as environment variables to every task, a consuming pipeline can pin or roll back its own edge image without another change here -- useful for testing the 1.6 work, or if one language needs to move at a different pace.
image_edgeHub/image_edgeAgentinvsts/templates/steps-pre-test.yamlmoved to 1.5 too. They are dead today (declared, never referenced anywhere in the repo) but leaving them saying 1.4 would be misleading.Gate status -- please read before merging
The
edgehub_modulejobs, which are what this change affects, are green on 1.5: 6/6 in 161749 and 6/6 again in 161758.The overall gate is nevertheless red on both runs, each time on a different
iothubsuite:py311_mqtt_iothub_module_and_device_asynctest_twin_desired_propsjava_mqtt_iothub_moduletest_twin_reported_props_5_timesBoth are
iothubscenarios, which deploy no edge runtime at all, so they cannot be affected by the image version. Both are the same failure shape: a twin operation against the language wrapper returns HTTP 500 after the wrapper blocks for 60 seconds waiting on the SDK's synchronous twin API.That is a pre-existing flake in the wrappers, not something this PR introduces or fixes -- it is the same problem that led to this investigation in the first place. It does mean this PR cannot go green on its own, and that the gate is unreliable independently of the IoT Edge version. Worth deciding whether to merge on the strength of the
edgehub_moduleresults, or to fix the twin flake first.Blast radius
This changes the edgeHub/edgeAgent version for every language gate -- Java, Python, Node, C, C#. That is deliberate: all of them are on the EOL 1.4 line today. If a specific gate needs to stay behind, it can now set
IOTHUB_E2E_EDGE_PRIVATE_HUBIMAGE/IOTHUB_E2E_EDGE_PRIVATE_AGENTIMAGEin its own pipeline rather than blocking this.Validation
python -m py_compile bin/deploy/edge_configuration.pypasses. Gate evidence as above.