fix(reset): don't hold the reset response on an in-flight EC2 boot - #2677
Merged
Merged
Conversation
A reset tore EC2 instances down under their lifecycle lock, which a boot holds while it pulls the instance image, so a reset right after RunInstances waited for the pull (seconds on a cold CI runner) and clients timed out. An instance whose lifecycle task is in flight is now torn down on a task of its own once that task lets go; the reset returns at once. Also keep the blocking trivy probe off the async workers.
vieiralucas
added a commit
that referenced
this pull request
Oct 4, 2026
After rebasing onto the deferred reset teardown (#2677): the reset test now also checks the IMDS sidecar is removed, the duplicate reset e2e is dropped (ec2_instance_runtime covers it), and fake CLI scripts wait out ETXTBSY before use so parallel tests forking can't fail their first exec.
vieiralucas
added a commit
that referenced
this pull request
Oct 4, 2026
After rebasing onto the deferred reset teardown (#2677): the reset test now also checks the IMDS sidecar is removed, the duplicate reset e2e is dropped (ec2_instance_runtime covers it), and fake CLI scripts wait out ETXTBSY before use so parallel tests forking can't fail their first exec.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
python-testintermittently fails on main (and deterministically on #2666): theclient.reset()right aftertest_ec2_get_instancesgets no response within the SDK's 5s timeout.The cause is not tokio workers blocked by sync calls. The reset handler awaits its container teardown, and EC2's
remove_instancestook each instance's lifecycle lock first. The RunInstances boot task holds that lock for the wholerun_instance, which includes pulling the instance image (amazonlinux:2023by default, seconds on a cold runner). So the reset response waited for the image pull.Ec2Runtime::remove_instancesnow tries the lifecycle lock. When an instance's lifecycle task is in flight, its teardown (container and data volume) runs on a task of its own once the lock frees, and the reset returns at once.tokio::process. The sync detection (detect_container_cli,HostNetworking::detect) runs only in startup constructors. Reset'sstop_allpaths are async. CodeBuild's syncrm -fis a panic-pathDropguard by design.trivy --versionprobe inscan_layers. It now runs viaspawn_blocking.Non-code surfaces: none affected. This is internal reset behavior; docs, SDKs and AGENTS.md are unchanged.
Test plan
ec2::runtime::reset_tests::reset_does_not_wait_for_an_in_flight_boot: with the lifecycle lock held (a boot pulling its image),remove_instancesreturns within 2s, and once the lock is released the deferred teardown removes the container (driven by a fake container CLI). It fails on main.ec2_instance_runtime::reset_right_after_run_instances_is_prompt: reset within 5s right after RunInstances. It is Docker-backed and runs in CI only, because Docker hangs on the local machine; it compiles locally.cargo test -p fakecloud-ec2 -p fakecloud-ecr --libpasses. Workspace clippy (--all-targets -D warnings) and fmt are clean.Summary by cubic
Fixes the reset endpoint so a reset right after
RunInstancesno longer waits on the instance boot, which previously held the response while pulling the image and could cause client timeouts. An EC2 instance with an in-flight lifecycle task is now torn down asynchronously once the task releases its lock; because instance ids are unique per instance, the deferred teardown can't affect a newer instance. Also moves ECR's blockingtrivy --versionprobe intospawn_blockingto keep it off the async request handlers.Written for commit 0739bfc. Summary will update on new commits.