[DX-4553] Optimize Integration Tests Workflow by kalverra · Pull Request #23164 · smartcontractkit/chainlink

kalverra · 2026-07-19T01:14:00Z

Integration Tests Simplification and Optimization

Integration Tests is our primary bottleneck in CI for PRs. This PR addresses some of the structural issues holding it back, along with smaller optimizations to improve speed and cost.

Big Changes

Run go test code compilation in parallel with Docker build. Cache pre-built binaries
Rip out the CCIP tests' reliance on the old run-e2e-tests monolith action. It's now rebuilt in a more minimal fashion.
Fix the CCIP v1.6 tests' execution decisions
- Removed some default skipped tests
- Only run on Merge Queue, and when CCIP changes are made

Small Changes

Expanded AI skill to cover right-sizing runners and conduct overall structural change trials
Some caching and build optimizations
Tweak runner sizes for optimal speed + cost

Results

More maintainable flow for CCIP v1.6 tests
Faster execution
Opens the doors for more improvements

	Speed (mins)	Cost
Before	14.92	$1.6827
After	11.78	$0.9161
Abs Δ	-3.14	-$0.7666
% Δ	-21.04%	-45.5%

Before Shape

After Shape

github-actions · 2026-07-19T01:15:18Z

✅ No conflicts with other open PRs targeting develop

trunk-io · 2026-07-19T01:23:48Z

_{View Full Report ↗︎ ⋅ Docs}

github-actions · 2026-07-19T02:22:36Z

CORA - Analysis Skipped

Reason: The number of code owners (4) is less than the minimum required (5) and/or the number of CODEOWNERS entries with changed files (3) is less than the minimum required (2).

Copilot

Pull request overview

Risk Rating: MEDIUM (significant CI/workflow orchestration changes that can block integration coverage if misconfigured)

This PR aims to reduce CI runtime/cost for integration/system tests by parallelizing CCIP reorg tests, adding a host-side compilation step with Go build caching, and enabling reuse of a prebuilt chainlink binary in Docker builds.

Changes:

Add USE_PREBUILT support in Chainlink Dockerfiles to copy a host-built ./bin/chainlink into the image instead of rebuilding.
Restructure integration test workflows: add a compile-tests job, adjust runner sizing, and update path-based change detection (including a new ccip-changes output).
Split CCIP reorg tests into per-lane matrix entries and tune runner resources in .github/e2e-tests.yml.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
plugins/chainlink.Dockerfile	Adds `USE_PREBUILT` build arg to optionally copy a prebuilt binary into the image.
core/chainlink.Dockerfile	Same `USE_PREBUILT` optimization for the core image build.
.gitignore	Generalizes trial-log ignores and re-organizes ignored AI agent settings files.
.github/workflows/run-nightly-in-memory-integration-tests.yml	Switches reusable workflow reference to `run-e2e-tests/v1`.
.github/workflows/integration-tests.yml	Adds `ccip-changes`, introduces `compile-tests`, updates runner sizing, and changes job dependency structure.
.github/workflows/integration-in-memory-tests.yml	Switches reusable workflow reference to `run-e2e-tests/v1`.
.github/workflows/cre-system-tests.yaml	Adds per-test runner sizing and migrates Go setup to `ctf-setup-go` with build caching.
.github/workflows/cre-regression-system-tests.yaml	Same as above for CRE regression tests.
.github/workflows/ccip-chaos-tests.yml	Switches reusable workflow reference to `run-e2e-tests/v1`.
.github/e2e-tests.yml	Splits CCIP reorg tests into OnSource/OnDest entries and adjusts runner sizing/triggers.
.github/actions/setup-go/action.yml	Bumps `actions/setup-go` usage from v6 to v7.
.github/.agents/skills/right-size-runners/SKILL.md	Removes the old skill definition.
.github/.agents/skills/right-size-runners/scripts/test_workflow_compare.py	Removes old test file (skill migration/cleanup).
.github/.agents/skills/optimize-workflow/SKILL.md	Adds new workflow-optimization skill documentation.
.github/.agents/skills/optimize-workflow/scripts/workflow_monitor.py	Updates monitor script to write structured per-trial output (json/md/logs) and adds runtime/outlier analysis.
.github/.agents/skills/optimize-workflow/scripts/workflow_compare.py	Updates compare script to resolve trials by name/path and write comparison output under the workflow’s trials directory.
.github/.agents/skills/optimize-workflow/scripts/test_workflow_monitor.py	Extends tests for new monitor behavior/output layout and helper functions.
.github/.agents/skills/optimize-workflow/scripts/test_workflow_compare.py	Adds tests for new trial resolution and comparison output behavior.

Comments suppressed due to low confidence (1)

.github/e2e-tests.yml:19

These newly split CCIP reorg entries don’t include the Merge Queue E2E CCIP v1.6 Tests trigger. Since integration-tests.yml sets test_trigger to Merge Queue E2E CCIP v1.6 Tests for merge_group, these tests will never run in the merge queue (only Nightly/Push/Dispatch).

If the intent is to keep reorg coverage in merge queue, add that trigger to these entries (and the other split reorg entries).

    triggers:
      - Nightly E2E Tests
      - Push E2E CCIP v1.6 Tests
      - Workflow Dispatch E2E CCIP v1.6 Tests

erikburt

LGTM - happy to approve if you think my comment on caching is an unnecessary optimization / adds more complexity than it's worth.

- Replace legacy .github/e2e-tests.yml + ctf-run-tests with ccip-system-tests.yaml - Static 8-test matrix, precompiled binaries, direct gotestsum execution - Use QA_AWS_ROLE_TO_ASSUME for ECR pull access - Use precompiled cre-env binary in chip_ingress_stack_provider + before_suite - Bump chip ingress timeout 2m -> 5m - Drop citool, k8s/in-memory paths, slack, GAP, otel, coverage, pyroscope - Delete .github/e2e-tests.yml

…optimizeIntegrationTests

Copilot

Pull request overview

Copilot reviewed 26 out of 29 changed files in this pull request and generated 4 comments.

cl-sonarqube-production · 2026-07-23T18:34:32Z

Quality Gate failed

Failed conditions
7 Security Hotspots
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Copilot

Pull request overview

Copilot reviewed 26 out of 29 changed files in this pull request and generated 3 comments.

+	binaryPath := filepath.Join(relativePathToRepoRoot, "system-tests", "tests", "bin", creEnvBinaryName)
+	if _, err := os.Stat(binaryPath); err == nil {
+		framework.L.Info().Str("binary", binaryPath).Msg("Using precompiled cre-env binary")
+		cmd := exec.CommandContext(ctx, binaryPath, args...)
+		cmd.Dir = environmentDir
+		return cmd
+	}


+              wait "$p1"; p1_exit=$?
+              wait "$p2"; p2_exit=$?
+              wait "$p3"; p3_exit=$?
+              if [[ "$p1_exit" -ne 0 || "$p2_exit" -ne 0 || "$p3_exit" -ne 0 ]]; then
+                echo "Compilation failed (cre-smoke=$p1_exit, cre-regression=$p2_exit, cre-env=$p3_exit)" >&2
+                exit 1
+              fi


+			if tt.createBinary {
+				binDir := filepath.Join(tmpDir, "system-tests", "tests", "bin")
+				require.NoError(t, os.MkdirAll(binDir, 0700))
+				binPath := filepath.Join(binDir, "cre-env")
+				require.NoError(t, os.WriteFile(binPath, []byte("#!/bin/sh\necho ok"), 0600))
+			}
+
+			cmd := resolveCreEnvCommand(ctx, tmpDir, environmentDir, tt.inputArgs...)
+			require.NotNil(t, cmd)
+
+			assert.Equal(t, environmentDir, cmd.Dir)
+			if tt.createBinary {
+				assert.True(t, filepath.IsAbs(cmd.Path) || filepath.Base(cmd.Path) == "cre-env", "path should be cre-env binary: %s", cmd.Path)
+			} else {
+				assert.Equal(t, "go", filepath.Base(cmd.Path))
+			}


kalverra marked this pull request as ready for review July 21, 2026 20:15

kalverra requested review from a team as code owners July 21, 2026 20:15

kalverra enabled auto-merge July 21, 2026 20:15

kalverra requested a review from Copilot July 21, 2026 20:15

product-security-plaid-production Bot requested review from Tofel, patrickhuie19, russell-stern, skudasov and tvc-robsondebraga and removed request for patrickhuie19 July 21, 2026 20:15

Copilot started reviewing on behalf of kalverra July 21, 2026 20:16 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread .github/workflows/integration-tests.yml Outdated

Comment thread .github/e2e-tests.yml Outdated

erikburt reviewed Jul 21, 2026

View reviewed changes

Comment thread .github/workflows/integration-tests.yml Outdated

Comment thread .github/workflows/cre-regression-system-tests.yaml

erikburt previously approved these changes Jul 21, 2026

View reviewed changes

kalverra dismissed erikburt’s stale review via 85d08d5 July 22, 2026 00:18

Tofel reviewed Jul 22, 2026

View reviewed changes

Comment thread .github/workflows/cre-regression-system-tests.yaml Outdated

Tofel reviewed Jul 22, 2026

View reviewed changes

Comment thread .github/workflows/cre-system-tests.yaml Outdated

kalverra force-pushed the optimizeIntegrationTests branch from 85d08d5 to 2e2e143 Compare July 23, 2026 00:42

kalverra requested review from a team as code owners July 23, 2026 00:42

Merge branch 'develop' of github.com:smartcontractkit/chainlink into …

988d1ea

…optimizeIntegrationTests

kalverra added 5 commits July 22, 2026 21:38

More memory

11b14a9

lint

1444c58

All default

36a5aaa

Simplify and consolidate

71d3dc7

Bigger compiler runner

ba08b34

kalverra requested a review from Copilot July 23, 2026 16:43

Copilot started reviewing on behalf of kalverra July 23, 2026 16:43 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread system-tests/tests/test-helpers/before_suite.go Outdated

Comment thread .github/workflows/cre-system-tests.yaml

Comment thread .github/workflows/cre-regression-system-tests.yaml

Comment thread .github/workflows/ccip-system-tests.yaml

kalverra added 2 commits July 23, 2026 13:07

Review comments

b3851e5

lint

a0432cf

kalverra requested review from Tofel, chainchad and erikburt July 23, 2026 18:13

cleanup

28dab83

kalverra requested a review from Copilot July 23, 2026 20:51

Copilot started reviewing on behalf of kalverra July 23, 2026 20:52 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Uh oh!

Conversation

kalverra commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration Tests Simplification and Optimization

Big Changes

Small Changes

Results

Before Shape

After Shape

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

trunk-io Bot commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CORA - Analysis Skipped

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

erikburt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cl-sonarqube-production Bot commented Jul 23, 2026

Quality Gate failed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kalverra commented Jul 19, 2026 •

edited

Loading

trunk-io Bot commented Jul 19, 2026 •

edited

Loading

github-actions Bot commented Jul 19, 2026 •

edited

Loading