Skip to content

Make 'spot' Configurable Based on Event - #23333

Open
kalverra wants to merge 13 commits into
developfrom
noSpot
Open

Make 'spot' Configurable Based on Event#23333
kalverra wants to merge 13 commits into
developfrom
noSpot

Conversation

@kalverra

@kalverra kalverra commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Spot instances are great for cost savings, and should be used for PRs in most cases, but when used in releases or merge queues, they can seriously hold up development. This gates us to use spot=false when we're merging or pushing a tag.

Big Changes

Small Changes

  • Action version bumps
  • Bump runner families to latest where possible (5+6 -> 7+8) for better cost + performance
  • Reformat runs-on runners to use list yaml approach instead of one-liner where possible. I think it makes it a bit easier to parse.
    runs-on:
      - runs-on=${{ github.run_id }}-fuzz
      - cpu=8
      - ram=32
      - family=m6id+m6idn+m8id+m8idn
      - ${{ needs.filter.outputs.spot_param }}
      - image=ubuntu24-full-x64
      - extras=s3-cache

@kalverra
kalverra marked this pull request as ready for review August 4, 2026 20:45
@kalverra
kalverra requested review from a team as code owners August 4, 2026 20:45
@kalverra
kalverra requested a review from Copilot August 4, 2026 20:45
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Risk Rating: MEDIUM

This PR updates multiple GitHub Actions workflows to prefer spot runners for most runs, while forcing on-demand (non-spot) runners for merge-queue and release-related events/refs to avoid blocking critical pipelines.

Changes:

  • Updated many runs-on / runner-label strings to dynamically set spot=co (default) vs spot=false for merge_group, release branches, release tags, and release events.
  • Updated several jq-generated per-test runner labels to parameterize spot via a bash variable.
  • Added a composite action intended to centralize the spot decision logic.

Scrupulous human review needed:

  • Verify the release/merge-queue detection logic matches the repo’s actual release and tagging flows (merge_group, refs/heads/release/…, refs/tags/v…, release event), since misclassification can materially impact runner cost and pipeline latency.
  • Confirm the runner label parser backing runs-on=.../spot=... treats spot=false and spot=co exactly as intended across all instance families used here.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
.github/workflows/integration-tests.yml Makes builder and compile runners switch spot based on event/ref conditions.
.github/workflows/go-mod-cache.yml Changes cache jobs from fixed spot=false to conditional spot behavior.
.github/workflows/docker-build.yml Makes Docker build runner labels conditionally use spot vs non-spot.
.github/workflows/cre-wf-caching-test.yml Switches CRE caching test runner to conditional spot behavior.
.github/workflows/cre-system-tests.yaml Parameterizes jq-generated runner labels with conditional spot.
.github/workflows/cre-soak-memory-leak.yml Switches soak test runner to conditional spot behavior.
.github/workflows/cre-regression-system-tests.yaml Parameterizes regression runner labels with conditional spot.
.github/workflows/cre-local-env-tests.yaml Switches local env test jobs to conditional spot behavior.
.github/workflows/codeql.yml Makes CodeQL Go runner label conditional on event/ref.
.github/workflows/ci-deployments.yml Switches deployment job runners to conditional spot behavior.
.github/workflows/ci-core.yml Switches multiple core CI runner labels to conditional spot behavior.
.github/workflows/ccip-system-tests.yaml Parameterizes CCIP system test runner labels with conditional spot.
.github/actions/get-runner-spot/action.yml Adds a composite action to compute spot settings (currently not wired into workflows).

Comment thread .github/actions/get-runner-spot/action.yml
Comment thread .github/workflows/integration-tests.yml Outdated
@trunk-io

trunk-io Bot commented Aug 4, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

.github/workflows/integration-tests.yml:251

  • build-chainlink runs two matrix jobs (core + plugins) that can execute concurrently, but SH_BUILDER_RUNNER is now the same label for both (based only on github.run_id). This reintroduces runner contention/serialization that the previous -core/-plugins suffixes avoided (the earlier comment explicitly called this out). Consider restoring per-matrix unique labels (e.g., runs-on=${{ github.run_id }}-core/... and ...-plugins/...) or otherwise including a matrix-specific suffix in the runs-on= label.
          GH_BUILDER_RUNNER: ubuntu22.04-8cores-32GB
          SH_BUILDER_RUNNER: runs-on=${{ github.run_id }}/cpu=16/ram=64/family=m7i+m8i/${{ steps.spot-config.outputs.spot_param }}/extras=s3-cache+tmpfs
        run: |
          if [[ "${OPT_OUT}" == "true" ]]; then
            echo "builder-runner-label=${GH_BUILDER_RUNNER}" | tee -a "$GITHUB_OUTPUT"

Comment thread .github/workflows/codeql.yml
Comment thread .github/workflows/cre-local-env-tests.yaml Outdated
Comment thread .github/actions/get-runner-spot/action.yml Outdated
Comment thread .github/workflows/ccip-system-tests.yaml Outdated
Comment thread .github/workflows/codeql.yml Outdated
Comment thread .github/workflows/cre-soak-memory-leak.yml Outdated
Comment thread .github/workflows/cre-wf-caching-test.yml Outdated
Comment thread .github/workflows/integration-tests.yml Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

.github/workflows/integration-tests.yml:288

  • Both matrix entries use the same needs.labels.outputs.builder-runner-label value. If the self-hosted runner provisioning expects a unique label per job (as the previous -core / -plugins suffixes implied), this can cause runner contention/serialization or runner-stealing between the core and plugins image builds.
          - name: ""
            runner: ${{ needs.labels.outputs.builder-runner-label || 'ubuntu22.04-8cores-32GB' }}
            dockerfile: core/chainlink.Dockerfile
            tag-suffix: ""
            cache-scope: core
            any-should-run: >-
              ${{
                github.event_name == 'workflow_dispatch' ||
                needs.changes.outputs.general-changes == 'true' ||
                needs.changes.outputs.core-changes == 'true' ||
                needs.changes.outputs.cre-changes == 'true' ||
                needs.changes.outputs.ccip-changes == 'true' ||
                needs.labels.outputs.run-e2e-tests-label-found == 'true'
              }}

          - name: (plugins)
            runner: ${{ needs.labels.outputs.builder-runner-label || 'ubuntu22.04-8cores-32GB' }}
            dockerfile: plugins/chainlink.Dockerfile

.github/workflows/codeql.yml:119

  • This changes the self-hosted Go CodeQL runner instance family to c6i, which is unrelated to the PR's stated goal (spot gating) and differs from the c7* families used elsewhere for similar 32cpu/64GB runners. If unintentional, it could reduce performance/capacity for CodeQL runs.
        env:
          OPT_OUT: ${{ steps.pr-labels.outputs.check-label-found || 'false' }}
          SH_GO_RUNNER: runs-on=${{ github.run_id }}/cpu=32/ram=64/family=c6i/${{ steps.spot-config.outputs.spot_param }}/extras=s3-cache+tmpfs
          GH_GO_RUNNER: ubuntu24.04-8cores-32GB

Comment thread .github/workflows/integration-in-memory-tests.yml

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (3)

.github/workflows/integration-in-memory-tests.yml:52

  • actions/checkout in the changes job can end up checking out the repository default branch when neither inputs.chainlink_version nor inputs.cl_ref is set (both are optional). For PR/merge_group events that means advanced-triggers may evaluate the wrong code revision, and it contradicts the chainlink_version input description (“defaults to github.sha”).
      - name: Checkout the repo
        uses: actions/checkout@v7
        with:
          persist-credentials: false
          repository: smartcontractkit/chainlink
          ref: ${{ inputs.chainlink_version || inputs.cl_ref }}

.github/workflows/integration-tests.yml:253

  • SH_BUILDER_RUNNER is now a single label shared by both the core and plugins build matrix entries. This removes the previous per-image isolation and can cause the two builds to contend for the same ephemeral self-hosted runner (serialization) or conflict over the workspace/caches, increasing CI latency and flakiness.
          OPT_OUT: ${{ steps.label-runs-on-opt-out.outputs.check-label-found || 'false' }}
          GH_BUILDER_RUNNER: ubuntu22.04-8cores-32GB
          SH_BUILDER_RUNNER: runs-on=${{ github.run_id }}/cpu=16/ram=64/family=m7i+m8i/${{ steps.spot-config.outputs.spot_param }}/extras=s3-cache+tmpfs
        run: |
          if [[ "${OPT_OUT}" == "true" ]]; then
            echo "builder-runner-label=${GH_BUILDER_RUNNER}" | tee -a "$GITHUB_OUTPUT"
          else
            echo "builder-runner-label=${SH_BUILDER_RUNNER}" | tee -a "$GITHUB_OUTPUT"

.github/workflows/ci-core.yml:186

  • This PR’s description is about gating spot by event/ref, but this hunk also changes the instance family selection (e.g., family=c7gd+c8g for golangci). Similar family changes appear elsewhere in this workflow (unit/fuzz). If unintentional, it could change capacity/availability characteristics or break runner matching.
    runs-on:
      - runs-on=${{ github.run_id }}-${{ strategy.job-index }}
      - cpu=16
      - ram=32
      - family=c7gd+c8g
      - ${{ needs.filter.outputs.spot_param }}
      - image=ubuntu24-full-arm64
      - extras=s3-cache

@kalverra
kalverra enabled auto-merge August 5, 2026 15:16
@kalverra
kalverra requested a review from chainchad August 5, 2026 15:16
@kalverra
kalverra requested a review from erikburt August 5, 2026 15:28
@cl-sonarqube-production

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants