[DO NOT MERGE] canary: force DON2DON non-determinism to validate mixed-env - #23291
Draft
prashantkumar1982 wants to merge 10 commits into
Draft
[DO NOT MERGE] canary: force DON2DON non-determinism to validate mixed-env#23291prashantkumar1982 wants to merge 10 commits into
prashantkumar1982 wants to merge 10 commits into
Conversation
Contributor
|
✅ No conflicts with other open PRs targeting |
Contributor
|
I see you updated files related to
|
|
|
prashantkumar1982
force-pushed
the
cre/mixed-env-canary-test
branch
from
July 31, 2026 18:51
93c8bb0 to
828f32a
Compare
prashantkumar1982
force-pushed
the
cre/mixed-env-non-determinism
branch
from
July 31, 2026 19:08
41c2e2f to
b4be2da
Compare
prashantkumar1982
force-pushed
the
cre/mixed-env-canary-test
branch
from
July 31, 2026 19:12
828f32a to
e8e7507
Compare
prashantkumar1982
force-pushed
the
cre/mixed-env-non-determinism
branch
from
July 31, 2026 20:11
b4be2da to
8b25b56
Compare
prashantkumar1982
force-pushed
the
cre/mixed-env-canary-test
branch
3 times, most recently
from
August 3, 2026 16:19
9ba41a3 to
604f6de
Compare
Adds a "mixed-env" Local CRE topology that runs each multi-node DON with 2 nodes built from the PR image and 2 from the develop image, and fails the run if the two code versions ever disagree — surfacing non-determinism and cross-version incompatibilities before merge. - configs/mixed-env-don.toml.tmpl + render-mixed-env.sh: 2-2 image split per DON via override_mode = "each"; rendered with envsubst (gitignored output). - test-helpers/nondeterminism_scan.go + smoke/cre TestMain: scan all node logs after the suite (mixed-env only) and fail with "Non-Determinism introduced". Markers: the libocr OCR3 signature-mismatch warning plus the DON2DON request/response divergence warnings. - cre-system-tests.yaml: run mixed-env for Bucket_A/B; render step resolves PR + develop-nightly images; empty chainlink-image so per-node TOML images are used; optional mixed_env_baseline_image input; exclude TestMain from discovery. - docs/mixed-env.md: developer-facing overview. Reuses the already-built per-PR image and the nightly develop image, so no extra image builds are added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
prashantkumar1982
force-pushed
the
cre/mixed-env-non-determinism
branch
from
August 3, 2026 18:20
8b25b56 to
63aada7
Compare
prashantkumar1982
force-pushed
the
cre/mixed-env-canary-test
branch
from
August 3, 2026 18:29
604f6de to
ca6403c
Compare
Review feedback: keep cre-system-tests.yaml simple by extracting the mixed-env topology into a dedicated reusable workflow, mirroring how cre-regression-system-tests.yaml is structured. - New .github/workflows/cre-mixed-env-tests.yaml: owns the mixed-env matrix (Bucket_A, Bucket_B, EVM_Read_*), the render step, env start (empty CTF_CHAINLINK_IMAGE), and the non-determinism gate. No per-topology conditionals — every job is mixed-env, so it's simpler than the inline version. - integration-tests.yml: call it from a new run-core-cre-mixed-env-tests job, gated the same as the other CRE e2e workflows. - cre-system-tests.yaml: revert all mixed-env additions (matrix entries, render step, gate step, mixed_env_baseline_image input, chainlink-image conditional). Only the TestMain discovery exclusion remains (TestMain lives in the smoke pkg). - docs updated to point at the new workflow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
prashantkumar1982
force-pushed
the
cre/mixed-env-canary-test
branch
from
August 3, 2026 19:30
ca6403c to
4a18325
Compare
…r in gate Addresses review feedback on cre-mixed-env-tests.yaml: - Fail-fast image presence check: before starting the local CRE, verify both the PR and develop images exist in the registry (docker manifest inspect) and fail with an informative error if not (e.g. transient build/registry failure or the develop nightly not yet published) instead of failing deep inside env start. - Non-determinism gate now reuses the Go scanner instead of a duplicated bash grep: the marker list is centralized as helpers.NonDeterminismNeedles, and the gate runs cmd/mixed-env-nondeterminism-check (go run) which calls helpers.ScanContainersForNeedles. The smoke-suite TestMain uses the same list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…alse positives Per-PR mixed-env now uses chainlink-integration-tests:<base-sha> — the exact develop commit the PR image is built on top of (the CI auto-merge base) — as the develop side of the 2-2 split, instead of the latest develop nightly. Because the PR image = that develop commit + the PR's changes, the two halves of each DON now differ ONLY by the PR's changes, so develop's own churn between the PR's branch point and HEAD no longer surfaces as a "Non-Determinism introduced" false positive. The base-sha image is the same build/repo as the PR image and is already published by the develop-push build, so no extra image build is added. Falls back to the develop nightly for non-PR runs or if the base-sha image has aged out of ECR (the existing image-presence check then guards the fallback). - cre-mixed-env-tests.yaml: new baseline_ref input; render step prefers chainlink-integration-tests:<baseline_ref> (existence-checked) over the nightly. - integration-tests.yml: pass baseline_ref: github.event.pull_request.base.sha. - docs/mixed-env.md: document the base-commit baseline and the nightly fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The baseline is now the exact commit of whatever branch the PR merges into (chainlink-integration-tests:<base.sha>), which already generalizes correctly to release branches — a release PR is compared against its release line, not develop. Fix the fallback for when that base image isn't published (release-branch commits aren't force-built like develop, so their branch-tip image is often absent): - develop base -> fall back to the develop nightly (a valid stand-in for develop). - non-develop base (e.g. release/2.57.1) -> SKIP mixed-env with a warning, instead of comparing a release-based PR against the develop nightly (which would flood the run with false positives from the entire release<->develop divergence). - cre-mixed-env-tests.yaml: split baseline resolution into a Resolve-baseline step (outputs image + skip); new base_ref_name input gates the fallback; all env-start, test, and gate steps are guarded on skip != 'true'. - integration-tests.yml: pass base_ref_name: github.event.pull_request.base.ref. - docs/mixed-env.md: document release-branch baseline + skip behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…escape hatch Make the mixed-env non-determinism check block merge, and give it a self-service release valve in the same change. Required: add run-core-cre-mixed-env-tests to the ETH Smoke Tests merge gate (check-e2e-test-results) alongside the other e2e suites, so a "Non-Determinism introduced" failure now fails the required gate. The gate treats a skipped job as a non-blocking warning, so non-CRE PRs and release-branch PRs with no baseline (which skip internally) don't block. Escape hatch: new `skip-mixed-env` PR label, mirroring skip-e2e-regression. The labels job detects it; the setup job turns it into a run-mixed-env output; the mixed-env job gates its `if` on it. Applying the label skips the job -> the gate sees a warning -> merge unblocked, no admin needed. Like the regression tests, mixed-env no longer runs in the merge queue, so the bypass carries through. docs/mixed-env.md: new "Required check & emergency bypass" section documenting the label, admin/ruleset bypass, and de-wiring, plus a link from the escape-hatch note. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d-env Temporary canary to verify the mixed-env non-determinism check catches a real divergence in CI, without breaking unit tests. - capability_executor.go: when CRE_CANARY_NONDETERMINISM is set, unconditionally stamps Metadata.OrgID on executable-capability requests (ignoring PropagateOrgIDInRequestMetadata). PR-built nodes then emit a different request payload than develop-built nodes; the capability DON logs "received messages with the same id and different payloads", which the mixed-env TestMain scan turns into "Non-Determinism introduced". - mixed-env-don.toml.tmpl: set CRE_CANARY_NONDETERMINISM on the workflow DON so the canary activates only in the mixed-env e2e run (unit tests unaffected). Revert this commit once CI has demonstrated the catch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed-env Second canary, at the OCR3 report-attestation layer. In vault/plugin.go wrapReportWithKeyBundleInfo, when CRE_CANARY_NONDETERMINISM is set, perturb the SIGNED report bytes on PR-built nodes so vault OCR3 nodes disagree during report attestation (the state/outcome is already committed by then, so this is a clean report-phase divergence). libocr logs "This is commonly caused by non-determinism in the ReportingPlugin", which the mixed-env gate catches (Bucket_B). Also sets CRE_CANARY_NONDETERMINISM on the capabilities DON. An earlier version omitted ReportInfo.Id, but that lives in the unsigned .Info and only broke report routing (a functional test failure), not attestation. Revert with the rest of the canary once CI has demonstrated the catch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
prashantkumar1982
force-pushed
the
cre/mixed-env-canary-test
branch
from
August 4, 2026 21:33
4a18325 to
32b7f94
Compare
prashantkumar1982
changed the base branch from
cre/mixed-env-non-determinism
to
develop
August 4, 2026 22:06
Contributor
CORA - Pending Reviewers
Legend: ✅ Approved | ❌ Changes Requested | 💬 Commented | 🚫 Dismissed | ⏳ Pending | ❓ Unknown For more details, see the full review summary. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Temporary canary stacked on top of #23290 (mixed-env topology) to prove the mixed-env non-determinism check actually fires in CI.
What it does:
capability_executor.gounconditionally stampsMetadata.OrgID = "canary-nondeterminism"on executable-capability requests (ignoringPropagateOrgIDInRequestMetadata). PR-built nodes then emit a different request payload than develop-built nodes.Expected result:
received messages with the same id and different payloads.TestMainscan fails themixed-envmatrix entries (Test_CRE_V2_Suite_Bucket_A/_B) withNon-Determinism introduced.Base is the feature branch, so the diff is just the 6-line canary. Close and delete this branch once CI has demonstrated the catch.