Select SSO nightly specs by tag so every job reports clean counts

[siddhant1]

Change

Tag the provider-specific describes and let each matrix entry exclude what its fixture cannot serve via --grep-invert.

Spec	Tag added
SSOLogin	—	untagged ⇒ runs on every provider
OktaSelfSignupClaims	@okta	needs the Okta tenant
SSORenewal	@tokenRenewal	shortens the global access-token TTL

Provider	excluded_tags	Collects
okta	@tokenRenewal	8
keycloak-azure-saml	@okta	9
keycloak-azure-saml-crosssite	@okta|@tokenRenewal	6

Renewal is tagged by capability, not by provider, because the TTL override — not Keycloak-ness — is what makes it unsafe against the live Okta tenant.

The -crosssite job excludes renewal too, deliberately this time. All three renewal tests act on the OM origin (they await AUTH_REFRESH_PATH and clear OM_SESSION) and never touch the IdP after login, whereas -crosssite only relocates the IdP to 127.0.0.1 so the SAML callback POST is cross-site. Post-login refresh is same-origin and identical in both keycloak jobs, so running it twice costs ~1.5 min for no added signal.

A side benefit: excluded specs are dropped at collection time, so their beforeAll — which performs an admin login and patches global security config — no longer runs at all.

Verification

Counts and composition both confirmed via --list, no server required:

exclude @tokenRenewal        → 8   (2 OktaSelfSignupClaims + 6 SSOLogin)
exclude @okta                → 9   (6 SSOLogin + 3 SSORenewal)
exclude @okta|@tokenRenewal  → 6   (6 SSOLogin)

Spec-for-spec identical to run 30973822261's passing tests, with zero skips. If a passed count moves, the retag is wrong — that's the review heuristic here.

On Slack

The nightly posts one message per matrix job. Only the last number changes:

Job	Before	After
okta	✅ 8 ❌ 0 🟡 0 ⏩ 3	✅ 8 ❌ 0 🟡 0 ⏩ 0
keycloak-azure-saml	✅ 9 ❌ 0 🟡 0 ⏩ 2	✅ 9 ❌ 0 🟡 0 ⏩ 0
crosssite	✅ 6 ❌ 0 🟡 0 ⏩ 5	✅ 6 ❌ 0 🟡 0 ⏩ 0

⏩ becomes a real signal: 0 in every healthy run, so any non-zero value is worth clicking.

Deliberately unchanged

• Missing credentials still skip rather than fail. The credential half of each guard stays, so a rotated secret keeps the job green — visible on Slack as ✅ 0 | ⏩ N, which at least reads as wrong now that healthy runs are always ⏩ 0. Turning that into a hard failure is a separate call.
• playwright.config.ts still hardcodes the 3-spec testMatch, so a new SSO spec must be registered there or it never runs. Pre-existing; out of scope.

Notes for reviewers

• SESSION_RENEWAL_TAGS is extracted to a const only to keep the test.describe line under Prettier's 80-char width — inlining the array reflows the call and reindents the whole ~200-line describe body, burying the real change.
• An entry that omits excluded_tags collects every spec, so a future provider added without one fails loudly instead of silently not running. The onboarding comment at the top of the workflow documents this as step 4.

🤖 Generated with Claude Code

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

✅ Playwright Results — workflow succeeded

Validated commit ceadde97a26ca42fea44c6991a2c9270f421d061 in Playwright run 31011039799, attempt 1.

✅ 548 passed · ❌ 0 failed · 🟡 2 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 56m 58s

⏱️ Max setup 3m 1s · max shard execution 18m 33s · max shard-job elapsed before upload 22m 39s · reporting 4s

🌐 203.51 requests/attempt · 2.83 app boots/UI scenario · 10.06% common-shard skew

Optimization targets still in progress:

• Browser traffic was 203.51 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 2.83 per UI scenario (1621 boots / 573 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	127	0	0	0	0	0
✅ Shard chromium-02	132	0	0	0	0	0
🟡 Shard chromium-03	139	0	1	0	0	0
✅ Shard data-asset-rules-01	61	0	0	0	0	0
✅ Shard domain-isolation-01	14	0	0	0	0	0
✅ Shard global-state-01	34	0	0	0	0	0
✅ Shard ingestion-01	1	0	0	0	0	0
✅ Shard reindex-01	2	0	0	0	0	0
✅ Shard search-01	10	0	0	0	0	0
🟡 Shard search-rbac-01	28	0	1	0	0	0

🟡 2 flaky test(s) (passed on retry)

• Pages/Entity.spec.ts › Domain Propagation (shard chromium-03, 1 retry)
• Flow/SearchRBAC.spec.ts › User without permission (shard search-rbac-01, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[Copilot]

Pull request overview

This PR updates the nightly SSO Playwright matrix to exclude provider/capability-specific specs at collection time (via tags + --grep-invert), eliminating “always skipped” noise and ensuring each job reports clean pass/fail counts.

Changes:

• Add Playwright tags for provider/capability-specific suites (@okta, @tokenRenewal) so the workflow can filter them per matrix provider.
• Remove provider-type-based test.skip() guards in the tagged specs, keeping only credential-based skipping.
• Extend the nightly workflow matrix with excluded_tags and pass it to Playwright via --grep-invert.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
openmetadata-ui/src/main/resources/ui/playwright/e2e/Auth/SSORenewal.spec.ts	Tags renewal suite with @tokenRenewal and removes provider-specific skip guard.
openmetadata-ui/src/main/resources/ui/playwright/e2e/Auth/OktaSelfSignupClaims.spec.ts	Tags Okta-only suite with @okta and removes provider-specific skip guard.
.github/workflows/playwright-sso-login-nightly.yml	Adds excluded_tags to provider matrix and filters suites using --grep-invert to prevent permanent skips.

[gitar-bot]

Code Review ✅ Approved

Updates Playwright SSO nightly specs to be selected by tag using --grep-invert, eliminating permanent skipped counts from matrix jobs. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar | Powered by Gitar — free for open source}

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (3)

.github/workflows/playwright-sso-login-nightly.yml:135

• --grep-invert "${{ matrix.provider.excluded_tags }}" is always passed. If a future matrix entry omits excluded_tags (as the onboarding comment suggests is allowed), this expands to an empty regex, which matches everything and inverts to exclude all tests. That contradicts the comment (“collects every spec”) and can lead to a confusing ‘no tests’ run.

Make the flag conditional so missing excluded_tags runs the full suite.

          # failure that the localhost-only job cannot, guarding the SAML RelayState fix.
          KEYCLOAK_SAML_BASE_URL: ${{ matrix.provider.name == 'keycloak-azure-saml-crosssite' && 'http://127.0.0.1:8080' || 'http://localhost:8080' }}
          PLAYWRIGHT_IS_OSS: true
        run: npx playwright test --project=sso-auth --workers=1 --grep-invert "${{ matrix.provider.excluded_tags }}"

openmetadata-ui/src/main/resources/ui/playwright/e2e/Auth/SSORenewal.spec.ts:48

• This spec mutates global security config to enforce an aggressively short access-token TTL (withShortSamlTokenValidity). Removing the provider-type guard means that if SSO_USERNAME/SSO_PASSWORD are set, the renewal suite can run against any configured provider (including the live Okta tenant) when executed outside this nightly workflow’s --grep-invert.

Reintroduce the provider guard so the TTL override cannot be applied against unsupported providers, and keep the lint-disable rationale inline (it’s useful context for why this file is allowed to skip).

const SESSION_RENEWAL_TAGS = ['@sso', '@Platform', '@tokenRenewal'];

test.describe.configure({ mode: 'serial' });

test.describe('SSO Session Renewal', { tag: SESSION_RENEWAL_TAGS }, () => {
  test.slow();
  // eslint-disable-next-line playwright/no-skipped-test
  test.skip(
    !username || !password,
    `${SSO_ENV.USERNAME} + ${SSO_ENV.PASSWORD} must be set`
  );

openmetadata-ui/src/main/resources/ui/playwright/e2e/Auth/OktaSelfSignupClaims.spec.ts:54

• This spec still derives the provider helper from SSO_PROVIDER_TYPE, but the provider guard was removed from test.skip. If someone runs the SSO project/specs without the nightly workflow’s --grep-invert (or adds a provider without the right excluded_tags), this describe can run against a non-Okta provider, applying the wrong SSO config and producing misleading failures.

Restore the provider-type guard (it won’t affect nightly counts because @okta specs are excluded at collection time for non-Okta jobs).

      // eslint-disable-next-line playwright/no-skipped-test
      test.skip(
        !username || !password,
        `Requires ${SSO_ENV.USERNAME}/${SSO_ENV.PASSWORD}`
      );