Skip to content

fix(workflows): eliminate null-businessKey race + silence non-OM log noise - #31017

Open
yan-3005 wants to merge 7 commits into
mainfrom
fix/workflow-log-noise-null-safety-30985
Open

fix(workflows): eliminate null-businessKey race + silence non-OM log noise#31017
yan-3005 wants to merge 7 commits into
mainfrom
fix/workflow-log-noise-null-safety-30985

Conversation

@yan-3005

@yan-3005 yan-3005 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Two independent fixes that showed up together in the Aug-5 AUT PostgreSQL 1.11.13 → 1.13 nightly (Loki namespace aut-30964402587): the MainWorkflowTerminationListener was NPE-ing every ~10s at ERROR, and the [STAGE_SKIP] / [STAGE_UPDATE_NO_ID] pair was flooding logs at WARN/ERROR on Flowable-internal transitions.

Both share the same trigger: process instances with a null businessKey. The root cause is in this PR (commit 2). Commit 1 handles the legacy tail and log noise.

Root cause (commit 2 — fd80276dc8)

The periodic + legacy fallback paths in WorkflowHandler.triggerWorkflow called startProcessInstanceByKey(key) with no businessKey. Design relied on WorkflowInstanceListener.updateBusinessKey firing at the trigger's startEvent to backfill a fresh UUID.

Two ways that broke:

  1. Race: if WorkflowInstanceListener.execute threw before reaching line 108 (Entity.getEntityTimeSeriesRepository transient failure, DB glitch, etc.), the trigger ran to completion with null businessKey. Its CallActivity (EventBasedEntityTrigger.java:170, inheritBusinessKey=true) spawned the MainWorkflow inheriting null. Every downstream listener that did UUID.fromString(businessKey) NPE-d.
  2. Silent overwrite: the listener unconditionally re-assigned UUID.randomUUID(), discarding caller-supplied businessKeys from TaskRepository.triggerByKey (task id) and MigrationUtil.triggerByKey (task id). Task-workflow correlation drifted.

Fix

  • WorkflowHandler.triggerWorkflow:1745,1759: pass UUID.randomUUID().toString() on both startProcessInstanceByKey calls. Trigger instance has a businessKey from the moment it exists — no race window.
  • WorkflowInstanceListener.addWorkflowInstance: only call updateBusinessKey when businessKey is absent. Preserves caller-supplied UUIDs and closes the race.

Log-noise fixes (commit 1 — 758b9768ba)

Belt-and-suspenders for any legacy pre-inheritance process instance still sitting in Flowable (upgrade rows, force-cancelled test teardowns).

  • MainWorkflowTerminationListener.execute: null-guard businessKey, log at DEBUG and return instead of NPE-ing the catch block at ERROR.
  • WorkflowInstanceStageListener: [STAGE_SKIP] WARN → DEBUG; [STAGE_UPDATE_NO_ID] ERROR → DEBUG. Both fire on the same non-OM-managed instances flagged by the businessKey check.
  • WorkflowFailureListener: add Workflow definition deleted to INTENTIONAL_CANCELLATION_CAUSES. On a workflow-definition hard-delete with N running instances, we were logging N × PROCESS_CANCELLED WARNs — now silent, matching the other intentional teardown causes already whitelisted.

Behavior change

  • OM-managed workflows: no behavior change. Every existing triggerByKey caller already passed a non-null businessKey; the listener's overwrite is now skipped for them, which is what the caller expected.
  • Periodic/legacy triggerWorkflow paths: now have a valid businessKey from the start, so the WorkflowInstance row and its stage records get written correctly on the FIRST listener firing instead of racing.
  • Non-OM Flowable-internal instances (pre-inheritance legacy rows, force-cancelled test teardowns): no ERROR/WARN spam.

Loki evidence (aut-30964402587, 01:20-02:00 UTC 2026-08-05)

  • MainWorkflowTerminationListener NPE: 50+ hits in a 40-min window (all UUID.fromString(null))
  • [STAGE_SKIP] / [STAGE_UPDATE_NO_ID]: paired hits from WorkflowInstanceStageListener on the same non-OM instances
  • PROCESS_CANCELLED × 17 on a single pw-user-approval-owners-table-* workflow deletion (test teardown emits one per running instance)

Test plan

  • mvn spotless:apply clean on the 4 touched files
  • mvn test -pl openmetadata-service -Dtest='WorkflowHandler*Test' — run in CI
  • Manual: trigger a periodic workflow, verify WorkflowInstance and WorkflowInstanceState rows land on the first firing (previously required a retry when the listener raced)
  • Manual: delete a WorkflowDefinition with running instances, verify no PROCESS_CANCELLED WARN in logs
  • Backport to 1.13 once merged

🤖 Generated with Claude Code


Summary by Gitar

  • Entity hard-delete workflow cleanup:
    • Added cancelInstancesForEntities batching mechanism and called it from EntityRepository hard-delete and bulk hard-delete paths
    • Whitelisted Entity deleted and Workflow definition deleted as intentional cancellation causes in WorkflowFailureListener
  • FilterEntityImpl robustness:
    • Short-circuits FilterEntityImpl with passesFilter=false when relatedEntity variable is absent or blank

This will update automatically on new commits.

yan-3005 and others added 2 commits August 5, 2026 14:20
Flowable fires MainWorkflowTerminationListener, WorkflowInstanceStageListener
and PROCESS_CANCELLED events on process instances that carry no business key
(force-cancel at test teardown, Flowable-internal transitions, legacy rows).
Those aren't OM-managed, so the log lines they produce aren't operational —
they were just drowning production Loki in recurring ERROR/WARN traces.

- MainWorkflowTerminationListener.execute: guard businessKey up-front and log
  at DEBUG when absent. Previously UUID.fromString(null) threw and the catch
  logged the NPE at ERROR every ~10s.
- WorkflowInstanceStageListener: STAGE_SKIP (WARN) and the downstream
  STAGE_UPDATE_NO_ID (ERROR) both fire on the same non-OM instances. Move
  both to DEBUG.
- WorkflowFailureListener: whitelist "Workflow definition deleted" as an
  intentional cancellation cause so per-instance PROCESS_CANCELLED WARNs stop
  when a WorkflowDefinition is deleted (test teardown emits N of these per
  running instance).

No behavior change for OM-managed workflows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…UID race

Root cause of the MainWorkflowTerminationListener NPEs (see prior commit): the
periodic/legacy triggerWorkflow paths call startProcessInstanceByKey without
a businessKey. They relied on WorkflowInstanceListener.updateBusinessKey to
overwrite it later via the trigger start-event listener. When that listener
threw before reaching line 108 (e.g. Entity.getEntityTimeSeriesRepository
failed transiently), the trigger ran to completion with a null businessKey,
its CallActivity spawned the MainWorkflow with inheritBusinessKey=true, and
every downstream listener that did UUID.fromString(businessKey) hit an NPE.

- WorkflowHandler.triggerWorkflow: pass UUID.randomUUID().toString() at both
  startProcessInstanceByKey call sites so the trigger process instance has a
  businessKey from the moment it exists, not "eventually, if the listener
  runs".
- WorkflowInstanceListener.addWorkflowInstance: only call updateBusinessKey
  when businessKey is absent. Preserves caller-supplied UUIDs (task ids,
  deterministic keys) instead of silently discarding them.

The DEBUG-level null guards added in the previous commit stay as
belt-and-suspenders for any legacy pre-inheritance process instance still
sitting in Flowable, but new triggers should no longer produce them.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yan-3005
yan-3005 requested a review from a team as a code owner August 5, 2026 09:02
Copilot AI review requested due to automatic review settings August 5, 2026 09:02
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

  • No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions github-actions Bot added backend safe to test Add this label to run secure Github workflows on PRs labels Aug 5, 2026
@yan-3005 yan-3005 added the To release Will cherry-pick this PR into the release branch label Aug 5, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens OpenMetadata’s Flowable workflow execution against null businessKey propagation (which can crash downstream listeners) and reduces log noise for non-OM-managed/legacy Flowable process instances.

Changes:

  • Ensure WorkflowHandler.triggerWorkflow(...) always starts trigger process instances with a generated UUID businessKey, closing the null-key race window.
  • Preserve caller-supplied businessKey in WorkflowInstanceListener by only backfilling a UUID when the key is absent.
  • Downgrade/guard logging and failure handling for non-OM-managed instances to avoid WARN/ERROR spam.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java Starts trigger process instances with a non-null UUID businessKey to prevent inheritance of null keys into MainWorkflow.
openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowInstanceListener.java Avoids overwriting an existing businessKey; only generates one when missing.
openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/MainWorkflowTerminationListener.java Adds an early businessKey null/blank guard to skip non-OM-managed instances without throwing.
openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowInstanceStageListener.java Reduces log severity for stage callbacks when instances are not OM-managed / missing identifiers.
openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowFailureListener.java Treats “Workflow definition deleted” cancellations as intentional to avoid repetitive warnings.

Comment on lines +203 to +207
if (workflowInstanceStateId == null) {
LOG.error(
// Non-OM-managed process instances (no business key) never had a stage record
// created in addNewStage, so this update-stage callback is expected to be a
// no-op for them. Downgrade to DEBUG so the [STAGE_UPDATE_NO_ID] noise stops
// showing up at ERROR level once per Flowable transition.
Comment on lines 1745 to 1751
// Assign the WorkflowInstance UUID up-front instead of relying on
// WorkflowInstanceListener.updateBusinessKey firing later. The CallActivity inside the
// trigger BPMN uses inheritBusinessKey=true, so a null businessKey here propagates
// into the MainWorkflow and eventually blows up UUID.fromString(null) in every
// downstream listener. Setting it at start time closes the race window.
runtimeService.startProcessInstanceByKey(baseProcessKey, UUID.randomUUID().toString());
return true;
- WorkflowInstanceStageListener: keep STAGE_UPDATE_NO_ID at ERROR when the
  process has a businessKey (real bug); only downgrade to DEBUG for non-OM
  process instances.
- WorkflowInstanceListener: reject non-UUID businessKeys and assign a fresh
  UUID instead of NPE-ing at UUID.fromString. Preserves caller-supplied UUIDs
  as before.
- Trim in-code comments to code-explanatory only (no incident / log-storage
  references).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 5, 2026 09:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java:1762

  • New behavior relies on always starting trigger processes with a non-null businessKey to prevent downstream UUID parsing failures, but there’s no unit test coverage for triggerWorkflow()/triggerProcessDefinitions() verifying the runtimeService is invoked with a generated businessKey. Existing WorkflowHandler tests focus on initialization/schema and don’t exercise this path.
      // Trigger BPMN's CallActivity has inheritBusinessKey=true; passing a businessKey
      // here ensures the spawned MainWorkflow inherits a non-null WorkflowInstance UUID.
      runtimeService.startProcessInstanceByKey(baseProcessKey, UUID.randomUUID().toString());
      return true;
    } catch (FlowableObjectNotFoundException ex) {
      LOG.error("No process definition found for key: {}", baseProcessKey);
      return false;
    }
  }

  private boolean triggerProcessDefinitions(
      RuntimeService runtimeService, List<String> processKeys) {
    boolean anyStarted = false;
    for (String processKey : processKeys) {
      try {
        LOG.info("Triggering process with key: {}", processKey);
        runtimeService.startProcessInstanceByKey(processKey, UUID.randomUUID().toString());
        anyStarted = true;

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/MainWorkflowTerminationListener.java:21

  • PR description says null/blank businessKey should be debug-logged before returning to aid diagnosing unexpected missing keys. Current implementation returns silently, which can hide real OM-managed failures (e.g., if a regression reintroduces null keys) and doesn’t match the stated behavior.
    String businessKey = execution.getProcessInstanceBusinessKey();
    if (businessKey == null || businessKey.isBlank()) {
      return;
    }

…absent

FilterEntityImpl reads the relatedEntity variable populated by change-event
triggers. When triggerWorkflow is invoked without an event context (e.g. the
manual /trigger REST endpoint hit on an eventBasedEntity workflow), the
variable is null and MessageParser.EntityLink.parse(null) NPEs inside a
regex Matcher.

Guard the null path: log at DEBUG and set passesFilter=false so the trigger
process ends cleanly instead of throwing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 5, 2026 09:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (2)

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowInstanceListener.java:126

  • In addWorkflowInstance, updateBusinessKey(...) updates the process instance via runtimeService.updateBusinessKey, but the subsequent UUID.fromString(execution.getProcessInstanceBusinessKey()) still relies on the DelegateExecution view of the businessKey. If Flowable doesn't reflect the updated businessKey on the current execution object immediately, this can still throw (null/old value) and reintroduce the null-businessKey race this PR is addressing. Use the locally generated UUID for parsing/DB insert after updating Flowable, instead of re-reading from execution.
    // Preserve caller-supplied businessKey when it is a valid UUID; otherwise assign a
    // fresh WorkflowInstance UUID. Guarantees the field UUID.fromString parses below.
    String existingBusinessKey = execution.getProcessInstanceBusinessKey();
    if (!isUuid(existingBusinessKey)) {
      updateBusinessKey(execution.getProcessInstanceId());

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/MainWorkflowTerminationListener.java:21

  • The PR description says MainWorkflowTerminationListener.execute should "log at DEBUG and return" for null/blank businessKey instances, but the implementation returns silently. Adding a debug log keeps the intended noise reduction while still leaving a breadcrumb when this legacy condition happens.
    // businessKey identifies the OM WorkflowInstance; a null key means the process was not
    // started through the OM trigger path so there is no WorkflowInstance row to update.
    String businessKey = execution.getProcessInstanceBusinessKey();
    if (businessKey == null || businessKey.isBlank()) {
      return;
    }

…leted

Hard-deleting an entity used to leave any MainWorkflow process instance tied
to it running against a row that no longer exists. Every downstream node
that called Entity.getEntity(entityLink) then threw EntityNotFoundException,
which read as an operational error in the logs even though the underlying
cause was the intentional entity delete.

Add WorkflowHandler.cancelInstancesForEntity(entityId, reason) that queries
Flowable for process instances whose global_relatedEntityId variable matches
and cancels each. Called from EntityRepository.hardDelete before the feed
threads are cleaned up (both single-entity and bulk paths). Guarded on
WorkflowHandler.isInitialized() so migration-mode deletes stay safe.

Whitelist "Entity deleted" in WorkflowFailureListener so the resulting
PROCESS_CANCELLED events stay silent, matching the existing pattern for
"Workflow definition deleted" and "Cleanup before redeployment".

Verified locally against Postgres on :10000: created a table, hard-deleted
it, no EntityNotFoundException or NPE in the server log for that entity id.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 5, 2026 10:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (4)

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java:1516

  • cancelInstancesForEntity swallows all FlowableExceptions, which can hide real failures (e.g., DB/connectivity issues) and leave workflow instances running after the entity has been deleted. At minimum, log the exception at DEBUG so operators can diagnose why cancellations didn’t happen.
      try {
        runtimeService.deleteProcessInstance(pi.getId(), reason);
      } catch (FlowableException ignored) {
        // Instance already ending or gone; deleteProcessInstance is idempotent enough here.
      }

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java:1499

  • The Javadoc says this cancels "MainWorkflow" instances, but the query does not filter by process definition key and will cancel any process instance that has relatedEntityId set (including trigger workflows). Update the comment to match actual behavior so callers don’t rely on an incorrect scope guarantee.
  /**
   * Cancel every running MainWorkflow process instance whose relatedEntityId variable matches
   * the given entity id. Called from EntityRepository.hardDelete so downstream nodes do not
   * hit EntityNotFoundException on an entity that no longer exists.
   */

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/MainWorkflowTerminationListener.java:21

  • PR description mentions logging at DEBUG when businessKey is missing, but the implementation returns silently. Adding a DEBUG log here would preserve that intent while avoiding ERROR spam and still keeping production logs clean at default levels.
    if (businessKey == null || businessKey.isBlank()) {
      return;
    }

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java:1775

  • triggerWorkflow now relies on always passing a generated businessKey to avoid null-businessKey races. There are existing WorkflowHandler*Test classes, but none cover triggerWorkflow(...) or cancelInstancesForEntity(...), so this behavior is currently untested and could regress silently.
  public boolean triggerWorkflow(String workflowName) {
    RuntimeService runtimeService = processEngine.getRuntimeService();
    RepositoryService repositoryService = processEngine.getRepositoryService();

    String baseProcessKey = getTriggerWorkflowId(workflowName);

    // Prefer the current workflow definition config to avoid triggering stale process keys left
    // behind by older deployments.
    List<String> configuredTriggerKeys =
        getConfiguredPeriodicTriggerProcessKeys(workflowName, baseProcessKey);
    if (!configuredTriggerKeys.isEmpty()) {
      return triggerProcessDefinitions(runtimeService, configuredTriggerKeys);
    }

    // Legacy fallback: trigger all latest process definitions matching the workflow prefix.
    List<ProcessDefinition> processDefinitions =
        repositoryService
            .createProcessDefinitionQuery()
            .processDefinitionKeyLike(baseProcessKey + "-%")
            .latestVersion()
            .list();
    if (!processDefinitions.isEmpty()) {
      return triggerProcessDefinitions(
          runtimeService, processDefinitions.stream().map(ProcessDefinition::getKey).toList());
    }

    // Fallback to original behavior for non-periodic trigger types.
    try {
      // Trigger BPMN's CallActivity has inheritBusinessKey=true; passing a businessKey
      // here ensures the spawned MainWorkflow inherits a non-null WorkflowInstance UUID.
      runtimeService.startProcessInstanceByKey(baseProcessKey, UUID.randomUUID().toString());
      return true;

The bulk hard-delete path was calling cancelInstancesForEntity per entity in
a loop, which fires one ACT_RU_VARIABLE round-trip per entity even for
subtrees where no entity has a running workflow.

Replace with cancelInstancesForEntities that:
- short-circuits when the engine has zero running instances (cheap COUNT),
- otherwise runs one variableValueEquals-per-id inside a single Flowable
  ProcessInstanceQuery.or()..endOr() block, chunked at 100 to stay inside
  driver bind-parameter limits.

Uses only Flowable's public builder API (no native SQL). Single-entity
cancelInstancesForEntity now delegates to the batched form.

Verified against Postgres on :10000: recursive service hard-delete
(service + 2 databases + schema + 5 tables) returns 200, zero
NullPointerException in server log.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 5, 2026 10:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (3)

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowInstanceListener.java:127

  • This listener now conditionally overwrites businessKey based on UUID parsing. Since downstream code assumes UUID.fromString(businessKey) always succeeds, add a regression test to confirm: (1) caller-supplied UUID businessKeys are preserved, and (2) null/blank/non-UUID keys are backfilled before the UUID.fromString call.
    // Preserve caller-supplied businessKey when it is a valid UUID; otherwise assign a
    // fresh WorkflowInstance UUID. Guarantees the field UUID.fromString parses below.
    String existingBusinessKey = execution.getProcessInstanceBusinessKey();
    if (!isUuid(existingBusinessKey)) {
      updateBusinessKey(execution.getProcessInstanceId());
    }
    UUID workflowInstanceId = UUID.fromString(execution.getProcessInstanceBusinessKey());

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java:1805

  • This change is central to eliminating the null-businessKey race (businessKey is now always passed to Flowable). There isn’t currently test coverage for triggerWorkflow/triggerProcessDefinitions, so a regression could reintroduce null business keys without being caught. Consider adding a focused unit test that stubs RuntimeService and asserts startProcessInstanceByKey is invoked with a non-null UUID businessKey.
      // Trigger BPMN's CallActivity has inheritBusinessKey=true; passing a businessKey
      // here ensures the spawned MainWorkflow inherits a non-null WorkflowInstance UUID.
      runtimeService.startProcessInstanceByKey(baseProcessKey, UUID.randomUUID().toString());

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java:1506

  • The Javadoc says this only cancels “MainWorkflow” instances, but the query matches any running process instance with the namespaced relatedEntityId variable (including trigger processes). Either constrain the query to MainWorkflow process definitions or update the Javadoc so it reflects the actual behavior.
  /**
   * Cancel every running MainWorkflow process instance whose relatedEntityId variable matches
   * the given entity id. Called from EntityRepository.hardDelete so downstream nodes do not
   * hit EntityNotFoundException on an entity that no longer exists.
   */

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

🔴 Playwright Results — workflow failed

Validated commit 6c46382f4eaa663eee9691f7f2397c93b48304c6 in Playwright run 31009549576, attempt 1.

✅ 577 passed · ❌ 0 failed · 🟡 1 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Pipeline and setup failures (1)

  • Playwright performance gate Maximum shard-job elapsed before upload failed (target ≤ 1800 s) — exceeded on 1 shard(s): chromium-03 1977 s.

Performance

Blocking targets: ❌ unmet · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 1h 9m 42s

⏱️ Max setup 3m 11s · max shard execution 18m 33s · max shard-job elapsed before upload 32m 57s · reporting 5s

🌐 199.91 requests/attempt · 2.80 app boots/UI scenario · 7.28% common-shard skew

Optimization targets still in progress:

  • Application boot ratio was 2.8 per UI scenario (1681 boots / 600 scenarios; convergence target: at most 1).
Shard Passed Failed Flaky Skipped Lifecycle failed Lifecycle flaky
✅ Shard chromium-01 136 0 0 0 0 0
✅ Shard chromium-02 136 0 0 0 0 0
🟡 Shard chromium-03 154 0 1 0 0 0
✅ Shard data-asset-rules-01 61 0 0 0 0 0
✅ Shard domain-isolation-01 14 0 0 0 0 0
✅ Shard global-state-01 34 0 0 0 0 0
✅ Shard ingestion-01 1 0 0 0 0 0
✅ Shard reindex-01 2 0 0 0 0 0
✅ Shard search-01 10 0 0 0 0 0
✅ Shard search-rbac-01 29 0 0 0 0 0
🟡 1 flaky test(s) (passed on retry)
  • Pages/Entity.spec.tsDomain Propagation (shard chromium-03, 1 retry)

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

…elete

The cancelInstancesForEntity / cancelInstancesForEntities calls run inside
the entity hard-delete transaction. A stray Flowable engine failure there
would abort a data-plane operation that must not depend on the workflow
engine's health. Wrap both call sites (single-entity and bulk paths) in
try/catch — log at WARN and continue with the actual entity cleanup.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 5, 2026 13:17
@gitar-bot

gitar-bot Bot commented Aug 5, 2026

Copy link
Copy Markdown
Code Review ✅ Approved 3 resolved / 3 findings

Fixes null-businessKey race conditions and silences non-OM workflow log noise by passing the businessKey at process start and batching cancellation handling. No issues found.

✅ 3 resolved
Edge Case: Caller-supplied non-UUID businessKey now fails instead of being replaced

📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowInstanceListener.java:113-117
By making the updateBusinessKey overwrite conditional (WorkflowInstanceListener.java:113-117), a caller-supplied businessKey is now preserved and fed straight into UUID.fromString(...) at line 117. Previously the unconditional overwrite guaranteed a valid random UUID here. If any current or future caller passes a non-UUID businessKey, UUID.fromString throws IllegalArgumentException, which the outer catch swallows at WARN and the WorkflowInstance row is silently never created. All present callers pass valid UUIDs so this is latent, but consider validating the businessKey parses as a UUID (and falling back to a fresh one) to avoid a silent failure mode.

Performance: Per-entity Flowable query in bulk hard-delete loop

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:6961-6968
bulkCleanupReferences runs one createProcessInstanceQuery().variableValueEquals(...).list() per entity (up to 500 per chunk, repeated across every chunk of a subtree delete), issuing a Flowable ACT_RU_VARIABLE round-trip for every entity even when none has a workflow. This contradicts the surrounding bulk-delete design, which deliberately uses single IN-list deletes per chunk (see bulkHardDeleteUsage at 6956-6959) to avoid per-entity round-trips. Consider gating the loop behind a cheaper existence check, or batching the variable query, so a large subtree delete does not fan out into thousands of Flowable queries.

Bug: Unguarded Flowable query can abort entity hard-delete

📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java:1504-1510 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:4944-4949 📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java:1523-1537 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:6961-6964
In cancelInstancesForEntity only deleteProcessInstance is wrapped in try/catch; the createProcessInstanceQuery().list() call and getRuntimeService() are not. This runs inside the cleanup() transaction lambda (EntityRepository.java:4946-4949) right before dao.delete(id), so a transient Flowable/DB failure on the query — exactly the class of transient failure this PR set out to harden against — now propagates out and rolls back the entire hard-delete transaction, blocking entity deletion which previously had no dependency on Flowable. Wrap the whole method body in a try/catch (log at DEBUG/WARN and return) so workflow cancellation is best-effort and never blocks the delete.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar | Powered by Gitar — free for open source

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (4)

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java:1806

  • These changes rely on always passing a non-null businessKey so CallActivity inheritance can't produce null WorkflowInstance IDs. There are existing source-level regression tests for WorkflowHandler; please add a similar test that asserts triggerWorkflow/triggerProcessDefinitions use the startProcessInstanceByKey overload that supplies a businessKey.
    // Fallback to original behavior for non-periodic trigger types.
    try {
      // Trigger BPMN's CallActivity has inheritBusinessKey=true; passing a businessKey
      // here ensures the spawned MainWorkflow inherits a non-null WorkflowInstance UUID.
      runtimeService.startProcessInstanceByKey(baseProcessKey, UUID.randomUUID().toString());
      return true;

openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:6981

  • Same as the single-entity path: this warning log drops the exception stack trace, which makes bulk hard-delete workflow cleanup failures difficult to debug. Pass the exception as the last argument.
          LOG.warn(
              "Failed to cancel workflow instances for {} entities: {}",
              entityIds.size(),
              cancelEx.getMessage());

openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:4957

  • The warning log drops the exception stack trace, which makes it hard to diagnose why Flowable cancellation failed (and this code explicitly swallows failures). Include the exception as the last argument so the stack trace is available when needed.

This issue also appears on line 6978 of the same file.

                  LOG.warn(
                      "Failed to cancel workflow instances for entity {}: {}",
                      entityInterface.getId(),
                      cancelEx.getMessage());

openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowHandler.java:1506

  • The Javadoc says this cancels "MainWorkflow" instances, but the query is not constrained by process definition key; it cancels any running process instance with the relatedEntityId variable set. Update the comment to match the actual behavior (or add a definition-key filter if MainWorkflow-only is intended).
  /**
   * Cancel every running MainWorkflow process instance whose relatedEntityId variable matches
   * the given entity id. Called from EntityRepository.hardDelete so downstream nodes do not
   * hit EntityNotFoundException on an entity that no longer exists.
   */

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants