server: skip HA restart for VMs in Error state

[Dogface2k]

Description

Fixes #13785.

Problem

When deployment of an HA-enabled VM fails, the VM can be left in Error state. CloudStack could subsequently pass that VM to HighAvailabilityManagerImpl.scheduleRestart(...) and create HA work for it. If the work item was created while the VM was already in Error, the worker's existing state and update checks still matched and HA processing could continue into host investigation, fencing, forced-stop, storage and restart handling.

This is inconsistent with the VM lifecycle: Error represents a failed or inconsistent VM state and there is no normal start transition from Error.

Root cause

The shared HA restart entry point did not reject VMs in Error state, and the HA worker did not independently reject an already-persisted work item when the current VM state was Error.

Change

This PR enforces the lifecycle invariant at both boundaries:

• scheduleRestart(...) returns before any HA work, forced stop, orchestration or alert side effect is attempted for an Error-state VM.
• restart(...) treats an already-queued HA restart for an Error-state VM as complete before host investigation, fencing, storage checks or VM start handling.

The execution-time check also covers work persisted before an upgrade and the race where a VM enters Error after scheduling but before the HA worker processes it.

The change deliberately does not make Error startable, rewrite the VM state, suppress an exception after the operation has begun, or alter HA behaviour for valid VM states. There are no API, database, configuration or UI changes.

The branch is based directly on the current 4.20 head so that the fix can be merged forward into later release branches.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate)

Not applicable; this is management-server HA behaviour with no UI change.

How Has This Been Tested?

Focused regression tests were added to HighAvailabilityManagerImplTest:

• scheduleRestartVMInErrorState verifies that an Error-state VM cannot create HA work or invoke orchestration or alert side effects.
• restartVMInErrorState verifies that already-queued work returns without host lookup, alerting, user-VM start handling, volume restart checks or direct work-step mutation.
• restartVMNotInErrorStateContinuesProcessing verifies that the new worker guard does not stop normal processing for a valid non-Error VM state.

The final branch diff was audited against the current Apache 4.20 head. It contains one commit and changes only the HA implementation and its regression test class: two files, 58 additions and no deletions.

The upstream checks must rerun after the rebase, so no new passing CI result is claimed here yet.

How did you try to break this feature and the system with this change?

• Covered the scheduling path with no host setup, ensuring the old null-host forced-stop path is not reached for an Error VM.
• Covered an HA work item that already exists in the database, rather than relying only on prevention at scheduling time.
• Covered the false side of the worker state guard to prove normal HA processing still continues for a non-Error VM.
• Reviewed the production scheduleRestart(...) call paths and retained all valid-state HA behaviour.
• Kept ForceHA, valid-state host recovery, migration timeout recovery, host maintenance/degraded handling and out-of-band stop recovery unchanged.
• Added or modified no workflow files.