Fix 13785 vm ha error state

[Dogface2k]

Description

Fixes #13785.

Problem

When deployment of an HA-enabled VM fails, the VM can be left in Error state. CloudStack could subsequently pass that VM to HighAvailabilityManagerImpl.scheduleRestart(...) and create HA work for it. If the work item was created while the VM was already in Error, the worker's existing state/update checks still matched and HA processing could continue into host investigation, fencing, forced-stop, storage and restart handling.

This is inconsistent with the VM lifecycle: Error represents a failed or inconsistent VM state and there is no normal start transition from Error.

Root cause

The shared HA restart entry point did not reject VMs in Error state, and the HA worker did not independently reject an already-persisted work item when the current VM state was Error.

Change

This PR enforces the lifecycle invariant at both boundaries:

• scheduleRestart(...) returns before any HA work, forced stop, orchestration or alert side effect is attempted for an Error-state VM.
• restart(...) treats an already-queued HA restart for an Error-state VM as complete before host investigation, fencing, storage checks or VM start handling.

The execution-time check also covers work persisted before an upgrade and the race where a VM enters Error after scheduling but before the HA worker processes it.

The change deliberately does not make Error startable, rewrite the VM state, suppress an exception after the operation has begun, or alter HA behaviour for valid VM states. There are no API, database, configuration or UI changes.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate)

Not applicable; this is management-server HA behaviour with no UI change.

How Has This Been Tested?

Focused regression tests were added to HighAvailabilityManagerImplTest:

• scheduleRestartVMInErrorState verifies that an Error-state VM cannot create HA work or invoke orchestration or alert side effects.
• restartVMInErrorState verifies that already-queued work returns without host lookup, alerting, user-VM start handling, volume restart checks or direct work-step mutation.

The final branch diff was audited against Apache main: it contains only the HA implementation change and its regression tests—two files, 39 additions and no unrelated files.

Upstream GitHub Actions checks are currently waiting for maintainer approval, so no passing CI result is claimed in this description.

How did you try to break this feature and the system with this change?

• Covered the scheduling path with no host setup, ensuring the old null-host forced-stop path is not reached for an Error VM.
• Covered an HA work item that already exists in the database, rather than relying only on prevention at scheduling time.
• Reviewed every production scheduleRestart(...) call site. Callers that already restrict execution to Starting, Running or Stopping remain unchanged; the central guard covers the unfinished-work path that can pass a VM whose current state has already become Error.
• Kept ForceHA, valid-state host recovery, migration timeout recovery, host maintenance/degraded handling and out-of-band stop recovery unchanged.

[codecov]

Codecov Report

❌ Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 19.65%. Comparing base (4f11707) to head (0115c1f).

Files with missing lines	Patch %	Lines
...java/com/cloud/ha/HighAvailabilityManagerImpl.java	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #13808   +/-   ##
=========================================
  Coverage     19.65%   19.65%           
+ Complexity    19792    19791    -1     
=========================================
  Files          6368     6368           
  Lines        574881   574887    +6     
  Branches      70351    70353    +2     
=========================================
+ Hits         112970   112974    +4     
+ Misses       449639   449638    -1     
- Partials      12272    12275    +3     

Flag	Coverage Δ
uitests	3.41% <ø> (ø)
unittests	20.92% <83.33%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[DaanHoogland]

@Dogface2k , you are fixing on main, but the bug is reported in 4.20. Can you rebase on that branch? We will merge forwards all fixes on older versions (i.e to 22 and main…)

[Dogface2k]

@Dogface2k , you are fixing on main, but the bug is reported in 4.20. Can you rebase on that branch? We will merge forwards all fixes on older versions (i.e to 22 and main…)

Being resolved, agents new environment didn't check the head it was suppose to be building from.