AB#12334: Prototype duration-balanced host build tests on Helix

[mmitche]

Live validation runs

• First opt-in run: build 1563210 - exposed an incorrect Linux NUnit filter (Name = BuildTest matched zero tests); Windows validation continues in that run.
• Corrected opt-in run at c41b6aa: build 1563289
• Metadata-corrected opt-in run at 112d251: build 1563300. The first run showed 1,504 Windows tests packed into 41 fallback work items; submission then failed because the project did not preserve generated Command metadata.
• Payload-partitioned opt-in run at 911c6b3: build 1563332. Build 1563300 successfully discovered 167 Linux tests in 5 work items, then exposed a 13.57 GiB monolithic payload and the Helix SDK in-memory ZIP limit; payloads are now capped at 1 GiB and the duplicated NuGet cache is omitted.
• Device-trimmed payload run at 39d7209: build 1563387. Build 1563332 showed that the prepared SDK included a device-only Android 29 x86_64 system image larger than 1 GiB; emulator images, skins, and sources are now excluded from host-only payloads.
• Measured payload composition: build 1563446 showed 7.75 GiB Linux / 8.69 GiB Windows. SDK docs/extras, Java.Interop source, unrelated test assemblies, and device-only content are now removed.
• Host-only iteration run at c6114de: build 1563474, using skipTests=true only to accelerate prototype validation; the final parity run will retain all existing jobs. Results: 50 Linux payloads / 5.60 GiB and 53 Windows payloads / 6.53 GiB; submission took 3-4 minutes. Linux ran 167 tests in five batches (four passed, one hit the 45-minute timeout). Windows ran all 1,504 tests, but forcing CONFIGURATION=Release caused CoreCLR Debug-output assertions to fail and result files were not downloaded.
• Environment/results corrected run at b0bec43: build 1565007. This removes the configuration override, sets RUNNINGONCI, uses Server 2022, raises first-run timeout to 90 minutes, and downloads TRX plus diagnostics. Linux passed all 167 tests (150 passed, 17 skipped) across five fallback batches; TRX, diagnostics, and timing artifacts published successfully. Windows published all 1,504 TRX results and reduced failures from 194 to 17; remaining failures identified missing NDK/local NuGet inputs plus one packaging race.
• Timing-balanced payload-complete run at 8377807, consuming timings from build 1565007: build 1565452. Linux passed all 167 outcomes again across 8 timing-based work items. Windows reduced to a single repeated multi-RID packaging failure; NDK/local-feed failures were eliminated. Observed timing-based medians were ~21.5 minutes Linux and ~23.4 minutes Windows with the initial 2.5 factor.
• Calibrated timing run at 75a5cbd, consuming build 1565452 timings: build 1565829. Effective parallelism is now 1.65, Linux omits Windows-only NDK/local packages, and Windows inner builds are serialized. Linux passed all 167 outcomes in 12 work items with test-duration P50 12.7m, P90 16.0m, max 18.8m. Windows published all 1,504 outcomes across 98 work items with P50 16.9m; only two file-sharing tests failed.
• Isolated-test validation at 44ce9e0, consuming build 1565829 timings: build 1566082. The two repeatedly racing parameterized tests are now non-parallel. Linux remained green. Windows ran all 1,504 outcomes in 101 work items with P50 15.6m, P90 20.2m; only the multi-RID package race and one transient file-lock test remained.
• Final host-only validation at 5ed46cc, consuming build 1566082 timings: build 1566181. Runtime-identifier inner builds are serialized and the move/rebuild test is isolated. Linux passed. Windows published all 1,504 outcomes in 85 work items; only the same AllForegroundServiceTypes multi-RID APK race remained.
• Targeted manifest validation at 280a2ed, consuming build 1566181 timings: build 1566277. Only that test now disables runtime-identifier parallelism. Linux passed. Windows again had only the longest-named AllForegroundServiceTypes case fail, confirming path length rather than cross-test parallelism.
• Short-path validation at 8fbd742, consuming build 1566277 timings: build 1566510. Test staging now uses the short work-item root and is archived afterward for diagnostics. Linux passed. Windows completed in ~53 minutes of Helix execution; the long-name path issue was reduced, leaving the same multi-RID test and one concrete apkdiff 30-second timeout.
• Stabilized host validation at aa0f9d9, consuming build 1566510 timings: build 1566817. The manifest test now uses one ABI and apkdiff gets a 120-second CI timeout. Linux passed. Windows had only the foreground-service case fail; diagnostics showed managed copy succeeded but native zipalign could not open the long relative APK path.
• Path-limit validation at d5f35a7, consuming build 1566817 timings: build 1567161. The foreground-service project now uses a compact explicit directory.

This run preserves the existing Windows/Linux jobs in the same build for inventory and outcome parity comparison. Results will be recorded here after both the baseline jobs and Helix work items complete.

Azure DevOps tracking - Parent: DNCENG Epic 11483 - Initiative: DNCENG Epic 12333 - Improve dotnet/android P80 PR build time - Implementation task: AB#12334 - Investigate moving dotnet/android tests to Helix ## Summary Prototype moving host-only Xamarin.Android.Build.Tests fan-out from fixed Azure Pipelines shards to fine-grained Helix work items. This is additive and guarded: the existing Windows and Linux jobs remain unchanged, and the new stage is omitted unless enableHostBuildTestsHelixPrototype is enabled. ## Duration-balanced work items - Default target: 15 estimated wall-clock minutes per work item. - Configure with hostBuildTestsHelixTargetMinutes. - Test discovery uses the repository's existing dotnet-test-slicer NUnit support. - Timing data uses the slicer's existing balance.xml format. - Work items use deterministic best-fit-decreasing duration packing. - A test whose duration exceeds the target remains in an isolated work item. - Missing tests use the average of matching historical timings. - With no matching history, generation explicitly reports count-fallback and uses hostBuildTestsHelixFallbackTestDurationSeconds rather than claiming count slicing is duration-balanced. - hostBuildTestsHelixDurationParallelism (default 2.5) translates summed per-test elapsed time to expected work-item wall time while preserving the current NUnit worker behavior. ## Architecture and data flow For each host OS, Azure Pipelines performs setup once and stages a Helix correlation payload containing: - the prepared repository-local .NET SDK and Android workload packs; - Android SDK and JDK; - NuGet and Gradle caches; - test assemblies and required repository test/build inputs; - the existing apkdiff tool. Small per-work-item payloads contain only NUnit runsettings, a native Windows/Linux command script, and generation metadata. Work items run on host queues that do not require an Android device: - Windows.Amd64.VS2026.Open - Ubuntu.2404.Amd64.Open Windows currently covers the full host test assembly. Linux initially covers the existing long, unsliced BuildTest group. Each work item runs in its own Helix sandbox and directs temporary/test output to work-item-specific paths. Failures preserve the dotnet test exit code. Uploaded diagnostics include TRX, console output, NUnit/MSBuild test attachments and binlogs, generation metadata, and crash dumps when produced. After completion, successful TRX timings are converted back to balance.xml and published as OS-specific artifacts. A subsequent run can consume them by setting hostBuildTestsHelixTimingBuildId to the prior build ID. ## Opt in Queue the public pipeline with: - enableHostBuildTestsHelixPrototype=true - hostBuildTestsHelixTargetMinutes=15 - optionally hostBuildTestsHelixTimingBuildId=<prior build ID> The per-work-item timeout defaults to 00:45:00 and is configurable through hostBuildTestsHelixWorkItemTimeout. Example: powershell az pipelines run ` --organization https://dev.azure.com/dnceng-public ` --project public ` --id <dotnet-android-public-definition-id> ` --parameters enableHostBuildTestsHelixPrototype=true hostBuildTestsHelixTargetMinutes=15 ## Validation - PowerShell generator tests pass, covering deterministic packing, explicit no-history fallback, partial history, oversized tests, zero-duration TRX entries, escaping, generated payloads, and Windows/Linux commands. - Both changed YAML files parse successfully. - The Helix MSBuild project parses, restores, and evaluates successfully with the repository SDK and synthetic work items. - Synthetic end-to-end payload preparation succeeds. - Historical-duration simulation produced: - Windows: 1,910 timed tests into 52 work items, estimated 10.6-15 minutes. - Linux BuildTest: 239 timed tests into 8 work items, estimated 8.8-15 minutes. - git diff --check passes. ## Known limitations / first experiment - No real Helix submission has been run yet; the first opt-in pipeline run should validate queue image prerequisites, correlation payload upload/download size, and actual setup amortization. - The 2.5 timing parallelism factor is an initial calibration and should be adjusted from observed Helix wall times. - To control correlation payload size, the source copy excludes src/Mono.Android; tests consume that product through the staged workload packs. The first run should confirm no host test requires additional source-only inputs. - Existing Azure Pipelines jobs intentionally remain enabled during comparison, so the first run duplicates coverage. - Linux initially moves only BuildTest; the other Linux smoke groups remain on their existing path.

[jonathanpeppers]

@mmitche I wanted to look into this, but I think the "APK Test" lanes might be suitable for Helix to prototype first.

The NUnit (ancient!) MSBuild integration tests we have, would require installing .NET SDK w/ the currently built Android workload on Helix -- and I'm not sure if Helix can do that or not. Those NUnit tests create projects, build them, run them on eumulators, and assert the output.

@simonrozsival enabled a simple Android arm32 job here, so Helix is being used & working in one place so far:

• [ci] Add MAUI full R2R android-arm Helix test #12020