Run the SOS test harness on Helix - #6002
Open
steveisok wants to merge 1 commit into
Open
Conversation
steveisok
force-pushed
the
steveisok-sos-helix-integration
branch
from
September 3, 2026 12:31
3ac4ddb to
df3d87e
Compare
Add deterministic capture-family sharding, a shared SOS Helix payload, cross-platform launchers, and pipeline scheduling while preserving the lower harness layers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
steveisok
force-pushed
the
steveisok-sos-helix-integration
branch
from
September 4, 2026 02:12
df3d87e to
c6775f5
Compare
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The new Helix MSBuild project has a payload-staging target gated by a likely incorrect Helix queue property name, which can prevent correlation payload staging and break Helix submissions.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
eng/helix/SOS.Tests.Helix.proj — PrepareSOSHelixPayload is currently conditioned on the MSBuild property HelixTargetQueue, but the… |
What changed in this PR
This PR integrates SOS.Tests with Helix execution across Windows, Linux (including musl), and macOS by adding deterministic sharding and Helix payload/launcher infrastructure, while keeping local CI test coverage intact (skipping only duplicate local SOS.Tests runs when Helix submission is enabled).
Changes:
- Add capture-family-based deterministic sharding (stable hash) and shard filtering throughout the SOS test matrix.
- Introduce Helix submission project + OS-specific launchers that stage a shared correlation payload and run named shard work items, with writable overlays for executables/runtime/native bits and enriched artifact/log uploads.
- Update pipeline templates to optionally submit
SOS.Teststo Helix (including Helix-only legs for cross-built architectures) while preserving other local test execution.
| File | Description |
|---|---|
| src/tests/SOS.Tests/WindowsTheoryAttribute.cs | Allow empty theory data when matrix partitioning/sharding is in effect. |
| src/tests/SOS.Tests/SosTheoryAttribute.cs | Same empty-matrix behavior adjustment for cross-platform theories. |
| src/tests/SOS.Tests/SosReplayAttribute.cs | Route replay output under a harness-owned upload root when provided (Helix-friendly). |
| src/tests/SOS.Tests/TestMatrices.cs | Apply shard filtering consistently; introduce unsharded helpers for composition. |
| src/tests/SOS.Tests/TestConfigShardTests.cs | Add unit tests for sharding/environment parsing and hash stability. |
| src/tests/SOS.Tests/UnixPayloadTests.cs | Add tests for writable overlay behavior and upload-root routing on Unix. |
| src/tests/SOS.Tests/SOS.Tests.csproj | Add a pipeline-controlled “skip SOS.Tests locally” switch; avoid chmod step when tests are skipped. |
| src/tests/SOS.Tests/README.md | Document Helix sharding and new harness environment variables and execution model. |
| src/tests/SOS.TestHarness/TestConfig.cs | Implement capture-family key, stable hash, shard selection parsing, and shard filtering entry points. |
| src/tests/SOS.TestHarness/SnapshotStore.cs | Support executable overlays and enforce “prebuilt-only” mode for Helix payload execution. |
| src/tests/SOS.TestHarness/RepoLayout.cs | Add environment-variable overrides for repo/runtime/native/executable/scratch roots and directory resolution helper. |
| src/tests/SOS.TestHarness/HostDiagnostics.cs | Add upload-root routing for host crash dump diagnostics. |
| src/tests/SOS.TestHarness/ToolPaths.cs | Add dbgeng root override and use configurable dotnet root for host runtime discovery. |
| global.json | Add Microsoft.DotNet.Helix.Sdk MSBuild SDK reference. |
| eng/helix/SOS.Tests.Helix.proj | New Helix submission project: defines shard work items and stages the correlation payload. |
| eng/helix/sos/run-sos-tests.sh | New Unix Helix work-item launcher: overlays runtime/executables/native as needed, configures LLDB, runs tests with MTP reporters. |
| eng/helix/sos/run-sos-tests.cmd | New Windows Helix work-item launcher: sets harness env, manages signature-check toggling, runs tests with MTP reporters. |
| eng/helix/sos/debuggee-entitlements.plist | New macOS entitlements used for codesigning staged debuggee executables. |
| eng/pipelines/build.yml | Add parameters/logic to submit SOS.Tests to Helix and support additional-architecture builds (macOS) and helix-only legs. |
| diagnostics.yml | Wire new Helix-enabled build/test legs for public CI (Windows/Linux/musl/macOS), including helix-only cross-arch scenarios. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+109
to
+112
| <Target Name="PrepareSOSHelixPayload" | ||
| DependsOnTargets="ValidateSOSHelixConfiguration" | ||
| BeforeTargets="CoreTest" | ||
| Condition="'$(HelixTargetQueue)' == ''"> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #6001
This is the top (layer 4) PR in the SOS harness stack. It adds the remaining SOS.Tests Helix integration on top of the coverage-migration layer without registering stack metadata.
Changes
HELIX_WORKITEM_UPLOAD_ROOTin launchers to the harness-ownedSOSHARNESS_UPLOAD_ROOT; production harness and test code do not readHELIX_*variables.Validation
SOS.TestHarnessandSOS.Testsdirectly.PrintSOSHelixWorkItemswithout submission for Windows x64, Linux x64, Alpine/musl x64, macOS x64, and macOS arm64, including a single-work-item smoke selection.0a60d3a638186671a8862110c25d27499931742eand fixed all high-confidence findings.