fix(csharp-ci): disable MSBuild node reuse on shared hetzner runners - #40
Open
monsieurleberre wants to merge 2 commits into
Open
monsieurleberre wants to merge 2 commits into
monsieurleberre wants to merge 2 commits into
Conversation
The hetzner-3/hetzner-4 self-hosted runners share one host and OS user. When one job finishes, its orphan-process cleanup kills idle MSBuild node processes that a different, still-running job on the sibling runner had left attached to for node reuse, so that job's build dies mid-flight with MSB4166 (node terminated unexpectedly). Set MSBUILDDISABLENODEREUSE: '1' on the build-and-test job so MSBuild never registers or reattaches to a shared out-of-proc node in the first place. scala-ci.yaml runs no dotnet/MSBuild and needs no change. Fixes the reusable-workflow half of peacefulstudio/canton-dotnet-sdk-internal#1086; that repo's own inline jobs were already fixed in PR #1312, but env vars set by a workflow_call caller do not propagate into this workflow's own job env.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
hetzner-3andhetzner-4are two self-hosted runners sharing one host andone OS user. When a job finishes, the runner's orphan-process cleanup kills
leftover MSBuild "node" processes belonging to that job — but it doesn't
distinguish those from idle out-of-proc MSBuild nodes that a different,
still-running job on the sibling runner registered for node reuse and is
still relying on. That sibling job's build then dies mid-flight with
MSB4166(node terminated unexpectedly).Evidence (MSBuild source)
XMake.csProcessNodeReuseSwitch—MSBUILDDISABLENODEREUSE=1forces node reuse off regardless of the/nrswitch or project settings.CommunicationsUtilities.cs:657— node-reuse eligibility check reads this env var before a node will register itself for reuse.NodeProviderOutOfProcBase.cs:265-307— the out-of-proc node provider's attach/reuse path is skipped entirely when node reuse is disabled, so no shared node is ever registered or reattached to.With the var set, MSBuild never registers/reattaches to a shared out-of-proc
node, so a sibling job's cleanup can no longer kill a node this job depends
on.
What changed
.github/workflows/csharp-ci.yaml: addedMSBUILDDISABLENODEREUSE: '1'tothe
build-and-testjob'senv:(the only job in this file that runsdotnet/MSBuild and can land on the self-hostedhetzner-3/hetzner-4pool —
normalize,coverage-output,matrix-output,matrix-commentand
packare all pinned toubuntu-latest).scala-ci.yamlwas checked and left untouched: it's pure Scala/sbt, itnever shells out to
dotnet/MSBuild anywhere, so it isn't exposed to thisbug.
csharp-publish-public.yamlwas checked too: itspublishjob ishardcoded to
runs-on: ubuntu-latestand can never land on the sharedself-hosted pool, so it's out of scope for this fix.
CHANGELOG.md: added the[Unreleased]entry.Background
This is the follow-up half of a fix already shipped in the caller repo.
peacefulstudio/daml-codegen-csharp-internalhit this as issue#1086,
and PR #1312
fixed it for that repo's own inline CI jobs. But that repo's CI also calls
this repo's
csharp-ci.yaml@v2andscala-ci.yaml@v2as reusableworkflows — and env vars set by a
workflow_callcaller do not propagateinto the called workflow's own job
env:, so the #1312 fix never reachedthese two reusable workflows. This PR applies the same fix inside
github-actionsitself.Reaching consumers
This only takes effect for consumers once the
v2tag moves to a releasethat includes this commit — merging to
devalone does not change what@v2resolves to. I'm not moving the tag or cutting a release myself; perthis repo's
CONTRIBUTING.md#release-process, that's the maintainer's stepafter merge (
git tag -a v2.X.Y ... && git tag -fa v2 ... && git push origin v2.X.Y v2).