Skip to content

fix(csharp-ci): disable MSBuild node reuse on shared hetzner runners - #40

Open
monsieurleberre wants to merge 2 commits into
devfrom
fix/msbuild-node-reuse-shared-runners
Open

monsieurleberre wants to merge 2 commits into
devfrom
fix/msbuild-node-reuse-shared-runners

Conversation

@monsieurleberre

Copy link
Copy Markdown
Contributor

Root cause

hetzner-3 and hetzner-4 are two self-hosted runners sharing one host and
one OS user. When a job finishes, the runner's orphan-process cleanup kills
leftover MSBuild "node" processes belonging to that job — but it doesn't
distinguish those from idle out-of-proc MSBuild nodes that a different,
still-running job on the sibling runner registered for node reuse and is
still relying on. That sibling job's build then dies mid-flight with
MSB4166 (node terminated unexpectedly).

Evidence (MSBuild source)

  • XMake.cs ProcessNodeReuseSwitch — MSBUILDDISABLENODEREUSE=1 forces node reuse off regardless of the /nr switch or project settings.
  • CommunicationsUtilities.cs:657 — node-reuse eligibility check reads this env var before a node will register itself for reuse.
  • NodeProviderOutOfProcBase.cs:265-307 — the out-of-proc node provider's attach/reuse path is skipped entirely when node reuse is disabled, so no shared node is ever registered or reattached to.

With the var set, MSBuild never registers/reattaches to a shared out-of-proc
node, so a sibling job's cleanup can no longer kill a node this job depends
on.

What changed

  • .github/workflows/csharp-ci.yaml: added MSBUILDDISABLENODEREUSE: '1' to
    the build-and-test job's env: (the only job in this file that runs
    dotnet/MSBuild and can land on the self-hosted hetzner-3/hetzner-4
    pool — normalize, coverage-output, matrix-output, matrix-comment
    and pack are all pinned to ubuntu-latest).
  • scala-ci.yaml was checked and left untouched: it's pure Scala/sbt, it
    never shells out to dotnet/MSBuild anywhere, so it isn't exposed to this
    bug.
  • csharp-publish-public.yaml was checked too: its publish job is
    hardcoded to runs-on: ubuntu-latest and can never land on the shared
    self-hosted pool, so it's out of scope for this fix.
  • CHANGELOG.md: added the [Unreleased] entry.

Background

This is the follow-up half of a fix already shipped in the caller repo.
peacefulstudio/daml-codegen-csharp-internal hit this as issue
#1086,
and PR #1312
fixed it for that repo's own inline CI jobs. But that repo's CI also calls
this repo's csharp-ci.yaml@v2 and scala-ci.yaml@v2 as reusable
workflows — and env vars set by a workflow_call caller do not propagate
into the called workflow's own job env:, so the #1312 fix never reached
these two reusable workflows. This PR applies the same fix inside
github-actions itself.

Reaching consumers

This only takes effect for consumers once the v2 tag moves to a release
that includes this commit — merging to dev alone does not change what
@v2 resolves to. I'm not moving the tag or cutting a release myself; per
this repo's CONTRIBUTING.md#release-process, that's the maintainer's step
after merge (git tag -a v2.X.Y ... && git tag -fa v2 ... && git push origin v2.X.Y v2).

The hetzner-3/hetzner-4 self-hosted runners share one host and OS user.
When one job finishes, its orphan-process cleanup kills idle MSBuild
node processes that a different, still-running job on the sibling
runner had left attached to for node reuse, so that job's build dies
mid-flight with MSB4166 (node terminated unexpectedly).

Set MSBUILDDISABLENODEREUSE: '1' on the build-and-test job so MSBuild
never registers or reattaches to a shared out-of-proc node in the
first place. scala-ci.yaml runs no dotnet/MSBuild and needs no change.

Fixes the reusable-workflow half of peacefulstudio/canton-dotnet-sdk-internal#1086;
that repo's own inline jobs were already fixed in PR #1312, but env
vars set by a workflow_call caller do not propagate into this
workflow's own job env.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-25T00:18:20.181470Z dd27129 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant