Skip to content

Run every gate a second time on a machine that has never seen this #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tamnd merged 2 commits into from
Sep 5, 2026
Merged

Run every gate a second time on a machine that has never seen this #38

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
db1f5db
Run every gate a second time on a machine that has never seen this
tamnd Sep 5, 2026
da06564
Add what the cold job costs on the hosted runners
tamnd Sep 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 49 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,8 +186,8 @@ jobs:
# a blueprint is transcribed by one. Both are enforced the same way, by regenerating the file and
# comparing it against what is committed, so both are exactly as true as that comparison is. This
# job goes and hand edits the generated files on purpose and requires the build to object to each
# one by name. Two of its seven cases change nothing, because a harness that failed on everything
# would otherwise report a clean sweep.
# one by name. Three of its sixteen cases expect a green build, because a harness that failed on
# everything would otherwise report a clean sweep.
tamper:
runs-on: ubuntu-latest
steps:
Expand All @@ -197,6 +197,53 @@ jobs:
global-json-file: global.json
- run: dotnet run --project tools/xray -- check --selftest

# A cold machine is not a slower warm machine, it is a different machine. Every other job here
# starts from a runner image with an SDK on it, a NuGet folder, a home directory and a package
# manager that has already been used. A reader starts from none of that, and so does a container.
# This job takes all of it away on all four platforms and runs the gates again.
#
# It has found three things a warm run cannot. A bare Ubuntu image cannot start .NET at all until
# libicu is installed, which dotnet-install.sh says it will not do for you. global.json names
# 10.0.100 but rolls forward, so every machine anybody had used resolved a 10.0.4xx, and the two
# feature bands disagree about the working directory of a file run with dotnet run. And the
# regeneration self test was copying a lesson without the two Directory.Build.props files above
# it, so it had been building its cases with the SDK's settings rather than this repository's.
#
# The steps are the same script the fleet and a reader can run, rather than a copy of it kept in
# a YAML file where nobody would notice it drifting.
cold:
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
rid: linux-x64
- os: ubuntu-24.04-arm
rid: linux-arm64
- os: windows-latest
rid: win-x64
- os: macos-latest
rid: osx-arm64
runs-on: ${{ matrix.os }}
timeout-minutes: 30
steps:
- uses: actions/checkout@v7
# No setup-dotnet in this job, on purpose. It is the one step every other job starts with and
# the one thing a cold machine does not have.
#
# The Linux legs run in a bare ubuntu:24.04 with the checkout mounted read only, so the
# container has to install everything down to ca-certificates, and cannot quietly write build
# output back into the checkout. The other two are the same script with no container: Windows
# containers are a whole operating system image rather than a directory tree, and the macOS
# runners cannot run containers at all. On those two the cold part is an empty HOME,
# DOTNET_ROOT, NuGet folder and artifact cache, and a path with no dotnet on it.
- if: runner.os == 'Linux'
run: docker run --rm -v "$PWD:/src:ro" ubuntu:24.04 bash /src/docs/probes/cold-start.sh /src
- if: runner.os == 'macOS'
run: bash docs/probes/cold-start.sh
- if: runner.os == 'Windows'
run: pwsh -File docs/probes/cold-start.ps1

# The pin moves to .NET 11 at general availability on 10 November 2026. Until then this job
# answers, on every pull request, whether the tooling still builds on the preview it is going
# to be pinned to. It is allowed to fail, because a broken preview is not a broken repository.
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ Every chapter therefore produces two things. The chapter teaches, in prose and p

**The one click environment is built and its cost is measured, except for the part GitHub owns.** The [E0 image](docs/probes/codespaces-cost.md) carries the pinned SDK, lldb, the diagnostics tools and a checked JIT that produces a real `JitDump` with nothing installed by the reader. It rebuilds from nothing in 51 seconds, it is 0.70 GB to download, a reader's first command in it takes 4.5 seconds and the pilot lesson takes 11.8. What is not measured is how long a real Codespace takes to appear, so the question of whether the local path should be the recommended default is still open rather than answered.

**A cold machine is a different machine, and three defects were only visible on one.** Every gate now runs a second time on all four platforms with nothing installed: no SDK on the path, no NuGet folder, no artifact cache, no home directory, and on Linux a bare `ubuntu:24.04` with the checkout mounted read only. [The probe is written up here](docs/probes/cold-start.md), and it found three things. A bare Ubuntu image cannot start .NET at all until `libicu` is installed, and the installer says in its own help that it will not do that for you. `global.json` names the `10.0.1xx` band but rolls forward, so every machine anybody had used resolved a `10.0.4xx`, and the two bands disagree about the working directory of a file run with `dotnet run`, which four files here were relying on by luck. And the regeneration self test had been copying a lesson without the two `Directory.Build.props` files above it, so it was building its cases with the SDK's settings rather than this repository's. All three are fixed, and all three are now gated rather than remembered.

## What "compatible" is allowed to mean

Nobody has ever built a fully compatible CLI that is not CoreCLR, and a project that claims otherwise gets taken apart in an afternoon, so the tiers are here rather than in a footnote.
Expand Down
147 changes: 147 additions & 0 deletions docs/probes/cold-start.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Probe: building the book on a machine that has never seen it

**Question.** Does the whole pipeline work on a machine with nothing on it, on all four platforms, or does it only work on the machines it was written on?

**Answer. It did not, and three separate things were wrong.** All three are now fixed, all three are now gated, and the cold run is a job in CI on all four platforms rather than something a person remembers to do.

**Measured on 6 September 2026.** Linux in a bare `ubuntu:24.04` image with nothing added, macOS and Windows on real machines with everything the build could inherit taken away.

## Why this was worth measuring

Every other job in CI starts from a runner image that has an SDK on it, a NuGet folder, a home directory the SDK has already written to, and a package manager somebody has already used. Every machine this project has been written on is the same, only more so. A reader has none of that, and neither does a container, which is what the one click environment is.

The failure mode this is looking for is not a slow first run. It is a build that passes everywhere it is tried and is wrong, because the thing that would have made it fail is sitting on every machine that tried it. That is not a hypothetical, and this probe found three of them.

## What is taken away

| Thing | Warm machine | Here |
|---|---|---|
| The SDK | on the path already | installed into an empty directory, and the path has no other one on it |
| Which SDK | whatever `global.json` rolls forward to | the version `global.json` literally names |
| NuGet packages | a populated folder in the home directory | an empty folder |
| The artifact cache | whatever previous runs left | an empty directory |
| The home directory | years of state | created a minute ago |
| System libraries | an image with development tools on it | `ubuntu:24.04` with nothing added |
| The checkout | the working tree, with its build output in it | a copy with no `obj`, no `bin` and no `.git`, and on Linux the original is mounted read only |

## The first thing: a bare Ubuntu image cannot start .NET at all

`dotnet --version` in a fresh `ubuntu:24.04` does not print a version. It aborts before reaching any of this project's code.

```
Process terminated. Couldn't find a valid ICU package installed on the system.
Please install libicu using your package manager and try again.
at System.Globalization.GlobalizationMode.GetInvariantSwitchValue()
at System.Globalization.GlobalizationMode+Settings..cctor()
```

.NET links against the system ICU on Linux, and the image does not ship one. The installer will not help, and says so in its own help text: `dotnet-install.sh` does not resolve dependencies. So the first thing a container needs is not the SDK, it is `ca-certificates`, `curl` and `libicu`, and none of the four hosted runners would ever have told us that, because all four have them.

The package is named after the ICU release rather than the distribution, so it is `libicu74` on 24.04 and something else on the next one. The script looks the name up instead of writing it down.

## The second thing: `global.json` names an SDK nothing had ever run on

```json
{ "sdk": { "version": "10.0.100", "rollForward": "latestFeature" } }
```

`rollForward: latestFeature` means a machine that already has a 10.0.4xx uses it. Every developer machine and every hosted runner has one, so every run anybody had ever done was on the 400 band. A reader who follows the installer with `--jsonfile global.json` gets exactly 10.0.100, because the installer takes the version and not the roll forward rule.

Those two bands disagree about something this repository was relying on. `dotnet run some/file.cs` does not promise the working directory of the file it runs. On 10.0.400 the file inherits the working directory the SDK was started with. On 10.0.100 it can get the directory the file itself is in.

Four files were reading a path relative to the working directory. A boss fight lives in `boss/` and reads `../lesson.cs` under one of those and `lesson.cs` under the other, and which one was right was decided by whichever SDK the machine had resolved. Nothing was wrong with the code. Nothing was wrong with the test. The two had only ever met on one of the two SDKs this repository accepts.

## The third thing: the self test was building with the wrong settings

The regeneration gate proves itself by copying a lesson somewhere temporary, breaking the copy on purpose, and requiring the build to refuse it. The copy took the lesson directory and nothing above it, and what is above it is two `Directory.Build.props` files.

So the copy was compiled with the SDK's defaults rather than this repository's: a different target framework rule, no invariant globalization, warnings not treated as errors. The self test was testing a build the build never does. On a machine where that mattered it produced five failures whose text was a stack trace where a verdict should be, which is the least useful shape a failure can have.

## Running it twice, which is not explained

This one is a defect in the probe rather than in the repository, it is not solved, and it is written down rather than quietly worked around because the next person to see it should not have to find it again.

Run the script twice at the same work directory and the second run fails. Three lessons and a blueprint generator die with the same two errors, an analyzer assembly missing out of the packages folder:

```
error CS0006: Metadata file '.../nuget/microsoft.net.illink.tasks/10.0.0/analyzers/dotnet/cs/ILLink.RoslynAnalyzer.dll' could not be found
```

The packages folder is empty on those runs. Restore fetched three packages on the run that passed and none at all on the run that failed, without saying anything about it.

What is known. It alternates exactly, and it did so for thirteen runs in a row. It is per work directory: four runs at four different paths all passed, then a second run at one of those paths failed. It is not the build servers, because the failing run had none running and disabling them changes nothing. It is not the home directory, the packages folder or the temporary directory, because all three are inside the work directory and get deleted with it. Re-running the same build by hand in the failed directory passes. So there is something outside the work directory that is keyed by its path, and the thing that would tell us what it is has not been found yet.

The script therefore uses a directory nothing has ever used, with the process id in the name, and deletes it when the run passes. That is not a fix, it is the honest version of what cold means, and it makes the probe reliable: a directory the machine has used before is not cold, whatever is going on.

The failure also went from useless to readable along the way. The tool used to report only the error stream of a lesson that failed, and the SDK puts "The build failed. Fix the build errors and run again." there and the compiler's actual errors on the output stream, so the report was a sentence with the reason discarded. It now prints both streams, which is the only reason the two lines above are in this page.

## What the build does about it now

The working directory contract is written down and handed over rather than assumed. Every process the build starts gets `XRAY_HERE`, set to the absolute path of the directory the file being run belongs to. A lesson or a boss fight starts from that.

```csharp
// The build says which directory this lesson is in. It has to, because dotnet run does not
// promise the working directory of a file you point it at.
var here = Environment.GetEnvironmentVariable("XRAY_HERE") ?? ".";

var lines = File.ReadAllLines(Path.Combine(here, "lesson.cs"));
```

Writing it down is not enough on its own, because the next person to write a lesson will write `File.ReadAllLines("lesson.cs")` and it will work on their machine. So the build refuses it. Every lesson file, every generator block and both files of every boss fight are scanned, and a call to `File`, `Directory` or `Path.Combine` whose first argument is a string literal is a build failure with the file and line named.

The gate has two cases in the regeneration self test, one that breaks a boss fight and one that breaks a lesson, so the rule is proved to go red rather than assumed to. That self test now copies both `Directory.Build.props` files along with the lesson, and it has sixteen cases where it had fourteen.

## The results

Every gate passes on all four, from nothing, on the SDK `global.json` names rather than the one a developer machine resolves.

| Platform | Machine | What cold meant | SDK | Gates | Cold start |
|---|---|---|---|---|---|
| linux-x64 | Ubuntu 24.04 server, 8 cpus, shared with other work | `ubuntu:24.04` container, checkout mounted read only | 10.0.100 | all pass | 252 s |
| linux-arm64 | container on a laptop | `ubuntu:24.04` container, checkout mounted read only | 10.0.100 | all pass | 181 s |
| osx-arm64 | laptop, shared with other work | no container, everything inheritable emptied | 10.0.100 | all pass | 141 s |
| win-x64 | Windows 11, bare metal, idle, on a slow link that afternoon | no container, everything inheritable emptied | 10.0.100 | all pass | 394 s |

The gates are the six step build offline, the regeneration self test with its sixteen cases, the prose lint, the numbers gate, the assertion self test and the cache self test. The two Linux rows also had to install `ca-certificates`, `curl` and `libicu74` before .NET would start.

## Where the time goes

| Phase | linux-x64 | linux-arm64 | osx-arm64 | win-x64 |
|---|---|---|---|---|
| the packages Ubuntu does not have | 29 s | 36 s | not needed | not needed |
| a copy of the checkout with none of its build output | 0 s | 0 s | 0 s | 0 s |
| the SDK named by `global.json` | 28 s | 49 s | 41 s | 349 s |
| the six step build, offline | 93 s | 45 s | 48 s | 21 s |
| the regeneration gate against itself | 68 s | 33 s | 37 s | 16 s |
| the rest of the gates | 28 s | 14 s | 12 s | 6 s |
| what the machine turned out to be | 6 s | 4 s | 3 s | 2 s |

One measurement per machine, on four machines that are not comparable with each other. The Linux server was carrying other work and its two build phases show it. The download phases are a function of somebody's network on one afternoon and not of anything in this repository, and the 349 s on Windows is that and nothing else: the same machine did the rest of the run faster than any of the other three.

The right way to read this table is by column and not by row. Within a column the shape is the same everywhere once the download is set aside, and that shape is the point.

Nothing here is a benchmark. The point of the numbers is the shape: the SDK install and the first build of the tool are most of a cold run, and both of them are things a warm machine has already paid for and a reader pays once.

The hosted runners are faster than any of the four above, because they sit next to the thing they are downloading. The first run of the job took 67 s on both Linux platforms, 61 s on macOS and 138 s on Windows, so running every gate a second time from nothing costs about a minute of wall clock on a pull request and none of the critical path, since it runs alongside everything else.

## Running it yourself

On Linux, in a container with nothing in it, with the checkout mounted read only so the build cannot write into it:

```
docker run --rm -v "$PWD:/src:ro" ubuntu:24.04 bash /src/docs/probes/cold-start.sh /src
```

On macOS, where there is no container and the cold part is an empty home directory, an empty NuGet folder, an empty cache and a path with no `dotnet` on it:

```
bash docs/probes/cold-start.sh
```

On Windows, the same, with [cold-start.ps1](cold-start.ps1):

```
pwsh -File docs/probes/cold-start.ps1
```

This is the same script CI runs, on all four platforms, on every pull request. A copy of it kept in a YAML file would drift from this page within a month, so there is one script and both the page and the job point at it.
Loading
Loading