Run every gate a second time on a machine that has never seen this

[tamnd]

Closes the last box on M1: cold execution on four platforms, meaning a fresh machine with no caches and no local state.

Why

Every other job in CI starts from a runner image that already has an SDK, a NuGet folder, a home directory the SDK has written to, and a package manager somebody has used. Every machine this book has been written on is the same, only more so. A reader has none of that, and neither does a container, which is what the one click environment is going to be.

The failure this is looking for is not a slow first run. It is a build that passes on every machine it is tried on and is still wrong, because the thing that would have made it fail is sitting on all of those machines. That is not hypothetical. There were three of them.

What it found

A bare ubuntu:24.04 cannot start .NET at all. Not a lesson failing, the runtime aborting before reaching any code in this repository:

Process terminated. Couldn't find a valid ICU package installed on the system.
Please install libicu using your package manager and try again.

.NET links against the system ICU on Linux and the image does not ship one. dotnet-install.sh says in its own help that it does not resolve dependencies, and this is what that sentence means. None of the four hosted runners would ever have told us, because all four have it. The package is named after the ICU release rather than the distribution, so the script looks the name up instead of writing libicu74 down.

global.json names an SDK nothing here had ever run on. It says 10.0.100 with rollForward: latestFeature, so any machine that already has a 10.0.4xx uses that, which is every developer machine and every hosted runner. A reader who installs with --jsonfile global.json gets exactly 10.0.100.

Those two bands disagree about something four files were relying on. dotnet run some/file.cs does not promise the working directory of the file it runs: on 10.0.400 the child inherits the directory the SDK was started in, on 10.0.100 it can get the directory the file itself is in. A boss fight living in boss/ and reading lesson.cs is correct under one and looking one directory too deep under the other, and which one you got was decided by whichever SDK the machine had resolved. Nothing was wrong with the code and nothing was wrong with the test. They had only ever met on one of the two SDKs this repository accepts.

The fix is to stop guessing. Every process the build starts is handed XRAY_HERE, the absolute path of the directory the file belongs to, and lessons start from that:

// The build says which directory this lesson is in. It has to, because dotnet run does not
// promise the working directory of a file you point it at.
var here = Environment.GetEnvironmentVariable("XRAY_HERE") ?? ".";

var lines = File.ReadAllLines(Path.Combine(here, "lesson.cs"));

Writing that down is not enough, because the next person writes File.ReadAllLines("lesson.cs") and it works on their machine. So the build refuses it. Every lesson file, every generator block and both files of every boss fight are scanned, and a call to File, Directory or Path.Combine whose first argument is a string literal fails the build with the file and line named. Two self test cases prove the rule goes red, one breaking a lesson and one breaking a boss fight.

The regeneration self test was building with the wrong settings. It copies a lesson somewhere temporary, breaks the copy on purpose, and requires the build to refuse it. The copy took the lesson directory and nothing above it, and what is above it is two Directory.Build.props files. So the cases were compiled with the SDK's defaults rather than this repository's: different target framework rule, no invariant globalization, warnings not errors. It was testing a build that the build never does. It has sixteen cases now where it had fourteen.

One thing found along the way

A lesson that failed to compile reported only its error stream, and the SDK puts "The build failed. Fix the build errors and run again." there while the compiler's actual diagnostics go to the output stream. So the report was a sentence with the reason discarded a moment earlier by the program printing it. It reports both streams now, and that is the only reason the CS0006 lines in the writeup exist.

What this adds to CI

A cold job with a four way matrix. No setup-dotnet step, on purpose, since that is the one step every other job starts with and the one thing a cold machine does not have.

On Linux it runs in a bare ubuntu:24.04 with the checkout mounted read only, so the container has to install everything down to ca-certificates and cannot quietly write build output back into the checkout. Windows containers are a whole operating system image rather than a directory tree and the macOS runners cannot run containers at all, so on those two the cold part is an empty HOME, DOTNET_ROOT, NuGet folder and artifact cache, and a path with no dotnet on it.

The job runs docs/probes/cold-start.sh and cold-start.ps1, the same scripts the writeup tells a reader to run and the same ones the test fleet ran. A copy of the steps kept in the YAML would drift from the page within a month.

Four new required contexts, taking the branch protection count from 17 to 21.

Measured

Every gate passes on all four from nothing, on the SDK global.json names rather than the one a developer machine resolves. The gates are the six step build offline, the regeneration self test with its sixteen cases, the prose lint, the numbers gate, the assertion self test and the cache self test.

Platform	Where	What cold meant	SDK	Gates	Cold start
linux-x64	server3, 8 cpus, carrying other work	ubuntu:24.04, checkout read only	10.0.100	all pass	252 s
linux-arm64	container on the laptop	ubuntu:24.04, checkout read only	10.0.100	all pass	181 s
osx-arm64	the laptop	no container, everything inheritable emptied	10.0.100	all pass	141 s
win-x64	the gaming pc, via pwsh 7	no container, everything inheritable emptied	10.0.100	all pass	394 s

The 394 s on Windows is a slow download of the SDK that afternoon and nothing else. The same machine did every other phase faster than the other three.

One thing that is not explained, and is written down as such

Run the script twice at the same work directory and the second run fails. Three lessons and a blueprint generator die with an analyzer assembly missing out of the packages folder, and the packages folder is empty on those runs.

It alternates exactly, and did so for thirteen runs in a row. It is keyed to the work directory path: four runs at four different paths all passed, then a second run at one of those paths failed. It is not the build servers, because the failing run had none running and disabling them changes nothing. It is not the home directory, the packages folder or the temporary directory, because all three live inside the work directory and are deleted with it. So there is state outside the work directory keyed by its path, and I did not find it.

The script therefore uses a directory nothing has ever used, with the process id in the name, and deletes it when the run passes. That is not a fix. It is the honest version of what cold means, and it makes the probe reliable, because a directory the machine has used before is not cold whatever is going on. The whole thing is in the writeup rather than quietly worked around, so the next person to see it does not have to find it again.

Checked before pushing

xray check --offline        0 problem(s)
xray check --selftest       0 failure(s), 16 cases
xray lint                   21 file(s), 0 problem(s)
xray numbers lessons        2 lesson(s), 0 problem(s)
xray assert --selftest      0 failure(s)
xray cache --selftest       0 failure(s)
dotnet format --verify-no-changes   clean

All six on all four platforms, cold, plus warm locally.