feat(installer): reserve the kubelet's own footprint on every node, measured per platform

[LukasWodka]

Status: unblocked pending REVIEW — not pending merge. The sequencing question is settled (measured, recorded on the internal ticket): a host at the 5 GiB preflight floor is already refused on develop today (2Gi floor + 3136Mi control plane + 140Mi addons > 5120Mi); this PR raises the Linux refusal threshold to ~6.5 GiB and leaves the stock 7.75 GiB macOS VM installing (2Gi envelope instead of 4Gi). A refusal prints the arithmetic and the TRACEBLOC_TRAINING_RESOURCES override, then errors — nothing broken is ever installed. Merging is Lukas's call, as always. The contract v3 PR in client-runtime stays held until the control-plane trim.

Review note: bugbot / review cannot report on this PR (more than 100 check contexts, no pagination — filed internally), so the human reviewer is the only review gate here; "no Bugbot findings" is not a clean bill.

What

On every node the installer has ever provisioned, allocatable == capacity: k3s sets no kube-reserved, no system-reserved and no memory eviction threshold (its stock evictionHard carries only the two disk keys). So "what a pod may have" meant the whole machine, k3s server, containerd and the Docker VM's own daemons included, and every consumer of allocatable inherited that.

Both installer twins now write kubeReserved, systemReserved and evictionHard.memory.available into the kubelet drop-in they already mount (the image-GC file from #912), and the values are measured per platform, never typed:

	where it comes from
kubeReserved.memory	1.25 × the max non-pod working set on the node (node minus its pods system container), idle and under a training-shaped load, from the platform's measurement record
kubeReserved.cpu	1.5 × p95 of the daemons' sustained CPU while a pod runs (the image-unpack burst is reported, not reserved: CPU is compressible)
systemReserved.memory	the envelope contract's vm_reserve — what lives in the VM/host beside the node containers, already measured under the topology work; the record's own outside-the-nodes snapshot is printed as a cross-check
evictionHard.memory.available	5% of the preflight's PF_MIN_MEM_GB floor, the same fraction k3s applies to its disk thresholds. k3s ships no memory threshold at all; the kubelet default is 100Mi

A platform with no record gets no reservation — not one borrowed from another platform — and the install says so. Today that is Windows/WSL2.

The chain, every link a file in this repo

1. scripts/tests/measure-node-reservation.sh brings a cluster up through the installer's own create_cluster(), samples the kubelet's /stats/summary idle and under a training-shaped load (the real 2 GB CPU task image pulled and run), and writes a JSON record.
2. scripts/spec/node-reservation/<os>-<arch>.json — the records (raw samples + p50/p95/max, capacity/allocatable, live /configz, how k3s wires its kubelet).
3. scripts/gen-node-reservation-embed.sh derives the block between the GENERATED markers in both twins; --check runs in make drift (the required Source-of-truth drift job).
4. scripts/tests/kubelet-config-agreement.sh holds the two blocks equal, derives the name set from both files (union), refuses a platform listed as measured with no values behind it, a value for a platform not in the list, a blank/zero value, an eviction threshold looser than the kubelet's 100Mi default, and a writer body that stops emitting a map. kubelet-config-mutations.sh proves each of those (27/27 caught; one case was vacuous before the emission check was scoped to the writer's body, and is recorded as such).
5. e2e-cluster.sh asserts on the live node that capacity − allocatable equals exactly what the drop-in the installer wrote declares, cpu and memory, and that the merged evictionHard still carries k3s's two disk keys. The expected values are parsed from the file, never from the test.
6. installer_parity.json gains a kubelet_reservation section: the shape both writers must show, driven through the real writer in each language (bats + Pester), once as a measured platform and once as an unmeasured probe.

Measured before writing any of it

k3s wiring (k3d v5.9.0 / rancher/k3s:v1.36.3-k3s1): k3s writes its kubelet defaults to kubelet.conf.d/00-k3s-defaults.conf and copies the file named by --kubelet-arg=config= into the same directory as 10-cli-config.conf. The kubelet merges config-dir drop-ins key by key, so the #912 file already is the drop-in this ticket asks for. A probe writing evictionHard: {memory.available: 500Mi, imagefs.available: 7%} read back as {imagefs 7%, memory.available 500Mi, nodefs.available 5%} — k3s's untouched key survived. Mounting straight into kubelet.conf.d/99-tracebloc.conf behaved identically. That corrects a claim in #912's comments (that the file's map replaced k3s's); the wholesale replacement is real but belongs to the --kubelet-arg=eviction-hard= CLI path, which stays refused. Because of the merge, the file writes only memory.available and lets k3s's disk keys through — restating them would pin k3s's numbers in a second place.

macOS / Docker Desktop, arm64 (6 cpu / 11.67 GiB VM, 1 server 0 agents — the installer's own topology here), 300 s idle + 300 s load:

	idle	pull	load (pod running)
node working set − pods	~540 MiB	rising	885 MiB max (page cache from the 5 GB unpack stays active)
node RSS − pods	~423 MiB		~440 MiB
kubelet's own kubelet/runtime rows	117 MiB	122 MiB	153 MiB
non-pod CPU	~50 m	1833 m burst (containerd unpack)	56 m p95

The kubelet's kubelet and runtime rows both map to the /k3s cgroup and report ~1/5 of the real non-pod footprint — the k3s server process sits outside it. That is why the reservation is derived from node-minus-pods and never from the self-report.

Result: kubeReserved cpu=100m memory=1120Mi, systemReserved memory=1024Mi, evictionHard memory.available=256Mi. Live, on a fresh cluster through the real create path: capacity 6 / 12233300Ki → allocatable 5900m / 9775700Ki, exactly the declared 100m / 2457600Ki; /configz shows all three maps and the merged eviction set.

Linux — measured, three records from the new node-reservation-measure job (ubuntu-22.04 amd64, 24.04 amd64, 24.04 arm64; 4 cpu / 16 GiB runners; label-gated on e2e, weekly on the schedule): node − pods working set max 828 / 889 / 812 MiB, sustained daemon CPU p95 62 / 64 / 49 m, unpack burst ~2 cores. The generator takes the max across a platform's records and lands on the same kubeReserved cpu=100m memory=1120Mi as darwin — four machines, two OSes, two architectures within ±40 MiB: the k3s-in-k3d footprint is a property of the node image, not the host OS. The Linux E2E cluster legs now exercise the live capacity − allocatable == drop-in assertion.

Windows/WSL2 is unmeasured and stays unreserved; the PowerShell twin writes the image-GC file it always wrote and warns once at create. No number is extrapolated for it.

Existing edges

The drop-in is a bind-mounted host file. Rewriting it would arm a smaller allocatable on the node's next restart under pods sized against the old one, so the reuse path advises, never rewrites: a mounted file with no kubeReserved: gets the recreate hint, only on a platform that has a record, and silence when the file cannot be read.

Chart

client/Chart.yaml 1.9.108 → 1.9.109 (version and appVersion). No template changes; the bump is the version-bump gate's, and the sibling control-plane-trim PR will need to re-bump after this lands.

Test plan

• make drift — 47/47 green (with PyYAML available; the two new guards are enrolled)
• bats scripts/tests/cluster.bats (kubelet subset 32/32) and installer-parity.bats (7/7); full make bats run before push
• kubelet-config-mutations.sh 27 caught / 0 vacuous
• Live macOS cluster through create_cluster() + the shared e2e_assert_node_reservation: PASS (numbers above)
• shellcheck -S warning -x + bash -n on every touched script; gen-manifest.sh --check, check-facts.sh --check, check-style.sh clean
• Pester: cannot run locally (no pwsh); runs in unit-pester on Linux and Windows

---

Note

High Risk
Changes kubelet reservations and allocatable math that training admission and envelope sizing depend on; mistakes can cause scheduling failures or node OOM, though values are measurement-derived and guarded by drift, parity, and e2e checks.

Overview
Replaces guessed control-plane overhead with measured kubeReserved, systemReserved, and evictionHard.memory.available in the kubelet drop-in both installers already mount, so node allocatable no longer equals full capacity on recorded platforms (darwin, linux).

Measurement → embed chain: JSON records under scripts/spec/node-reservation/ (macOS arm64 plus Linux Ubuntu legs in this PR) feed scripts/gen-node-reservation-embed.sh, which writes the generated block into scripts/lib/cluster.sh and scripts/install-k8s.ps1. make drift runs --check on that generator; gen-installer-parity.sh now emits parity constants for reservation writer shape.

Installer behavior: Bash _write_kubelet_config and PowerShell Write-KubeletConfig add reservation maps only when the platform is listed; unmeasured platforms (e.g. Windows/WSL2) get image-GC only plus a one-time warn. Existing clusters with a mount but no kubeReserved: get recreate hints—no in-place rewrite. Comments are corrected: k3s merges config-dir drop-ins key-by-key, so only memory.available is written and disk eviction keys stay from k3s.

CI: New label-gated node-reservation-measure job runs measure-node-reservation.sh on three Ubuntu runners and uploads artifacts (non-gating evidence). Chart 1.9.109; manifest.sha256 updated for touched bootstrap scripts.

^{Reviewed by Cursor Bugbot for commit 362ac18. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[LukasWodka]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a87c55e. Configure here.}

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[saqlainsyed007]

Approve. Reviewed at 362ac18.

Verified the load-bearing claims against the code rather than the prose:

• Twins are provably equal. The generator emits every value into both cluster.sh (NAME=v) and install-k8s.ps1 ($NAME = v) from one computation, and kubelet-config-agreement.sh §7 derives the name set as the union of both twins (never a held list), comparing each value field-by-field and flagging divergence, absence, non-positive values, a platform-listed-without-values, and values-without-platform. Not a tautology.
• Measured, not typed. systemReserved=1024Mi = the contract's vm_reserve (1073741824 B), evictionHard=256Mi = 5% × PF_MIN_MEM_GB=5 GiB, both derived by the generator; the required Source-of-truth drift job re-derives from the records and would redden on a hand-edit. An unmeasured platform (Windows/WSL2) writes no reservation block and the install says so — confirmed in both writers and the bats coverage.
• Only memory.available under evictionHard, so k3s's imagefs/nodefs disk keys survive the key-by-key drop-in merge — asserted in the bats writer test and re-checked live against /configz by e2e_assert_node_reservation, which parses the expected reservation out of the file the installer wrote and asserts capacity − allocatable equals it exactly (cpu side correctly expects only kubeReserved.cpu). Fails closed on an unreadable file/node.
• Fail-closed on a broken embed in both twins (bash EOF-via-newline + [1-9][0-9]* guard; PowerShell $null → throw), with the zero case refused too.
• No hidden regression on stock nodes: ~2.34 GiB reserved on Linux is below the envelope's old blind 3 GiB memory guess; the 7.75 GiB macOS VM still leaves ~5.4 GiB allocatable, and preflight reads physical capacity so its 5 GiB floor is untouched.
• The formerly-vacuous "writer stops emitting kubeReserved" mutation is now scoped to the writer function body, so the existing-cluster advisory's whole-file grep no longer masks it — verified the extraction boundaries.

CI is green end to end (drift + mutation + agreement + Pester + the 3-platform measurement jobs), both Bugbot threads resolved. Nice work — the derivation chain and the mutation-proofing are exactly right.