From 10570818b4de18f4889993d7ead7bd8434a22de3 Mon Sep 17 00:00:00 2001 From: castrojo Date: Sun, 13 Sep 2026 18:44:25 -0400 Subject: [PATCH 1/8] docs(spec): plan migration of the OS payload to a Flatcar base Bluefin Server composes userspace from FSDK 26.08 while taking its kernel and ZFS from Flatcar 4593.2.5. Every installer bring-up defect came from that seam: dracut built the target initrd against a foreign, nested module tree, storage drivers had to be force-loaded by hand, and os-release-flatcar.bst claims ID=flatcar purely so Flatcar sysexts attach. Record the design that collapses the OS payload to a single ABI domain by importing Flatcar's /usr as a pinned binary, keeping BuildStream as the only build system and leaving the installer on FSDK, since systemd-sysinstall is a systemd 261 tool with no Flatcar equivalent. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-09-13-flatcar-base-migration-design.md | 221 ++++++++++++++++++ 1 file changed, 221 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md diff --git a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md new file mode 100644 index 00000000..bb5912fd --- /dev/null +++ b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md @@ -0,0 +1,221 @@ +# Flatcar Base Migration Design + +**Status:** Proposed design + +## Problem + +Bluefin Server currently composes its OS payload from two incompatible +software domains: + +- Userspace from freedesktop-sdk 26.08 (`components/*`): systemd 261, glibc + from the FSDK bootstrap, `uutils-coreutils`, `openssh-systemd`, `podman`, + `xfsprogs`, `gnupg`. +- Kernel and out-of-tree modules from Flatcar stable 4593.2.5 + (`flatcar/flatcar-kernel.bst`, `flatcar/flatcar-zfs.bst`): kernel + `6.12.102-flatcar`, ZFS built against that kernel, both shipped as prebuilt + binaries. + +Every defect class hit during installer bring-up came from the seam between +those two domains, not from either domain individually: + +- The installer builds the *target* OS initrd with FSDK `dracut` against the + Flatcar module tree (`elements/oci/bluefin-server-installer.bst`, step 1b, + `--kmoddir /target-root/usr/lib/modules/${TARGET_KVER}`). Flatcar nests its + modules at `usr/lib/modules/6.12.102-flatcar/6.12.102-flatcar/`, which FSDK + tooling does not expect, and storage drivers silently went missing from the + initrd. +- USB and SCSI storage drivers had to be force-loaded and `udevadm settle` + polled by hand before `systemd-sysinstall` would see the target disk. +- `os-release-flatcar.bst` already lies about identity (`ID=flatcar`, + `CPE_NAME=cpe:/o:flatcar-linux:...`) purely so Flatcar sysexts will attach to + an FSDK userspace. The sysext compatibility check passes on a fiction. + +The repository therefore maintains a permanent ABI straddle to obtain one +thing: a long-term-support kernel with a matching, prebuilt ZFS module. + +## Goals + +- Collapse to a single ABI domain: kernel, userspace, and system extensions + from one upstream with one vermagic, one glibc, one systemd. +- Keep BuildStream as the only build system and the DDI/installer contract + unchanged. +- Keep the installer `systemd-sysinstall`-native and `systemd-repart`-based. +- Delete the hand-written initrd repair path in favor of upstream artifacts. +- Gain an upstream SBOM, package manifest, and signed digests for the OS + payload. +- Make `ID=flatcar` true rather than cosmetic. + +## Non-goals + +- Rebuilding Flatcar packages from source in BuildStream. Flatcar is a + Gentoo/portage cross-build driven by its own SDK; reproducing it under + `bst` is a multi-year effort with no payoff. This design imports Flatcar + binaries the same way `flatcar-kernel.bst` and `flatcar-zfs.bst` already do. +- Adopting Flatcar's update stack (`update_engine`, `locksmithd`) or its + provisioning stack (Ignition, `coreos-cloudinit`). Bluefin Server stays on + `systemd-sysupdate` plus Kured. +- Changing the k0s delivery model. k0s stays an optional sysext. +- arm64 support. Flatcar publishes `arm64-usr`; that is follow-on work. + +## Upstream artifact survey + +Measured against `https://stable.release.flatcar-linux.net/amd64-usr/4593.2.5/` +on 2026-09-13. Every artifact carries `.DIGESTS`, `.DIGESTS.asc`, and `.sig` +companions, so each import can be pinned by sha256 in `bst` and independently +GPG-verified at release time. + +| Artifact | Size | Contents | +|---|---|---| +| `flatcar-container.tar.gz` | 377 MiB | Complete OS tree: `/usr` (19,658 entries), `/boot`, `/oem` | +| `flatcar_production_image_sysext.squashfs` | 418 MiB | The same `/usr`, packaged as a verity-capable sysext squashfs | +| `flatcar_production_image.vmlinuz` | in use today | Kernel `6.12.102-flatcar` | +| `flatcar_production_pxe_image.cpio.gz` | in use today | Upstream-built initrd for that kernel | +| `flatcar-zfs.raw` | 3 MiB | ZFS sysext (in use today) | +| `flatcar-podman.raw` | 33 MiB | Podman sysext | +| `rootfs-included-sysexts/containerd-flatcar.raw` | 24 MiB | containerd sysext | +| `rootfs-included-sysexts/docker-flatcar.raw` | 54 MiB | Docker sysext | +| `flatcar_production_image_packages.txt`, `_contents.txt`, SBOM | text | Package manifest and provenance | +| `version.txt` | text | `FLATCAR_VERSION=4593.2.5`, `FLATCAR_BUILD_ID="2026-08-11-2350"` | + +Verified contents of `flatcar-container.tar.gz`: + +- systemd 257 (`usr/lib/systemd/libsystemd-shared-257.so`). +- glibc 2.41 (`usr/lib64/glibc-2.41/`). +- Everything the current design depends on is present: `systemd-repart`, + `systemd-sysext`, `systemd-confext`, `systemd-sysupdate`, `systemd-creds`, + `bootctl`, `machinectl`, `systemd-nspawn`, `bash`, `sshd`, `crictl`. +- Flatcar's own stack that must be masked or stripped: `update_engine`, + `update_engine_client`, `locksmithd`, `ignition`, `coreos-cloudinit`, + `flatcar-update`, `download_sysext`, `ensure-sysext.service`. +- `systemd-sysinstall` is **absent** — it is an FSDK 261 tool. + +## Design + +### Split the two images by role + +The installer and the installed OS stop sharing a userspace. + +- **Installer** stays FSDK 26.08. It is the only consumer of + `systemd-sysinstall`, which does not exist in systemd 257. Hard rule 3 is + preserved untouched. +- **Installed OS DDI** becomes Flatcar `/usr` plus the Bluefin overlay. + +This split is safe because the installer's contract with the DDI is +byte-level, not content-level: `files/installer/repart.d/20-root.conf` copies +the DDI into the root partition with `CopyBlocks=`. Nothing in the installer +inspects the payload's userspace. Changing what is inside the XFS image does +not change how it is written. + +### New element: `flatcar/flatcar-usr.bst` + +`kind: manual`, mirroring the existing `flatcar-zfs.bst` import pattern: + +- `sources:` one `kind: remote` entry for + `flatcar:stable/%{flatcar-board}/%{flatcar-version}/flatcar-container.tar.gz`, + pinned by `ref:` sha256, using the existing `flatcar:` alias in + `include/aliases.yml`. +- `variables: strip-binaries: ""` (prebuilt binaries; the FSDK stripper must + not touch them). +- `install-commands:` extract `./usr` into `%{install-root}/usr`, then remove + the update and provisioning stack listed above, and flatten Flatcar's nested + module directory to the single-level layout the rest of the tree expects. + +Removals are explicit `rm` lines with a comment naming the replacement, not a +wildcard sweep, so a future Flatcar bump that renames a unit fails loudly. + +### Overlay: what Bluefin keeps + +These elements are OS policy, not upstream software, and carry over unchanged: + +`os-release-flatcar.bst` (now truthful), `os-sysupdate.bst`, +`os-k0s-sysupdate.bst`, `os-sysupdate-keys.bst`, `os-networkd.bst`, +`os-k0s-first-boot.bst`, `os-creds-prov.bst`, `os-kured-hook.bst`, +`os-justfile.bst`, `os-issue.bst`, `os-image-info.bst`, `os-countme.bst`, +`os-sshd-preset.bst`, `os-sshd-config.bst`. + +### Overlay: what Flatcar displaces + +Removed from the OS payload once the Flatcar base lands: + +| Current element | Replacement | +|---|---| +| `freedesktop-sdk.bst:public-stacks/runtime-minimal.bst` | Flatcar `/usr` | +| `freedesktop-sdk.bst:components/systemd.bst` | Flatcar systemd 257 | +| `freedesktop-sdk.bst:components/dbus.bst`, `dbus-broker.bst`, `kmod.bst`, `shadow.bst` | Flatcar `/usr` | +| `freedesktop-sdk.bst:bootstrap/bash.bst` | Flatcar `/usr/bin/bash` | +| `bluefin-server/uutils-coreutils.bst` | Flatcar coreutils | +| `freedesktop-sdk.bst:components/openssh-systemd.bst` | Flatcar `/usr/bin/sshd` | +| `freedesktop-sdk.bst:components/podman.bst` | `flatcar-podman.raw` sysext | +| `freedesktop-sdk.bst:components/xfsprogs.bst`, `gnupg.bst`, `ca-certificates.bst`, `tzdata.bst` | Flatcar `/usr` | +| `bluefin-server/linux-firmware-split.bst` | Flatcar firmware in `/usr/lib/firmware` | + +Moving Podman from the base DDI to a sysext also brings the tree into line +with hard rule 4, which already forbids container runtimes in the base DDI. + +### Initrd: stop generating, start importing + +The FSDK `dracut` invocation for the target OS initrd is deleted. The target +UKI is assembled from `flatcar_production_image.vmlinuz` plus +`flatcar_production_pxe_image.cpio.gz`, both already pinned upstream artifacts +from the same release as the kernel modules. `ukify` still assembles the UKI, +so hard rules 3 and 5 are untouched, but nothing regenerates an initrd against +a foreign module tree. This removes the entire failure mode that required the +manual initrd repacking work. + +### Versioning + +`project.conf` `release-version` is currently derived from and enforced +against the FSDK junction ref by `.github/scripts/check-release-version.py`. +After the split, the FSDK pin describes only the installer, while the OS +payload's real version is `FLATCAR_VERSION`. The release version needs two +axes recorded and enforced separately. This is an invariant change, not a +string edit, and gets its own ticket ahead of any element work. + +## Rejected alternatives + +- **Rebuild Flatcar from source under `bst`.** Rejected: Flatcar is a + portage/SDK cross-build of thousands of ebuilds. The effort is unbounded and + buys nothing that a pinned, digest-verified binary import does not. +- **Status quo hybrid.** Rejected: it permanently straddles two ABI domains + and requires the identity fiction in `os-release-flatcar.bst` to function. +- **Boot Flatcar's `/usr` squashfs with dm-verity, as upstream does.** + Deferred, not rejected. It is the natural follow-on once the base lands and + pairs directly with roadmap items 1-3 (A/B slots, read-only `/usr`, verity). + Taking it in the same change would couple the base swap to a boot-chain + rewrite. + +## Migration phases + +Each phase is independently landable and independently verifiable. + +1. **Decision and invariants.** Amend hard rule 1 to scope FSDK composition to + the installer and permit pinned Flatcar binary imports for the OS payload. + Record this design as an ADR. Split the release-version axes. +2. **Import elements.** `flatcar/flatcar-usr.bst` plus a sysext family for + `podman`, `containerd`, `docker`. Contract tests assert the update and + provisioning stack is absent and the module layout is flat. +3. **Parallel DDI.** A project option (`os-base: fsdk | flatcar`) builds both + payloads so they can be A/B compared on the ghost cluster before anything is + deleted. +4. **Boot proof.** `just show-me-the-future` installs and boots the Flatcar + payload in QEMU; the Lima end-to-end test drives the KubeStellar console + login against it. +5. **Cutover.** `os-stack.bst` switches to the Flatcar base, displaced FSDK + elements and the `dracut` target-initrd path are deleted, and the `os-base` + option is removed. +6. **Follow-on.** dm-verity `/usr` and A/B slots; arm64. + +## Verification + +- `just validate` after every element change. +- `python3 .github/scripts/docs-checks.py`. +- `pytest tests/unit` including new contract tests for the import elements. +- `just cluster-build` on the ghost cluster for heavy builds. +- `just show-me-the-future` QEMU install-and-boot smoke test. +- `just test-e2e-lima` for the console login path. + +## See also + +- [docs/skills/gap-analysis-distros.md](../../skills/gap-analysis-distros.md) - distro comparison that framed this. +- [docs/skills/architecture-roadmap.md](../../skills/architecture-roadmap.md) - A/B slots and verity follow-on. +- [docs/skills/ddi-installer.md](../../skills/ddi-installer.md) - installer and DDI contract. From 578a4a5817308babe8a9d3f44169f283aaf9abd8 Mon Sep 17 00:00:00 2001 From: castrojo Date: Sun, 13 Sep 2026 18:51:59 -0400 Subject: [PATCH 2/8] docs(spec): correct the Flatcar initrd artifact to bootengine.img flatcar_production_pxe_image.cpio.gz is not a driver initrd. Measured: four cpio entries wrapping a 374 MiB usr.squashfs, i.e. the whole OS for a RAM boot. Flatcar's real initramfs is /usr/lib/flatcar/bootengine.img, a 50 MiB squashfs with /init and /etc/initrd-release built by sys-kernel/bootengine 0.0.38-r40, shipped inside /usr and versioned with the release. Record the Ignition constraint that comes with it: bootengine drives ignition-fetch/disks/mount/files and sysroot-boot, which this design rejects, so those units must be masked or the fallback is mkosi-initrd --generic. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-09-13-flatcar-base-migration-design.md | 44 ++++++++++++++----- 1 file changed, 34 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md index bb5912fd..20dece57 100644 --- a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md +++ b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md @@ -69,7 +69,8 @@ GPG-verified at release time. | `flatcar-container.tar.gz` | 377 MiB | Complete OS tree: `/usr` (19,658 entries), `/boot`, `/oem` | | `flatcar_production_image_sysext.squashfs` | 418 MiB | The same `/usr`, packaged as a verity-capable sysext squashfs | | `flatcar_production_image.vmlinuz` | in use today | Kernel `6.12.102-flatcar` | -| `flatcar_production_pxe_image.cpio.gz` | in use today | Upstream-built initrd for that kernel | +| `flatcar_production_pxe_image.cpio.gz` | 374 MiB | Four cpio entries wrapping `usr.squashfs`; a RAM-boot OS payload, **not** a driver initrd | +| `usr/lib/flatcar/bootengine.img` (inside the tarball) | 50 MiB | Flatcar's real initramfs: squashfs, 2,280 entries, `/init` + `/etc/initrd-release` | | `flatcar-zfs.raw` | 3 MiB | ZFS sysext (in use today) | | `flatcar-podman.raw` | 33 MiB | Podman sysext | | `rootfs-included-sysexts/containerd-flatcar.raw` | 24 MiB | containerd sysext | @@ -152,15 +153,38 @@ Removed from the OS payload once the Flatcar base lands: Moving Podman from the base DDI to a sysext also brings the tree into line with hard rule 4, which already forbids container runtimes in the base DDI. -### Initrd: stop generating, start importing - -The FSDK `dracut` invocation for the target OS initrd is deleted. The target -UKI is assembled from `flatcar_production_image.vmlinuz` plus -`flatcar_production_pxe_image.cpio.gz`, both already pinned upstream artifacts -from the same release as the kernel modules. `ukify` still assembles the UKI, -so hard rules 3 and 5 are untouched, but nothing regenerates an initrd against -a foreign module tree. This removes the entire failure mode that required the -manual initrd repacking work. +### Initrd: stop generating, ship it in `/usr` + +The FSDK `dracut` invocation for the target OS initrd is deleted. The +replacement is the initramfs Flatcar already builds for exactly this kernel: +`/usr/lib/flatcar/bootengine.img`, which arrives inside +`flatcar-container.tar.gz` at no extra cost. Measured: a 50 MiB squashfs +(not a cpio), 2,280 entries, carrying `/init`, `/etc/initrd-release`, and +`/etc/cmdline.d/10-default.conf`. It is produced by the `sys-kernel/bootengine` +package (0.0.38-r40 in this release, per `usr/share/SLSA/`), which is Flatcar's +dracut module set, and it is built once per release rather than regenerated +per install. + +`ukify` still assembles the UKI, so hard rules 3 and 5 are untouched, but +nothing regenerates an initrd against a foreign module tree. This removes the +entire failure mode that required the manual initrd repacking work. + +Two constraints apply and are ticket-level work, not hand-waves: + +- `bootengine.img` drives Flatcar's provisioning state machine + (`ignition-fetch.service`, `ignition-disks.service`, `ignition-mount.service`, + `ignition-files.service`, `ignition-diskful.target`, `sysroot-boot.service`). + This design rejects Ignition, so those units must be masked and the root + mount driven from the UKI cmdline `root=PARTUUID=` instead. If masking proves + to fight the image rather than configure it, the fallback is `mkosi-initrd + --generic --kernel-version=`, the systemd project's own generator. +- `flatcar_production_pxe_image.cpio.gz` is **not** a candidate. It is a + four-entry cpio whose payload is `usr.squashfs` at 374 MiB: the whole OS for + a RAM boot, not a driver initrd. + +For reference, `flatcar_production_image.vmlinuz` is a 32 MiB PE bzImage with +an EFI stub (sections `.setup`, `.text`, `.data`) - not a UKI. Flatcar boots it +with `bootengine.img` supplied separately. ### Versioning From 656a6c271c2bf329c96abf2186cb69dbe0b3eb06 Mon Sep 17 00:00:00 2001 From: castrojo Date: Sun, 13 Sep 2026 18:57:09 -0400 Subject: [PATCH 3/8] docs(spec): the Flatcar kernel already carries a two-stage initramfs flatcar_production_image_kernel_config.txt declares CONFIG_INITRAMFS_SOURCE="bootengine.cpio" with XZ compression, so the kernel flatcar-kernel.bst already imports is self-sufficient. Flatcar publishes a manifest per stage: 339 entries for the busybox/veritysetup shim that sets up dm-verity for /usr, 2,280 for the systemd initrd it pivots into, matching bootengine.img exactly. flatcar_production_image.vmlinuz and flatcar_production_pxe.vmlinuz are the same 34,245,760 bytes. The installer has therefore been generating a second initrd to lay over a working one. Deleting the dracut step needs no replacement artifact, only conformance to the cmdline that initramfs expects: root=LABEL=ROOT with usr=PARTLABEL=USR-A. That promotes dm-verity A/B from deferred follow-on to the leading option, since it is the only layout upstream tests. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-09-13-flatcar-base-migration-design.md | 122 ++++++++++++------ 1 file changed, 84 insertions(+), 38 deletions(-) diff --git a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md index 20dece57..0c92d218 100644 --- a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md +++ b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md @@ -68,9 +68,11 @@ GPG-verified at release time. |---|---|---| | `flatcar-container.tar.gz` | 377 MiB | Complete OS tree: `/usr` (19,658 entries), `/boot`, `/oem` | | `flatcar_production_image_sysext.squashfs` | 418 MiB | The same `/usr`, packaged as a verity-capable sysext squashfs | -| `flatcar_production_image.vmlinuz` | in use today | Kernel `6.12.102-flatcar` | +| `flatcar_production_image.vmlinuz` | 32 MiB, in use today | Kernel `6.12.102-flatcar` **with a two-stage initramfs compiled in** (`CONFIG_INITRAMFS_SOURCE="bootengine.cpio"`) | +| `flatcar_production_pxe.vmlinuz` | 32 MiB | Byte-for-byte the same size as the above; same kernel, same embedded initramfs | | `flatcar_production_pxe_image.cpio.gz` | 374 MiB | Four cpio entries wrapping `usr.squashfs`; a RAM-boot OS payload, **not** a driver initrd | -| `usr/lib/flatcar/bootengine.img` (inside the tarball) | 50 MiB | Flatcar's real initramfs: squashfs, 2,280 entries, `/init` + `/etc/initrd-release` | +| `flatcar_production_image_initrd_contents.txt` / `_realinitrd_contents.txt` | text | Per-stage manifests for the embedded initramfs: 339 and 2,280 entries | +| `usr/lib/flatcar/bootengine.img` (inside the tarball) | 50 MiB | Stage 2 of that initramfs, standalone: squashfs, 2,280 entries, `/init` + `/etc/initrd-release` | | `flatcar-zfs.raw` | 3 MiB | ZFS sysext (in use today) | | `flatcar-podman.raw` | 33 MiB | Podman sysext | | `rootfs-included-sysexts/containerd-flatcar.raw` | 24 MiB | containerd sysext | @@ -153,38 +155,79 @@ Removed from the OS payload once the Flatcar base lands: Moving Podman from the base DDI to a sysext also brings the tree into line with hard rule 4, which already forbids container runtimes in the base DDI. -### Initrd: stop generating, ship it in `/usr` - -The FSDK `dracut` invocation for the target OS initrd is deleted. The -replacement is the initramfs Flatcar already builds for exactly this kernel: -`/usr/lib/flatcar/bootengine.img`, which arrives inside -`flatcar-container.tar.gz` at no extra cost. Measured: a 50 MiB squashfs -(not a cpio), 2,280 entries, carrying `/init`, `/etc/initrd-release`, and -`/etc/cmdline.d/10-default.conf`. It is produced by the `sys-kernel/bootengine` -package (0.0.38-r40 in this release, per `usr/share/SLSA/`), which is Flatcar's -dracut module set, and it is built once per release rather than regenerated -per install. - -`ukify` still assembles the UKI, so hard rules 3 and 5 are untouched, but -nothing regenerates an initrd against a foreign module tree. This removes the -entire failure mode that required the manual initrd repacking work. - -Two constraints apply and are ticket-level work, not hand-waves: - -- `bootengine.img` drives Flatcar's provisioning state machine - (`ignition-fetch.service`, `ignition-disks.service`, `ignition-mount.service`, - `ignition-files.service`, `ignition-diskful.target`, `sysroot-boot.service`). - This design rejects Ignition, so those units must be masked and the root - mount driven from the UKI cmdline `root=PARTUUID=` instead. If masking proves - to fight the image rather than configure it, the fallback is `mkosi-initrd - --generic --kernel-version=`, the systemd project's own generator. -- `flatcar_production_pxe_image.cpio.gz` is **not** a candidate. It is a - four-entry cpio whose payload is `usr.squashfs` at 374 MiB: the whole OS for - a RAM boot, not a driver initrd. - -For reference, `flatcar_production_image.vmlinuz` is a 32 MiB PE bzImage with -an EFI stub (sections `.setup`, `.text`, `.data`) - not a UKI. Flatcar boots it -with `bootengine.img` supplied separately. +### Initrd: the kernel already has one + +The FSDK `dracut` invocation for the target OS initrd is deleted, and nothing +replaces it. The kernel this repository already imports ships a complete, +two-stage initramfs compiled in. + +`flatcar_production_image_kernel_config.txt` for this release states it +directly: + +``` +CONFIG_BLK_DEV_INITRD=y +CONFIG_INITRAMFS_SOURCE="bootengine.cpio" +CONFIG_INITRAMFS_COMPRESSION_XZ=y +``` + +Flatcar publishes a manifest for each stage: + +| Stage | Manifest | Entries | Contents | +|---|---|---|---| +| 1 | `flatcar_production_image_initrd_contents.txt` | 339 | `rootfs-0/` shim: a 6,772-byte `/init`, busybox, `kmod`, `dmsetup`, `veritysetup`, and empty `/realinit` + `/sysusr/usr` mount points | +| 2 | `flatcar_production_image_realinitrd_contents.txt` | 2,280 | The systemd initrd - byte-identical entry count to `/usr/lib/flatcar/bootengine.img` | + +Stage 1 is a busybox shim that sets up dm-verity for `/usr` and pivots into +stage 2. Stage 2 is `bootengine.img`, a 50 MiB squashfs built by +`sys-kernel/bootengine` 0.0.38-r40 from Flatcar's dracut module set, carrying +`/init`, `/etc/initrd-release`, and `/etc/cmdline.d/10-default.conf`. It +conforms to the systemd initrd interface exactly as +`docs/INITRD_INTERFACE.md` prescribes. + +`flatcar_production_image.vmlinuz` and `flatcar_production_pxe.vmlinuz` are +both 34,245,760 bytes: one kernel, one embedded initramfs, two names. + +The consequence is that `elements/flatcar/flatcar-kernel.bst` has been +importing a self-sufficient kernel all along, and the installer has been +building a second initrd to lay on top of it. + +`flatcar_production_pxe_image.cpio.gz` is **not** a candidate for anything. It +is a four-entry cpio whose payload is `usr.squashfs` at 374 MiB: the OS for a +RAM boot, not a driver initrd. + +### The boot contract that comes with it + +Taking the built-in initramfs means taking Flatcar's boot contract. Its +cmdline, from the shipped `usr/boot/syslinux/root.A.cfg`, is: + +``` +root=LABEL=ROOT rootflags=subvol=root usr=PARTLABEL=USR-A +``` + +`veritysetup` in stage 1 and `usr=PARTLABEL=USR-A` mean `/usr` is expected as +a dm-verity-protected A/B partition pair. Stage 2 then runs Flatcar's +provisioning state machine: `ignition-fetch.service`, `ignition-disks.service`, +`ignition-mount.service`, `ignition-files.service`, `ignition-kargs.service`, +`ignition-diskful.target`, `sysroot-boot.service`. + +The current DDI has neither a `USR-A` verity pair nor a `ROOT` label, and this +design rejects Ignition. Three ways out, to be settled by the boot proof in +phase 4 rather than asserted here: + +1. **Conform to the layout.** Give the installer's `repart.d` a `USR-A`/`USR-B` + verity pair and a `ROOT`-labelled root, and mask the Ignition units. This is + the largest change, but it delivers roadmap items 1, 2, and 3 - A/B slots, + read-only `/usr`, dm-verity - as a consequence of conforming rather than as + three separate projects, and it is the only option upstream actually tests. +2. **Override the built-in initramfs.** Supply an external initrd, which the + kernel unpacks over the built-in one. This is today's behavior and keeps us + owning a generator forever. +3. **Generate with `mkosi-initrd --generic --kernel-version=`.** The systemd + project's own generator, if the current partition layout must be preserved + and Flatcar's contract cannot be met. + +Option 1 is the recommendation. Options 2 and 3 exist so the boot proof has +somewhere to fall back to. ### Versioning @@ -203,10 +246,13 @@ string edit, and gets its own ticket ahead of any element work. - **Status quo hybrid.** Rejected: it permanently straddles two ABI domains and requires the identity fiction in `os-release-flatcar.bst` to function. - **Boot Flatcar's `/usr` squashfs with dm-verity, as upstream does.** - Deferred, not rejected. It is the natural follow-on once the base lands and - pairs directly with roadmap items 1-3 (A/B slots, read-only `/usr`, verity). - Taking it in the same change would couple the base swap to a boot-chain - rewrite. + Reclassified from "deferred follow-on" to the leading option, because the + kernel's built-in stage-1 initramfs ships `veritysetup` and expects + `usr=PARTLABEL=USR-A`. Conforming to that layout is how the embedded + initramfs boots at all, and it delivers roadmap items 1-3 as a side effect. + The alternative is overriding the built-in initramfs, which is what the + repository does today and what this design exists to stop. Settled by the + phase 4 boot proof. ## Migration phases From bc5df5e981a379fe36a0b99991eb6ef9861aa834 Mon Sep 17 00:00:00 2001 From: castrojo Date: Sun, 13 Sep 2026 19:00:59 -0400 Subject: [PATCH 4/8] docs(spec): prove the embedded initramfs by booting the kernel bare Booting the pinned flatcar_production_image.vmlinuz in QEMU with no -initrd and no disk reaches "Run /init as init process" at 1.05s, then blocks on "Waiting for drive..." while stage 1 hunts for usr=PARTLABEL=USR-A. That demonstrates the initramfs is compiled in and live rather than merely declared, and shows the boot contract asserting itself. The same trace also shows the kernel prepends a built-in CONFIG_CMDLINE of "rootflags=rw mount.usrflags=ro", so a UKI cmdline merges with it rather than replacing it. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-09-13-flatcar-base-migration-design.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md index 0c92d218..f22240b8 100644 --- a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md +++ b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md @@ -195,6 +195,41 @@ building a second initrd to lay on top of it. is a four-entry cpio whose payload is `usr.squashfs` at 374 MiB: the OS for a RAM boot, not a driver initrd. +### Verified by experiment + +The claim above is not inferred from the kernel config alone. Booting the +pinned kernel in QEMU with **no `-initrd` argument and no disk attached**: + +``` +qemu-system-x86_64 -enable-kvm -m 2048 -cpu host -nographic -no-reboot \ + -kernel \ + -append "console=ttyS0,115200n8 root=LABEL=ROOT usr=PARTLABEL=USR-A" +``` + +produced: + +``` +[ 0.000000] Linux version 6.12.102-flatcar (build@pony-truck.infra.kinvolk.io) ... +[ 0.025767] Kernel command line: rootflags=rw mount.usrflags=ro console=ttyS0,115200n8 root=LABEL=ROOT usr=PARTLABEL=USR-A +[ 1.053101] Run /init as init process +[ 1.171015] SCSI subsystem initialized +... +Waiting for drive... +Still waiting for drive... +``` + +Three facts fall out of those six lines: + +- `Run /init as init process` with no initrd supplied proves the initramfs is + compiled in and live, not merely declared in the config. +- Stage 1 loaded storage modules and then blocked on `Waiting for drive...`, + which is the busybox shim hunting for `usr=PARTLABEL=USR-A`. With no disk + attached it waits forever. That message is the boot contract asserting + itself. +- `rootflags=rw mount.usrflags=ro` appears **before** the appended arguments: + the kernel carries a built-in `CONFIG_CMDLINE` that any UKI cmdline is + merged with, not a replacement for. + ### The boot contract that comes with it Taking the built-in initramfs means taking Flatcar's boot contract. Its From dca48509da67c666925055dc8b33121c589db876 Mon Sep 17 00:00:00 2001 From: castrojo Date: Sun, 13 Sep 2026 19:17:44 -0400 Subject: [PATCH 5/8] docs(spec): adopt Flatcar's boot contract and partition layout Decision: follow Flatcar's design. The OS payload becomes a /usr image on a verity-protected A/B pair and / becomes writable state. Overriding the built-in initramfs, by dracut or mkosi-initrd, is rejected. Extracted stage 1's /init from the kernel's embedded initramfs and read the contract instead of guessing it. veritysetup runs with a hardcoded --hash-offset=1065345024, so verity is one partition with the hash tree appended, not systemd-repart's two-partition model; the image carries its own hash and repart keeps using CopyBlocks. Verity is skipped when verity.usrhash is empty, so layout and verity split into separate tickets. mount -t passes usrfstype through, so /usr stays XFS. Stage 1 loop-mounts /usr/lib/flatcar/bootengine.img from the mounted /usr, so stage 2 ships with the payload and must survive the import strip list. Records Flatcar's GPT read from flatcar_production_image.bin and maps it onto repart.d: EFI-SYSTEM, USR-A, USR-B, ROOT, with BIOS-BOOT, OEM, and OEM-CONFIG dropped as BIOS-only or Ignition-only. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-09-13-flatcar-base-migration-design.md | 193 ++++++++++++------ 1 file changed, 135 insertions(+), 58 deletions(-) diff --git a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md index f22240b8..6dab3103 100644 --- a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md +++ b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md @@ -177,23 +177,27 @@ Flatcar publishes a manifest for each stage: | 1 | `flatcar_production_image_initrd_contents.txt` | 339 | `rootfs-0/` shim: a 6,772-byte `/init`, busybox, `kmod`, `dmsetup`, `veritysetup`, and empty `/realinit` + `/sysusr/usr` mount points | | 2 | `flatcar_production_image_realinitrd_contents.txt` | 2,280 | The systemd initrd - byte-identical entry count to `/usr/lib/flatcar/bootengine.img` | -Stage 1 is a busybox shim that sets up dm-verity for `/usr` and pivots into -stage 2. Stage 2 is `bootengine.img`, a 50 MiB squashfs built by -`sys-kernel/bootengine` 0.0.38-r40 from Flatcar's dracut module set, carrying -`/init`, `/etc/initrd-release`, and `/etc/cmdline.d/10-default.conf`. It -conforms to the systemd initrd interface exactly as +Stage 1 is a busybox shim. It is the only stage compiled into the kernel; the +`realinit` entry in its cpio is an empty directory. Stage 2 is +`bootengine.img`, a 50 MiB squashfs built by `sys-kernel/bootengine` 0.0.38-r40 +from Flatcar's dracut module set, carrying `/init`, `/etc/initrd-release`, and +`/etc/cmdline.d/10-default.conf`. Stage 1 loop-mounts it **from the `/usr` it +just mounted**, so stage 2 versions with the OS payload automatically rather +than with the kernel. It conforms to the systemd initrd interface exactly as `docs/INITRD_INTERFACE.md` prescribes. `flatcar_production_image.vmlinuz` and `flatcar_production_pxe.vmlinuz` are -both 34,245,760 bytes: one kernel, one embedded initramfs, two names. +both 34,245,760 bytes: one kernel, one embedded stage 1, two names. The consequence is that `elements/flatcar/flatcar-kernel.bst` has been importing a self-sufficient kernel all along, and the installer has been building a second initrd to lay on top of it. -`flatcar_production_pxe_image.cpio.gz` is **not** a candidate for anything. It -is a four-entry cpio whose payload is `usr.squashfs` at 374 MiB: the OS for a -RAM boot, not a driver initrd. +`flatcar_production_pxe_image.cpio.gz` is not a driver initrd. It is a +four-entry cpio whose payload is `usr.squashfs` at 374 MiB, and stage 1 has an +explicit branch for it: when no `/usr` partition is found and `/usr.squashfs` +exists in the initramfs, that file becomes `/usr` with `usrfstype=squashfs`. +It is the PXE path, not the disk path. ### Verified by experiment @@ -230,39 +234,106 @@ Three facts fall out of those six lines: the kernel carries a built-in `CONFIG_CMDLINE` that any UKI cmdline is merged with, not a replacement for. -### The boot contract that comes with it +### The boot contract: decided, conform to it + +**Decision: follow Flatcar's design.** The OS payload becomes a `/usr` image on +a verity-protected A/B partition pair, and `/` becomes writable state. The +alternatives - overriding the built-in initramfs, or generating one with +`mkosi-initrd` - are dropped. + +The contract is not guesswork. Stage 1's `/init` was extracted from the kernel +and read directly; it is 6,772 bytes of shell and the relevant logic is exact: + +```sh +verityusr=$(cmdline_arg verity.usr) +usrhash=$(cmdline_arg verity.usrhash) +verityusr=$(find_drive "${verityusr}") +if echo "${verityusr}" | grep -q "^/" && [ "${usrhash}" != "" ]; then + veritysetup --panic-on-corruption --hash-offset=1065345024 open "${verityusr}" usr "${verityusr}" "${usrhash}" + status=$(dmsetup status usr | cut -d " " -f 4) + [ "${status}" = V ] || { echo "Verity setup failed" >&2; false; } +fi +usr=$(cmdline_arg mount.usr $(cmdline_arg usr)) +usrfstype=$(cmdline_arg mount.usrfstype $(cmdline_arg usrfstype auto)) +usrflags=$(cmdline_arg mount.usrflags $(cmdline_arg usrflags ro)) +mount -t "${usrfstype}" -o "${usrflags}" "${usr}" /sysusr/usr +losetup -r "${LOOP}" /sysusr/usr/lib/flatcar/bootengine.img +mount -t squashfs "${LOOP}" /underlay +mount -t overlay -o rw,lowerdir=/underlay,upperdir=/work/realinit,workdir=/work/work overlay /realinit +mount -o move /sysusr/usr /realinit/sysusr/usr +exec switch_root /realinit /init +``` -Taking the built-in initramfs means taking Flatcar's boot contract. Its -cmdline, from the shipped `usr/boot/syslinux/root.A.cfg`, is: +Seven consequences, each load-bearing: + +1. **Verity is one partition, not two.** `--hash-offset=1065345024` is + hardcoded, with the comment "Hardcoded expected value from the image GPT + layout". The filesystem occupies the first 1,065,345,024 bytes and the + verity hash tree follows it in the same partition. This is incompatible with + `systemd-repart`'s `Verity=data` / `Verity=hash` two-partition model, so the + image is built with the hash appended by the DDI element and `repart` simply + `CopyBlocks=` the result - the existing contract, unchanged. +2. **Our `/usr` must fit in 1,065,345,024 bytes.** Flatcar's own uses 454 MiB of + it. This is a hard build-time budget, and the DDI element must fail loudly + when exceeded rather than silently corrupt the hash offset. +3. **Verity is optional.** The block is guarded by + `[ "${usrhash}" != "" ]`. Landing the partition layout without verity is a + valid intermediate state, so layout and verity split cleanly into two + tickets. +4. **The root hash must reach the kernel cmdline** as `verity.usrhash=`. + Flatcar publishes theirs per release in + `flatcar_production_image_verity.txt` + (`20b08968dc4527a622b7f9f0ba9b6e1a16377500f1f9f93231712ccb45150570`). Ours is + an output of our own DDI build, baked into the UKI cmdline by `ukify`. That + binds each UKI to exactly one `/usr` image, which is precisely the property + A/B updates need. +5. **The `/usr` filesystem stays XFS.** `mount -t "${usrfstype}"` passes the + type through, so `mount.usrfstype=xfs` works. Only `auto` and `btrfs` get + Flatcar's `rescue=nologreplay` special-casing. +6. **`/usr/lib/flatcar/bootengine.img` must survive the import.** Stage 1 + loop-mounts it from the mounted `/usr`; without it the boot stops between + stages. It is an explicit keep in the `flatcar-usr.bst` strip list, not an + incidental leftover. +7. **Ignition units must be masked.** Stage 2 runs `ignition-fetch.service`, + `ignition-disks.service`, `ignition-mount.service`, `ignition-files.service`, + `ignition-kargs.service`, `ignition-diskful.target`, and + `sysroot-boot.service`. This design rejects Ignition. + +### Target partition layout + +Flatcar's own GPT, read from `flatcar_production_image.bin`: + +| # | PARTLABEL | MiB | Type GUID | +|---|---|---|---| +| 1 | `EFI-SYSTEM` | 1024 | `c12a7328-f81f-11d2-ba4b-00a0c93ec93b` | +| 2 | `BIOS-BOOT` | 2 | `21686148-6449-6e6f-744e-656564454649` | +| 3 | `USR-A` | 2048 | `5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6` | +| 4 | `USR-B` | 2048 | `5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6` | +| 6 | `OEM` | 1024 | `0fc63daf-8483-4772-8e79-3d69d8477de4` | +| 7 | `OEM-CONFIG` | 64 | `c95dc21a-df0e-4340-8d7b-26cbfa9a03e0` | +| 9 | `ROOT` | 1784 | `3884dd41-8582-4404-b9a8-e9b84f2df50e` | -``` -root=LABEL=ROOT rootflags=subvol=root usr=PARTLABEL=USR-A -``` +Partitions 5 and 8 are absent; the numbering is ChromeOS heritage. + +What Bluefin Server adopts, as `repart.d` drop-ins replacing the current +`10-esp.conf` / `20-root-a.conf` / `30-var.conf`: + +| PARTLABEL | Type | Source | Notes | +|---|---|---|---| +| `EFI-SYSTEM` | ESP, vfat | `bootctl install` + UKI | Carries the UKI whose cmdline pins `verity.usrhash=` | +| `USR-A` | `5dfbf5f4-…` | `CopyBlocks=` the `/usr` DDI | Read-only, verity, 2048 MiB | +| `USR-B` | `5dfbf5f4-…` | empty | The A/B slot `50-root.transfer` already names but the installer never provisioned | +| `ROOT` | `3884dd41-…` | `Format=`, `GrowFileSystem=yes` | Writable state | + +`BIOS-BOOT` is dropped: this is a UEFI-only image, per hard rule 5. +`OEM` and `OEM-CONFIG` are dropped: they exist for Flatcar's cloud-provider +hooks and Ignition, both of which this design rejects. -`veritysetup` in stage 1 and `usr=PARTLABEL=USR-A` mean `/usr` is expected as -a dm-verity-protected A/B partition pair. Stage 2 then runs Flatcar's -provisioning state machine: `ignition-fetch.service`, `ignition-disks.service`, -`ignition-mount.service`, `ignition-files.service`, `ignition-kargs.service`, -`ignition-diskful.target`, `sysroot-boot.service`. - -The current DDI has neither a `USR-A` verity pair nor a `ROOT` label, and this -design rejects Ignition. Three ways out, to be settled by the boot proof in -phase 4 rather than asserted here: - -1. **Conform to the layout.** Give the installer's `repart.d` a `USR-A`/`USR-B` - verity pair and a `ROOT`-labelled root, and mask the Ignition units. This is - the largest change, but it delivers roadmap items 1, 2, and 3 - A/B slots, - read-only `/usr`, dm-verity - as a consequence of conforming rather than as - three separate projects, and it is the only option upstream actually tests. -2. **Override the built-in initramfs.** Supply an external initrd, which the - kernel unpacks over the built-in one. This is today's behavior and keeps us - owning a generator forever. -3. **Generate with `mkosi-initrd --generic --kernel-version=`.** The systemd - project's own generator, if the current partition layout must be preserved - and Flatcar's contract cannot be met. - -Option 1 is the recommendation. Options 2 and 3 exist so the boot proof has -somewhere to fall back to. +This closes the known gap recorded as an `xfail` in +`tests/unit/test_repart_layout.py`, where `50-root.transfer` names both slots +but the installer provisions only one. Roadmap items 1, 2, and 3 - A/B slots, +read-only `/usr`, dm-verity - arrive as a consequence of conforming rather than +as three separate projects. ### Versioning @@ -280,14 +351,15 @@ string edit, and gets its own ticket ahead of any element work. buys nothing that a pinned, digest-verified binary import does not. - **Status quo hybrid.** Rejected: it permanently straddles two ABI domains and requires the identity fiction in `os-release-flatcar.bst` to function. -- **Boot Flatcar's `/usr` squashfs with dm-verity, as upstream does.** - Reclassified from "deferred follow-on" to the leading option, because the - kernel's built-in stage-1 initramfs ships `veritysetup` and expects - `usr=PARTLABEL=USR-A`. Conforming to that layout is how the embedded - initramfs boots at all, and it delivers roadmap items 1-3 as a side effect. - The alternative is overriding the built-in initramfs, which is what the - repository does today and what this design exists to stop. Settled by the - phase 4 boot proof. +- **Boot Flatcar's `/usr` with dm-verity, as upstream does.** **Adopted.** Not + a follow-on and not optional: stage 1 hardcodes + `--hash-offset=1065345024` and expects `verity.usr=` / `verity.usrhash=`, so + this is simply how the embedded initramfs boots. See "The boot contract: + decided, conform to it". +- **Override the built-in initramfs with a generated one**, via dracut or + `mkosi-initrd --generic`. Rejected: it is what the repository does today and + what this design exists to stop. Keeping it means owning an initrd generator + and a foreign module tree forever. ## Migration phases @@ -298,17 +370,22 @@ Each phase is independently landable and independently verifiable. Record this design as an ADR. Split the release-version axes. 2. **Import elements.** `flatcar/flatcar-usr.bst` plus a sysext family for `podman`, `containerd`, `docker`. Contract tests assert the update and - provisioning stack is absent and the module layout is flat. -3. **Parallel DDI.** A project option (`os-base: fsdk | flatcar`) builds both - payloads so they can be A/B compared on the ghost cluster before anything is - deleted. -4. **Boot proof.** `just show-me-the-future` installs and boots the Flatcar - payload in QEMU; the Lima end-to-end test drives the KubeStellar console - login against it. -5. **Cutover.** `os-stack.bst` switches to the Flatcar base, displaced FSDK - elements and the `dracut` target-initrd path are deleted, and the `os-base` - option is removed. -6. **Follow-on.** dm-verity `/usr` and A/B slots; arm64. + provisioning stack is absent, the module layout is flat, and + `/usr/lib/flatcar/bootengine.img` is preserved. +3. **Partition layout.** Replace `repart.d/10-esp.conf`, `20-root-a.conf`, and + `30-var.conf` with Flatcar's `EFI-SYSTEM` / `USR-A` / `USR-B` / `ROOT` + layout using upstream type GUIDs. Verity stays off at this stage, which + stage 1 explicitly permits, so the layout can be proven on its own. +4. **`/usr` DDI.** The OS payload becomes a `/usr` image sized to the + 1,065,345,024-byte budget with the verity hash tree appended, and its root + hash is baked into the UKI cmdline as `verity.usrhash=`. +5. **Boot proof.** `just show-me-the-future` installs and boots on the kernel's + built-in initramfs with Ignition masked; the Lima end-to-end test drives the + KubeStellar console login against it. +6. **Cutover.** `os-stack.bst` switches to the Flatcar base; the displaced FSDK + elements and the `dracut` target-initrd path are deleted. +7. **Follow-on.** Wire `50-root.transfer` to the real `USR-A`/`USR-B` slots and + retire the `xfail` in `tests/unit/test_repart_layout.py`; arm64. ## Verification From b152024902b00123bef9959b30d14923d294f788 Mon Sep 17 00:00:00 2001 From: castrojo Date: Sun, 13 Sep 2026 19:27:11 -0400 Subject: [PATCH 6/8] docs(spec): boot the adopted design end to end and weigh the parity plan Built a conforming disk from upstream artifacts alone, unprivileged: GPT with Flatcar's type GUIDs, /usr as a 356 MiB squashfs in USR-A, ext4 OEM and ROOT via mke2fs -d. The pinned kernel booted it with no external initrd through stage 1, bootengine, switch_root, and on to multi-user.target and a login prompt, with SSH host keys and DHCP up. Two corrections from the trace. OEM is required, not droppable: stage 2 waits on dev-disk-by-label-OEM.device, matched by filesystem label rather than partition label, and drops to an emergency shell after 90s without it. Ignition needs no masking: ignition-setup-pre completes, delete-config skips on its trigger condition, and ignition-subsequent.target reports "Not Ignition" boot complete. Also records the competing version-parity plan. Adopts its pin-discipline, systemd-creds provisioning, and reboot coordination. Rejects rebuilding Flatcar versions from source, since version parity is not ABI parity under a different toolchain; rejects three tarball-built kernels, which would each lose CONFIG_INITRAMFS_SOURCE and re-create the initrd problem; rejects dropping k0s, which the KubeStellar path depends on; and supersedes its root-a/root-b model with the measured USR-A/USR-B pair. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-09-13-flatcar-base-migration-design.md | 101 ++++++++++++++++-- 1 file changed, 95 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md index 6dab3103..bbf1a58a 100644 --- a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md +++ b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md @@ -294,10 +294,12 @@ Seven consequences, each load-bearing: loop-mounts it from the mounted `/usr`; without it the boot stops between stages. It is an explicit keep in the `flatcar-usr.bst` strip list, not an incidental leftover. -7. **Ignition units must be masked.** Stage 2 runs `ignition-fetch.service`, - `ignition-disks.service`, `ignition-mount.service`, `ignition-files.service`, - `ignition-kargs.service`, `ignition-diskful.target`, and - `sysroot-boot.service`. This design rejects Ignition. +7. **Ignition needs no masking.** Measured, not assumed: on a disk with no + Ignition config, stage 2 runs `ignition-setup-pre.service` to completion, + skips `ignition-delete-config.service` ("no trigger condition checks were + met"), and reaches `ignition-subsequent.target - Subsequent (Not Ignition) + boot complete`. The state machine degrades to a no-op on its own. Masking + is available if a future unit misbehaves, but it is not a prerequisite. ### Target partition layout @@ -323,11 +325,15 @@ What Bluefin Server adopts, as `repart.d` drop-ins replacing the current | `EFI-SYSTEM` | ESP, vfat | `bootctl install` + UKI | Carries the UKI whose cmdline pins `verity.usrhash=` | | `USR-A` | `5dfbf5f4-…` | `CopyBlocks=` the `/usr` DDI | Read-only, verity, 2048 MiB | | `USR-B` | `5dfbf5f4-…` | empty | The A/B slot `50-root.transfer` already names but the installer never provisioned | +| `OEM` | `0fc63daf-…` | `Format=ext4`, `Label=OEM` | Required: stage 2 waits on `dev-disk-by-label-OEM.device` | | `ROOT` | `3884dd41-…` | `Format=`, `GrowFileSystem=yes` | Writable state | `BIOS-BOOT` is dropped: this is a UEFI-only image, per hard rule 5. -`OEM` and `OEM-CONFIG` are dropped: they exist for Flatcar's cloud-provider -hooks and Ignition, both of which this design rejects. +`OEM` is **required**, not optional. Stage 2 declares a dependency on +`dev-disk-by-label-OEM.device`; without it the boot waits 90 seconds and drops +to an emergency shell. Note the match is on **filesystem label** `OEM`, not +partition label, so the partition must be formatted with `mke2fs -L OEM` or +equivalent. `OEM-CONFIG` is dropped: nothing in the boot path references it. This closes the known gap recorded as an `xfail` in `tests/unit/test_repart_layout.py`, where `50-root.transfer` names both slots @@ -335,6 +341,45 @@ but the installer provisions only one. Roadmap items 1, 2, and 3 - A/B slots, read-only `/usr`, dm-verity - arrive as a consequence of conforming rather than as three separate projects. +### Boot proof + +The adopted design was booted end to end before any Bluefin code was written, +using only upstream artifacts and unprivileged tooling (`mksquashfs`, +`mke2fs -d`, `sfdisk`; no root, no loop mounts). + +A 10 GiB GPT was built with Flatcar's type GUIDs: `EFI-SYSTEM`, `USR-A`, +`USR-B`, `OEM`, `ROOT`. Flatcar's `/usr` tree was packed with `mksquashfs` +(356 MiB, against the 1,065,345,024-byte budget) and written into `USR-A`; +`OEM` and `ROOT` were `mke2fs -d` ext4 images. The pinned kernel was booted +with no external initrd: + +``` +-append "console=ttyS0,115200n8 mount.usr=PARTLABEL=USR-A \ + mount.usrfstype=squashfs mount.usrflags=ro root=PARTLABEL=ROOT rootfstype=ext4" +``` + +The full chain ran: + +``` +[ 1.122448] Run /init as init process +Mounting /usr from /dev/vda2 +[ 1.776610] systemd[1]: Successfully made /usr/ read-only. +[ 1.788585] systemd[1]: systemd 257.9 running in system mode +[ 1.794580] systemd[1]: Running in initrd. +[ 4.133550] systemd[1]: Switching root. +Welcome to Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo)! +[ OK ] Reached target multi-user.target - Multi-User System. +localhost login: +``` + +SSH host keys were generated and DHCP brought `ens3` up on `10.0.2.15`. Every +claim in this section - `PARTLABEL` resolution, read-only `/usr`, +`bootengine.img` loop-mount, `switch_root`, Ignition degrading to a no-op - is +from that trace rather than from reading upstream code. + +Two corrections came out of it, both folded in above: `OEM` is required, and +Ignition needs no masking. + ### Versioning `project.conf` `release-version` is currently derived from and enforced @@ -344,6 +389,50 @@ payload's real version is `FLATCAR_VERSION`. The release version needs two axes recorded and enforced separately. This is an invariant change, not a string edit, and gets its own ticket ahead of any element work. +## Second opinion: the version-parity plan + +A competing plan proposes **version parity**: read the component versions +Flatcar ships and rebuild those same versions from source inside BuildStream, +with three kernels (Flatcar LTS, Fedora CoreOS, Ubuntu), a Flatcar-versioned +`k8s` sysext replacing k0s, and A/B slots as `root-a`/`root-b`. + +Adopted from it: + +- **Single source of truth for pins.** Extend `include/flatcar.yml` to carry + every pinned upstream version, with a single-consumer rule so no element or + script hardcodes one. Good discipline, orthogonal to the base swap. +- **Provisioning parity via `systemd-creds`**, covering SSH keys, networkd + configuration, `systemd-firstboot`, and TPM2 sealing. This design keeps + `systemd-sysinstall` + `systemd-creds` and rejects Ignition, which the other + plan agrees with; its provisioning workstream is real work this spec did not + cover. +- **Reboot coordination for non-Kubernetes hosts**, matching roadmap item 6. + +Rejected, with reasons: + +- **"Rebuild Flatcar's versions from source in BuildStream."** Version parity + is not ABI parity. Flatcar's kernel banner reads + `x86_64-cros-linux-gnu-gcc (Gentoo Hardened 14.3.1_p20250801 p4)`; rebuilding + "the same version" under the FSDK toolchain produces different binaries with + different behavior, which is precisely the seam this migration exists to + close. It also means maintaining forks of Flatcar's forks. +- **Three kernels built from upstream tarballs.** The premise that Flatcar has + "Ubuntu-based builds" is not true of any published release. More decisively, + the boot proof above depends on `CONFIG_INITRAMFS_SOURCE="bootengine.cpio"`: + a kernel built from a plain upstream tarball has no embedded initramfs, so + each additional kernel re-creates the initrd problem this design removes. +- **Deleting the k0s sysext for a Flatcar `k8s` sysext.** That plan's own risk + table concedes the Flatcar sysext is "binaries-only, not a full control + plane" and that dropping k0s "removes the single-node k8s story". The + KubeStellar console path depends on it. +- **A/B as `root-a`/`root-b`.** Superseded by measurement: Flatcar's A/B pair + is `USR-A`/`USR-B` with type GUID `5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6`, + and `/` is writable state. Mirroring the whole-rootfs DDI into a second slot + does not satisfy the initramfs contract. +- **"Enforce read-only `/usr` via fstab or cmdline `ro`."** Already automatic: + `mount.usrflags=ro` produced `Successfully made /usr/ read-only` in the + trace. No additional mechanism needed. + ## Rejected alternatives - **Rebuild Flatcar from source under `bst`.** Rejected: Flatcar is a From 2ad4ac27f7a445ef8d05762e117e661374e10ae9 Mon Sep 17 00:00:00 2001 From: castrojo Date: Sun, 13 Sep 2026 19:45:22 -0400 Subject: [PATCH 7/8] docs(spec): reframe as version parity on the FSDK base The objective is version parity, not ABI parity, and the question the project answers is what Flatcar Container Linux looks like built with BuildStream. For each component Flatcar ships, build that same upstream version from source. Binaries will differ because the toolchain differs - Flatcar's kernel banner reads Gentoo Hardened gcc 14.3.1 - and that is accepted. Success is that the image boots and behaves and that every divergence is recorded, not that bytes match. The build base stays freedesktop-sdk 26.08 and hard rule 1 is unchanged. Flatcar's binaries are imported as a reference to boot against and measure, not as the shipped payload, so the method is import first then substitute, with the boot proof as the control. Release version becomes 26.08.XX.$FLATCARVERSION, one string with both halves enforced: the FSDK point release against the junction ref, the Flatcar release against the pin in include/flatcar.yml. This supersedes the earlier two-field proposal. Adds a version audit phase producing a parity matrix of component, Flatcar version, FSDK version, gap, which orders all downstream work. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-09-13-flatcar-base-migration-design.md | 261 ++++++++++++------ 1 file changed, 179 insertions(+), 82 deletions(-) diff --git a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md index bbf1a58a..3a52b1a8 100644 --- a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md +++ b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md @@ -1,7 +1,41 @@ -# Flatcar Base Migration Design +# Flatcar Version Parity Design **Status:** Proposed design +## Purpose + +This project is an experiment with a concrete question: **what would Flatcar +Container Linux look like if it were built with BuildStream?** + +Flatcar is a Gentoo/portage cross-build driven by its own SDK. Bluefin Server +builds with `bst`. The question is not academic - it decides whether an +image-based server OS of Flatcar's shape can be produced by a declarative, +cache-backed build graph instead of a distribution toolchain, and what is lost +or gained on the way. + +The target is **version parity, not ABI parity.** + +- **Version parity** means: for each component Flatcar ships, Bluefin Server + builds the same upstream version from source in BuildStream. systemd 257 + means systemd 257. Kernel `6.12.102` means kernel `6.12.102`. +- **ABI parity is explicitly not the goal.** Flatcar's binaries come from + `x86_64-cros-linux-gnu-gcc (Gentoo Hardened 14.3.1_p20250801 p4)` with their + patch set; ours will not. The binaries will differ, and that is expected and + accepted. Where a difference in behavior falls out of a difference in + toolchain, that difference is a finding worth recording, not a defect to + paper over. + +The distinction matters because it sets what "done" means. Done is not "our +bytes match theirs". Done is "we build the versions Flatcar builds, the result +boots and behaves, and we can say precisely where a BuildStream-built Flatcar +diverges from the real one." + +**The build base stays freedesktop-sdk 26.08.** Hard rule 1 is unchanged: +compose from FSDK `components/*`. Version parity is pursued *within* the FSDK +graph, by building the versions Flatcar ships rather than by importing +Flatcar's tree wholesale. Where FSDK's pinned version of a component differs +from Flatcar's, closing that difference is the parity work. + ## Problem Bluefin Server currently composes its OS payload from two incompatible @@ -35,28 +69,54 @@ thing: a long-term-support kernel with a matching, prebuilt ZFS module. ## Goals -- Collapse to a single ABI domain: kernel, userspace, and system extensions - from one upstream with one vermagic, one glibc, one systemd. +- Achieve version parity with Flatcar 4593.2.5: build each component Flatcar + ships at the version Flatcar ships it, from source, in BuildStream. - Keep BuildStream as the only build system and the DDI/installer contract unchanged. - Keep the installer `systemd-sysinstall`-native and `systemd-repart`-based. -- Delete the hand-written initrd repair path in favor of upstream artifacts. -- Gain an upstream SBOM, package manifest, and signed digests for the OS - payload. +- Conform to Flatcar's boot contract and partition layout, so a + BuildStream-built payload is substitutable for the real one. +- Delete the hand-written initrd repair path. +- Record every divergence between the BuildStream build and upstream, with + evidence, as the primary output of the experiment. - Make `ID=flatcar` true rather than cosmetic. ## Non-goals -- Rebuilding Flatcar packages from source in BuildStream. Flatcar is a - Gentoo/portage cross-build driven by its own SDK; reproducing it under - `bst` is a multi-year effort with no payoff. This design imports Flatcar - binaries the same way `flatcar-kernel.bst` and `flatcar-zfs.bst` already do. +- **ABI or byte-level parity with Flatcar's binaries.** Different toolchain, + different binaries. See "Purpose". +- Reproducing Flatcar's build system. `coreos-assembler`, the Flatcar SDK, and + portage stay out of the tree; `bst` is the point of the exercise. +- Matching Flatcar's full package set. Parity is scoped to the components + Flatcar publishes versions for. - Adopting Flatcar's update stack (`update_engine`, `locksmithd`) or its provisioning stack (Ignition, `coreos-cloudinit`). Bluefin Server stays on - `systemd-sysupdate` plus Kured. -- Changing the k0s delivery model. k0s stays an optional sysext. + `systemd-sysupdate` plus Kured with `systemd-creds`. +- Changing the k0s delivery model. k0s stays an optional sysext; the + KubeStellar console path depends on it. - arm64 support. Flatcar publishes `arm64-usr`; that is follow-on work. +## Method: import first, then substitute + +Version parity needs an oracle. Building a component and asking "is this +right?" is unanswerable without something known-good to compare against. + +So the work runs in two movements: + +1. **Import.** Bring Flatcar's own binaries in as pinned, digest-verified + artifacts and boot them on Bluefin Server's installer and partition layout. + This establishes a known-good baseline and forces every interface question - + boot contract, partition types, cmdline, verity offsets - into the open + before any compiler runs. The "Boot proof" section below records this + baseline already achieved. +2. **Substitute.** Replace imported binaries with BuildStream-built ones at the + same versions, one component at a time, re-running the same boot proof after + each swap. A component is at parity when the image still boots and behaves + with it built rather than imported. + +The imported base is scaffolding with a purpose: it is the control against +which each BuildStream-built component is measured. It is not the destination. + ## Upstream artifact survey Measured against `https://stable.release.flatcar-linux.net/amd64-usr/4593.2.5/` @@ -94,14 +154,17 @@ Verified contents of `flatcar-container.tar.gz`: ## Design -### Split the two images by role +### The installer and the payload diverge in version, not in base -The installer and the installed OS stop sharing a userspace. +Both images are composed from FSDK 26.08. They differ in which versions they +pin. -- **Installer** stays FSDK 26.08. It is the only consumer of - `systemd-sysinstall`, which does not exist in systemd 257. Hard rule 3 is +- **Installer** tracks FSDK's own versions, including systemd 261. It is the + only consumer of `systemd-sysinstall`, which does not exist in systemd 257, + so it cannot follow the payload down to Flatcar's version. Hard rule 3 is preserved untouched. -- **Installed OS DDI** becomes Flatcar `/usr` plus the Bluefin overlay. +- **Installed OS payload** tracks Flatcar's versions - systemd 257 and the rest + of the parity matrix - still built from FSDK `components/*`. This split is safe because the installer's contract with the DDI is byte-level, not content-level: `files/installer/repart.d/20-root.conf` copies @@ -382,64 +445,92 @@ Ignition needs no masking. ### Versioning -`project.conf` `release-version` is currently derived from and enforced -against the FSDK junction ref by `.github/scripts/check-release-version.py`. -After the split, the FSDK pin describes only the installer, while the OS -payload's real version is `FLATCAR_VERSION`. The release version needs two -axes recorded and enforced separately. This is an invariant change, not a -string edit, and gets its own ticket ahead of any element work. +The release version carries both axes in one string: + +``` +26.08.XX.$FLATCARVERSION +``` + +- `26.08` - the freedesktop-sdk series this image is composed from. +- `XX` - the FSDK point release, parsed from the junction ref by + `.github/scripts/check-release-version.py` exactly as today. +- `$FLATCARVERSION` - the Flatcar release whose component versions this image + targets, from `FLATCAR_VERSION` in the upstream `version.txt`. + +For FSDK 26.08.13 targeting Flatcar 4593.2.5 the version is +`26.08.13.4593.2.5`. Both halves are enforced: the FSDK half against the +junction ref, the Flatcar half against the pin in `include/flatcar.yml`. A +release cannot claim parity with a Flatcar version it does not target, and +cannot claim an FSDK series it is not built from. + +This supersedes the earlier proposal to split `release-version` into two +separate fields. One string, two enforced halves, and the parity claim is +legible at a glance in an artifact filename. -## Second opinion: the version-parity plan +## The version-parity plan, folded in -A competing plan proposes **version parity**: read the component versions -Flatcar ships and rebuild those same versions from source inside BuildStream, -with three kernels (Flatcar LTS, Fedora CoreOS, Ubuntu), a Flatcar-versioned -`k8s` sysext replacing k0s, and A/B slots as `root-a`/`root-b`. +A competing plan proposed version parity directly. It is now the frame of this +document rather than an alternative to it. What it contributes: -Adopted from it: +Adopted as the frame: +- **Version parity as the objective**, with feature parity as a side effect + rather than the target. See "Purpose". +- **Build the matched versions from source in BuildStream**, on the FSDK base. - **Single source of truth for pins.** Extend `include/flatcar.yml` to carry every pinned upstream version, with a single-consumer rule so no element or - script hardcodes one. Good discipline, orthogonal to the base swap. + script hardcodes one. - **Provisioning parity via `systemd-creds`**, covering SSH keys, networkd - configuration, `systemd-firstboot`, and TPM2 sealing. This design keeps - `systemd-sysinstall` + `systemd-creds` and rejects Ignition, which the other - plan agrees with; its provisioning workstream is real work this spec did not - cover. + configuration, `systemd-firstboot`, and TPM2 sealing, in preference to + Ignition. Both plans agree here. - **Reboot coordination for non-Kubernetes hosts**, matching roadmap item 6. -Rejected, with reasons: - -- **"Rebuild Flatcar's versions from source in BuildStream."** Version parity - is not ABI parity. Flatcar's kernel banner reads - `x86_64-cros-linux-gnu-gcc (Gentoo Hardened 14.3.1_p20250801 p4)`; rebuilding - "the same version" under the FSDK toolchain produces different binaries with - different behavior, which is precisely the seam this migration exists to - close. It also means maintaining forks of Flatcar's forks. -- **Three kernels built from upstream tarballs.** The premise that Flatcar has - "Ubuntu-based builds" is not true of any published release. More decisively, - the boot proof above depends on `CONFIG_INITRAMFS_SOURCE="bootengine.cpio"`: - a kernel built from a plain upstream tarball has no embedded initramfs, so - each additional kernel re-creates the initrd problem this design removes. -- **Deleting the k0s sysext for a Flatcar `k8s` sysext.** That plan's own risk - table concedes the Flatcar sysext is "binaries-only, not a full control +Clarified: + +- **Version parity is not ABI parity, and does not aim to be.** Flatcar's + kernel banner reads + `x86_64-cros-linux-gnu-gcc (Gentoo Hardened 14.3.1_p20250801 p4)`; building + the same version under the FSDK toolchain produces different binaries. That + is accepted. The measure of success is that the image boots and behaves, + and that each divergence is recorded - not that bytes match. + +Amended by measurement: + +- **A/B is `USR-A`/`USR-B`, not `root-a`/`root-b`.** Flatcar's pair uses type + GUID `5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6`, with `/` as writable state. + Mirroring a whole-rootfs DDI into a second slot does not satisfy the + initramfs contract. +- **Read-only `/usr` needs no fstab or cmdline `ro` work.** `mount.usrflags=ro` + alone produced `Successfully made /usr/ read-only` in the boot trace. +- **A kernel built from a plain upstream tarball has no embedded initramfs.** + The boot proof depends on `CONFIG_INITRAMFS_SOURCE="bootengine.cpio"`, so any + BuildStream-built kernel must also build a `bootengine.cpio` equivalent and + set that config, or supply an external initrd. This is a concrete parity + requirement, and it multiplies per additional kernel. + +Deferred: + +- **Three kernels (LTS, Fedora CoreOS, Ubuntu).** No published Flatcar release + has an Ubuntu-based build, so there is no upstream version to match for that + third kernel. Revisit once single-kernel parity holds. +- **Replacing the k0s sysext with a Flatcar `k8s` sysext.** That plan's own + risk table concedes the Flatcar sysext is "binaries-only, not a full control plane" and that dropping k0s "removes the single-node k8s story". The KubeStellar console path depends on it. -- **A/B as `root-a`/`root-b`.** Superseded by measurement: Flatcar's A/B pair - is `USR-A`/`USR-B` with type GUID `5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6`, - and `/` is writable state. Mirroring the whole-rootfs DDI into a second slot - does not satisfy the initramfs contract. -- **"Enforce read-only `/usr` via fstab or cmdline `ro`."** Already automatic: - `mount.usrflags=ro` produced `Successfully made /usr/ read-only` in the - trace. No additional mechanism needed. ## Rejected alternatives -- **Rebuild Flatcar from source under `bst`.** Rejected: Flatcar is a - portage/SDK cross-build of thousands of ebuilds. The effort is unbounded and - buys nothing that a pinned, digest-verified binary import does not. -- **Status quo hybrid.** Rejected: it permanently straddles two ABI domains - and requires the identity fiction in `os-release-flatcar.bst` to function. +- **Replacing the FSDK base with Flatcar's `/usr` tree.** Rejected: the build + base stays freedesktop-sdk 26.08 and hard rule 1 is unchanged. Flatcar's + binaries are imported as a *reference* to boot against and measure, not as + the shipped payload. See "Method: import first, then substitute". +- **Reproducing Flatcar's build system** - the Flatcar SDK, portage, or + `coreos-assembler`. Rejected: building with `bst` is the point of the + experiment. Matching Flatcar's *versions* under FSDK is the work; matching + their toolchain would answer a different question. +- **Status quo drift.** Rejected: leaving component versions wherever FSDK + happens to pin them means no parity claim can be made, and + `os-release-flatcar.bst` keeps asserting `ID=flatcar` on a fiction. - **Boot Flatcar's `/usr` with dm-verity, as upstream does.** **Adopted.** Not a follow-on and not optional: stage 1 hardcodes `--hash-offset=1065345024` and expects `verity.usr=` / `verity.usrhash=`, so @@ -452,28 +543,34 @@ Rejected, with reasons: ## Migration phases -Each phase is independently landable and independently verifiable. - -1. **Decision and invariants.** Amend hard rule 1 to scope FSDK composition to - the installer and permit pinned Flatcar binary imports for the OS payload. - Record this design as an ADR. Split the release-version axes. -2. **Import elements.** `flatcar/flatcar-usr.bst` plus a sysext family for - `podman`, `containerd`, `docker`. Contract tests assert the update and - provisioning stack is absent, the module layout is flat, and - `/usr/lib/flatcar/bootengine.img` is preserved. -3. **Partition layout.** Replace `repart.d/10-esp.conf`, `20-root-a.conf`, and - `30-var.conf` with Flatcar's `EFI-SYSTEM` / `USR-A` / `USR-B` / `ROOT` - layout using upstream type GUIDs. Verity stays off at this stage, which - stage 1 explicitly permits, so the layout can be proven on its own. -4. **`/usr` DDI.** The OS payload becomes a `/usr` image sized to the +Each phase is independently landable and independently verifiable. Hard rule 1 +is untouched throughout: the base stays FSDK 26.08. + +1. **Invariants.** Adopt the `26.08.XX.$FLATCARVERSION` release version with + both halves enforced. Extend `include/flatcar.yml` to the single source of + truth for pins. Record this design as an ADR. +2. **Version audit.** Inventory the component versions Flatcar 4593.2.5 ships, + from `flatcar_production_image_packages.txt` and the SBOM, and diff them + against what FSDK 26.08 pins. The output is a parity matrix: component, + Flatcar version, FSDK version, gap. This decides the order of every phase + after it, and nothing downstream should start before it exists. +3. **Reference import.** Bring Flatcar's binaries in as pinned artifacts to + boot against, per "Method: import first, then substitute". They are the + oracle, not the payload. Contract tests assert + `/usr/lib/flatcar/bootengine.img` is present and the module layout is flat. +4. **Partition layout.** Replace `repart.d/10-esp.conf`, `20-root-a.conf`, and + `30-var.conf` with `EFI-SYSTEM` / `USR-A` / `USR-B` / `OEM` / `ROOT` using + upstream type GUIDs. Verity stays off here, which stage 1 explicitly + permits, so the layout is provable on its own. +5. **`/usr` payload.** The OS payload becomes a `/usr` image sized to the 1,065,345,024-byte budget with the verity hash tree appended, and its root - hash is baked into the UKI cmdline as `verity.usrhash=`. -5. **Boot proof.** `just show-me-the-future` installs and boots on the kernel's - built-in initramfs with Ignition masked; the Lima end-to-end test drives the - KubeStellar console login against it. -6. **Cutover.** `os-stack.bst` switches to the Flatcar base; the displaced FSDK - elements and the `dracut` target-initrd path are deleted. -7. **Follow-on.** Wire `50-root.transfer` to the real `USR-A`/`USR-B` slots and + hash baked into the UKI cmdline as `verity.usrhash=`. +6. **Substitution.** Working down the parity matrix, replace each imported + binary with an FSDK-built component at Flatcar's version, re-running the + boot proof after each swap. Record every behavioral divergence. +7. **Boot proof in CI.** `just show-me-the-future` installs and boots; the Lima + end-to-end test drives the KubeStellar console login against the result. +8. **Follow-on.** Wire `50-root.transfer` to the real `USR-A`/`USR-B` slots and retire the `xfail` in `tests/unit/test_repart_layout.py`; arm64. ## Verification From 444468c1b7c8a50ea060a9f17b05874a98d30893 Mon Sep 17 00:00:00 2001 From: castrojo Date: Sun, 13 Sep 2026 19:50:48 -0400 Subject: [PATCH 8/8] docs(spec): use Flatcar's version verbatim as release-version release-version becomes the Flatcar release the image targets, 4593.2.5, not a composite and not a translation. If the image claims parity with Flatcar 4593.2.5 it says so. The tree is already half-way there and inconsistent with itself: os-release-flatcar.bst sets VERSION_ID and CPE_NAME from FLATCAR_VERSION while PRETTY_NAME carries the unrelated release-version, so one image answers two different questions about its own version. This collapses them. It also makes sysext attachment honest. Flatcar sysexts match on ID and VERSION_ID, and flatcar-zfs.raw attaches today only because the identity file already asserts Flatcar's. With release-version equal to the Flatcar version the match rests on a true statement. check-release-version.py inverts: it stops enforcing against the FSDK junction ref and starts enforcing against flatcar-version in include/flatcar.yml. The FSDK pin stays for provenance in fsdk_ref and gets its own os-release field rather than being smuggled into the version string. Prerelease suffixes keep working, so an alpha is 4593.2.5-alpha.1. Supersedes both the two-field split and the 26.08.XX.$FLATCARVERSION composite. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-09-13-flatcar-base-migration-design.md | 57 ++++++++++++------- 1 file changed, 37 insertions(+), 20 deletions(-) diff --git a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md index 3a52b1a8..d1f37148 100644 --- a/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md +++ b/docs/superpowers/specs/2026-09-13-flatcar-base-migration-design.md @@ -445,27 +445,43 @@ Ignition needs no masking. ### Versioning -The release version carries both axes in one string: +**Use Flatcar's version.** `release-version` is the Flatcar release the image +targets, verbatim: ``` -26.08.XX.$FLATCARVERSION +4593.2.5 ``` -- `26.08` - the freedesktop-sdk series this image is composed from. -- `XX` - the FSDK point release, parsed from the junction ref by - `.github/scripts/check-release-version.py` exactly as today. -- `$FLATCARVERSION` - the Flatcar release whose component versions this image - targets, from `FLATCAR_VERSION` in the upstream `version.txt`. - -For FSDK 26.08.13 targeting Flatcar 4593.2.5 the version is -`26.08.13.4593.2.5`. Both halves are enforced: the FSDK half against the -junction ref, the Flatcar half against the pin in `include/flatcar.yml`. A -release cannot claim parity with a Flatcar version it does not target, and -cannot claim an FSDK series it is not built from. - -This supersedes the earlier proposal to split `release-version` into two -separate fields. One string, two enforced halves, and the parity claim is -legible at a glance in an artifact filename. +Not a composite, not a translation. If the image claims parity with Flatcar +4593.2.5, it says `4593.2.5`. + +This is already half-true in the tree. `os-release-flatcar.bst` sets +`VERSION_ID=${FLATCAR_VERSION}` and `CPE_NAME=cpe:/o:flatcar-linux:flatcar_linux:${FLATCAR_VERSION}` +while `PRETTY_NAME` carries the unrelated `%{release-version}`. One image +currently answers two different questions about what version it is. Adopting +Flatcar's version collapses them. + +It also makes the sysext story honest. Flatcar's `systemd-sysext` images match +on `ID` and `VERSION_ID`; `flatcar-zfs.raw` attaches today only because +`os-release-flatcar.bst` already asserts Flatcar's identity. With +`release-version` equal to the Flatcar version, `flatcar-podman.raw`, +`containerd-flatcar.raw`, and the rest attach on a true statement rather than +a convenient one. + +Consequences: + +- `.github/scripts/check-release-version.py` inverts. It stops enforcing + `release-version` against the FSDK junction ref and starts enforcing it + against `flatcar-version` in `include/flatcar.yml`. +- The FSDK pin does not disappear, it stops being user-facing. It remains in + the junction ref and in `fsdk_ref` for provenance, and is recorded in + `os-release` as its own field rather than smuggled into the version string. +- Artifact names follow: `bluefin-server-ddi-4593.2.5.raw.zst`. +- Prerelease suffixes keep working. `check-release-version.py` already allows + them, so an alpha is `4593.2.5-alpha.1`. + +This supersedes both earlier proposals: the two-field split and the +`26.08.XX.$FLATCARVERSION` composite. ## The version-parity plan, folded in @@ -546,9 +562,10 @@ Deferred: Each phase is independently landable and independently verifiable. Hard rule 1 is untouched throughout: the base stays FSDK 26.08. -1. **Invariants.** Adopt the `26.08.XX.$FLATCARVERSION` release version with - both halves enforced. Extend `include/flatcar.yml` to the single source of - truth for pins. Record this design as an ADR. +1. **Invariants.** Adopt Flatcar's version as `release-version` and invert + `check-release-version.py` to enforce it against `include/flatcar.yml`. + Extend that file to the single source of truth for pins. Record this design + as an ADR. 2. **Version audit.** Inventory the component versions Flatcar 4593.2.5 ships, from `flatcar_production_image_packages.txt` and the SBOM, and diff them against what FSDK 26.08 pins. The output is a parity matrix: component,