Skip to content

ci(lima): automated nested-virt QEMU installation and boot verification - #119

Open
mrbobbytables wants to merge 3 commits into
projectbluefin:mainfrom
mrbobbytables:ci-lima-nested-virt
Open

ci(lima): automated nested-virt QEMU installation and boot verification#119
mrbobbytables wants to merge 3 commits into
projectbluefin:mainfrom
mrbobbytables:ci-lima-nested-virt

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Resolves #115
Supersedes #113 (and resolves #111, #101)

Summary

Migrates the lima-e2e CI workflow from macos-latest to ubuntu-24.04 with nested virtualization and QEMU/KVM acceleration to resolve the VZ hostagent exit failure on Apple Silicon GitHub runners:

  1. Configures the Lima template (.github/scripts/bluefin-lima-template.yaml) with vmType: qemu, arch: x86_64, and nestedVirtualization: true.
  2. Updates .github/workflows/lima-e2e.yml to run on ubuntu-24.04 with KVM permissions (/dev/kvm), installs QEMU (qemu-system-x86, qemu-utils, ovmf) and Lima 2.2.0.
  3. Provisions guest permissions for repository copying and automated unattended installation and target boot verification.
  4. Updates unit test coverage in tests/unit/test_lima_template.py to validate vmType and arch schema definitions alongside limactl validate.

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 0fa460f

kylerankin and others added 3 commits September 13, 2026 21:09
Add a GitHub Actions job that verifies the full installer chain inside a
nested-virt Lima VM on a macOS runner, since the runner has no native KVM:

  Source elements -> Installer Raw Image ->
  systemd-sysinstall auto-partitioning & DDI copy (Lima VM) ->
  Installed target disk boot (Lima VM) -> multi-user.target / login prompt

- .github/workflows/lima-e2e.yml: macOS job that installs Lima + QEMU, starts
  a nested-virt VM, and runs the verification there. Read-only token (PR code).
- .github/scripts/lima-e2e-install-boot.sh: validate + build/export the
  installer, boot it unattended (systemd-sysinstall auto-partition + DDI copy),
  then boot the installed target and verify it reaches multi-user.target.
- .github/scripts/bluefin-lima-template.yaml: Lima template with
  nested: enabled to expose /dev/kvm to the guest.

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>
Assisted-by: lemonade/Ornith-1.5-35B-A3B-GGUF-Q6_K via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>
Update the Lima template schema to resolve limactl validate errors:
- Extend base with template:_images/ubuntu-24.04
- Use nestedVirtualization: true instead of legacy nested: enabled
- Remove unknown localShell field under ssh
- Clean up containerd configuration to standard schema (system/user: false)
- Add limactl validate step in CI workflow before VM creation
- Add unit test coverage in tests/unit/test_lima_template.py

Fixes projectbluefin#111

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Migrate lima-e2e CI workflow from macos-latest to ubuntu-24.04 with
nested virtualization and QEMU/KVM acceleration:
- Use ubuntu-24.04 Linux runner with /dev/kvm hardware virtualization
- Configure Lima template with vmType: qemu and arch: x86_64
- Install QEMU (qemu-system-x86, qemu-utils, ovmf) and Lima 2.2.0 in CI job
- Ensure guest repository path is provisioned with appropriate permissions
- Update test_lima_template.py unit tests to verify vmType and arch schema

Resolves projectbluefin#115
Supersedes projectbluefin#113 (and resolves projectbluefin#111, projectbluefin#101)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The move from macos-latest to ubuntu-24.04 with host KVM is the right call, and the --tty=false / limactl delete --force / cp -r fixes are all real. Four things before merge.

1. chmod -R 777 /home/ubuntu will likely break limactl shell

.github/scripts/bluefin-lima-template.yaml, mode: system provision:

mkdir -p /home/ubuntu/repo
chmod -R 777 /home/ubuntu 2>/dev/null || true

That recurses into /home/ubuntu/.ssh. With StrictModes yes (the sshd default), a world-writable home directory or authorized_keys makes sshd refuse public-key auth — and every limactl shell / limactl cp after this provision runs goes over ssh. The narrow form is what's actually needed:

install -d -o ubuntu -g ubuntu -m 0755 /home/ubuntu/repo

2. The KVM setup cannot fail

sudo modprobe kvm || true
sudo chown $(whoami) /dev/kvm 2>/dev/null || true
sudo chmod 666 /dev/kvm 2>/dev/null || true

All three swallow failure, so a runner without /dev/kvm proceeds and QEMU silently falls back to TCG. Against a 240-minute timeout and a double boot, that's a four-hour job that eventually fails for an unrelated-looking reason. Please assert instead:

test -e /dev/kvm || { echo "::error::/dev/kvm unavailable on this runner"; exit 1; }

and prefer sudo usermod -aG kvm "$USER" over chmod 666 on a device node.

3. Lima is downloaded unpinned while every action here is SHA-pinned

LIMA_VERSION="2.2.0"
curl -fsSL ".../lima-${LIMA_VERSION}-Linux-x86_64.tar.gz" | sudo tar -xz -C /usr/local

actions/checkout and taiki-e/install-action in this same file are pinned by commit SHA. Piping an unverified tarball straight into sudo tar -xz -C /usr/local is a weaker link than either. Please add a sha256sum -c against the published checksum.

4. The paths: filter misses the changes this job exists to catch

elements/installer/**
elements/oci/bluefin-server-installer.bst
elements/oci/bluefin-server-ddi.bst
files/installer/**

elements/bluefin-server/** and elements/flatcar/** are not listed, so this workflow will not run on #132 (rewrites os-stack.bst), #140 (adds flatcar-usr.bst) or #141 (adds three sysext elements) — the three open PRs most likely to change whether the image boots. A blanket elements/** plus project.conf would close that.

Worth doing here rather than later, because this PR is the one introducing the boot gate, and the Flatcar migration set is landing right behind it.

Sequencing

#113 is a strict subset of this PR — it carries the same two commits (7da3439, 9b192fd) and nothing else. Merging both into main conflicts on all four files. Suggest closing #113 in favour of this one; I've left a note there.

Verified locally on this branch: pytest tests/unit 214 passed / 1 xfailed, bats tests/unit 62 ok, docs-checks.py passed. I did not run the workflow itself — no KVM available here.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci(lima): automated nested-virt QEMU installation and boot verification ci(lima): fix Lima template schema errors in CI workflow

3 participants