vms-101: run heavy builds/tests on the k3s rail (run-on-rail.sh) - #673
vms-101: run heavy builds/tests on the k3s rail (run-on-rail.sh)#673baron-3dl wants to merge 5 commits into
Conversation
Offload cmake/ctest/QEMU-KVM smokes from the shared workshop host onto the k3s rail. BUILD/TEST TOOLING ONLY (Rule 9) -- never an OVMX runtime. - tools/k3s/run-on-rail.sh <git-ref> <command...>: clone the repo at the ref in a pod on k3s-worker, stream logs live, exit with the command's exit code. --keep leaves the Job for debugging. - tools/k3s/Dockerfile.rail: builder image (ubuntu:24.04 + the tests/qemu toolchain + git + genisoimage), pushed to the in-cluster registry. - tools/k3s/namespace.yaml: ovmx-ci namespace + ResourceQuota (~6 CPU / 40Gi) + LimitRange so CI never starves the shared tenants. - tools/k3s/job-template.yaml: privileged Job pinned to k3s-worker with hostPath /dev/kvm for KVM-accelerated QEMU; command passed base64 so arbitrary shell never touches the YAML; pod exit code = command exit code. - tools/k3s/kvm-smoke.sh: boots a kernel under strict -accel kvm and asserts the guest logs "Hypervisor detected: KVM" (KVM proof, not TCG fallback). - tools/k3s/README.md: registry push/pull ref split + KVM notes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…generous quota Verified end-to-end on k3s-worker: clone-at-ref, run command, stream logs, propagate exit code (exit 7 -> script exit 7). - Image ref fixed to 192.168.2.43:30500/ovmx-builder:latest (the k3s-cp NodePort). The in-cluster Service DNS name ImagePullBackOffs — node containerd resolves image hosts via node resolv.conf, not CoreDNS. Push must also use .43 (workshop daemon trusts only that NodePort as insecure HTTP). - Per-pod request lowered to 1 CPU / 4Gi so jobs schedule on k3s-worker (shared tenants already hold ~5 of its 8 CPU); build still bursts to the 6-CPU limit. - run-on-rail.sh surfaces FailedScheduling / pull failures instead of hanging, and never streams logs from a still-Pending pod. - ResourceQuota made generous (operator: rail capacity is ample) so it never gates a wave of parallel jobs; node free-CPU still bounds real concurrency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified on k3s-worker: full build (232 targets) + `ctest` = 100% passed, 0 failed out of 195, exit 0. Real exit-code propagation also proven (a red ctest returned rc 8 -> script exit 8). Added the deps the plain-ctest path needs that the tests/qemu toolchain layer lacked (the authenticity symbol-scan gate compiles every product source, so a missing build header is a hard FAIL, not a skip): - libssh-dev, libreadline-dev (src/vmsssh, DCL readline) - strace, gawk, binutils (integration gates: monitor/show-device/register) - ccache, clang (match ci.yml's ctest job env) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reap's remaining e2e verification is docker-wrapped (docker build distro/Dockerfile.bootable then docker run ... qemu) and SKIP(77)s in the plain builder pod for lack of docker. --dind starts an in-container dockerd so those gates run on cluster hardware. - run-on-rail.sh --dind: sets OVMX_DIND=1; the pod starts dockerd (overlay2, vfs fallback) and waits for it before running the command. Non-dind path unchanged. - dockerd runs IN the same container as the repo (not a sidecar) on purpose: the e2e gates use `docker run -v $REPO_ROOT/...` bind mounts, which resolve on the DAEMON's filesystem -- a sidecar daemon would resolve them on the wrong host and break the mounts. - job-template.yaml: OVMX_DIND env + a node-backed emptyDir at /var/lib/docker (so overlay2 works off the pod rootfs) + gated dockerd preamble. /dev/kvm is already in the pod, so an inner `docker run --device /dev/kvm ... -accel kvm` is KVM-accelerated (nested KVM). - Dockerfile.rail: add docker.io iptables uidmap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The --dind image now ships dockerd (Dockerfile.rail already carried docker.io; the registry image just predated it and has been rebuilt + pushed). This wires the in-pod dockerd to the same plain-HTTP registry the nodes' containerd trusts (192.168.2.43:30500) via --insecure-registry, so an inner `docker pull`/`docker run <registry-image>` resolves in --dind -- needed for the nested-KVM proof that boots the builder image under `docker run --device /dev/kvm ... -accel kvm`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Closed per operator directive: NO GitHub Actions on the mainframe. The self-hosted-GitHub-runners approach hauls GitHub CI infrastructure onto our platform, which we don't want. Big/heavy runs will instead run NATIVELY on the k3s rail as kubectl-driven k3s Jobs with KVM (extending the lab-2 pattern that already runs VMScluster labs on the rail) — zero GitHub on-platform. vms-101 re-scoped to the native-rail-run harness. |
|
Reopening — closed in error. On assessment, #673 does NOT haul GitHub Actions onto the mainframe: the rd title was stale ('self-hosted GitHub runners'), but the operator overrode that 2026-08-20 ('no self-hosted GitHub actions, just use k3s to run hard work') and #673 implements THAT corrected design — a native k3s-Job dev-loop offload (run-on-rail.sh clones + builds in a privileged rail pod pinned to k3s-worker with hostPath /dev/kvm, kubectl-driven, streams logs, propagates exit code; --dind for docker-wrapped e2e). No GitHub runner registration, no PAT/secret, no runs-on flip. This is exactly the 'big runs natively on the rail' the operator wants. Additive tooling-only (new tools/k3s/ dir), mergeable, zero CI-behavior change. |
|
Conductor triage 2026-08-30: tools/k3s/ on this branch is BYTE-IDENTICAL to main (the rail tooling landed via #689), so this PR reaps no code. Keeping it OPEN only as the tracking anchor for the live disk-crisis escalation (root fs 99%). Disposition is the operator's: if the rail is prioritized as disk relief, re-scope this to the net-new bits; otherwise close. Not an autonomous reap. |
1 participant
Offload heavy OVMX builds/tests (
cmake,ctest,tests/qemu/run_tests.sh, QEMU-KVM smokes) from the sharedworkshophost onto the local k3s rail. Tracks vms-101.BUILD/TEST TOOLING ONLY — per CLAUDE.md Rule 9, k3s here is only a place to run builds/tests on cluster hardware; it is never an OVMX runtime and is not presented as one.
What lands (all under
tools/k3s/)run-on-rail.sh <git-ref> <command...>— clones the repo at<git-ref>in a pod on k3s-worker, streamskubectl logs -flive, and exits with the command's exit code.--keepleaves the Job for debugging.Dockerfile.rail— the builder image:ubuntu:24.04+ the same toolchain layer astests/qemu/Dockerfile(build tools + qemu + kernel pkgs + musl + cmake) plusgitandgenisoimage/xorriso. Pushed to the in-cluster registry.namespace.yaml—ovmx-cinamespace + ResourceQuota (~6 CPU / 40Gi) + LimitRange so CI never starves the shared tenants.job-template.yaml— privileged Job pinned tok3s-workerwith hostPath/dev/kvmfor KVM-accelerated QEMU. The command is passed base64-encoded so arbitrary shell never touches the YAML; the pod's exit code is the command's exit code.kvm-smoke.sh— boots a kernel under strict-accel kvmand asserts the guest logsHypervisor detected: KVM(positive KVM proof, never a TCG fallback).README.md— registry push/pull ref split + KVM notes.Verification
Proven on the rail (not locally) — full
cmake+ctest, a KVM-accelerated QEMU smoke, and exit-code propagation. See the PR thread / task report for the in-cluster tails (node/pod names +Hypervisor detected: KVM).🤖 Generated with Claude Code