docker: fix linux/arm64 images shipping amd64 binaries - #1216
Open
starius wants to merge 12 commits into
Open
Conversation
The final stage was pinned to ${BUILDPLATFORM}, so every platform
requested from buildx was built on the build machine's architecture.
The published image index then advertised linux/arm64 for a filesystem
that was in fact amd64, and the build reported success while doing it.
Every stage resolves to the target platform by default, so the pin is
simply dropped. This corrects the root filesystem of the image. The
binaries copied into it are still built for the build machine and are
cross-compiled in the following commit.
The builder stage stays pinned to ${BUILDPLATFORM} so that the Go
toolchain is never emulated, which means it has to be told what to
build for. Without GOOS and GOARCH it produced binaries for the build
machine, and an arm64 image holding amd64 binaries fails with "exec
format error" as soon as loopd is started.
The Makefile already builds with CGO_ENABLED=0, so this needs no cross
C toolchain. When cross-compiling, "go install" writes to
$GOPATH/bin/$GOOS_$GOARCH rather than $GOPATH/bin, so the binaries are
moved back to keep the COPY that follows platform independent. GOBIN
cannot be used for that: setting it while cross-compiling is a hard
error in the Go tool.
Nothing checked the images this workflow publishes. A multi-architecture build that produces the build machine's architecture for every platform exits successfully, so the mislabelled arm64 images went out unnoticed across every release since the workflow was added. The build now takes two outputs: a local OCI layout, and an untagged push that stages the same build in the registry. The layout is checked, the staged image is confirmed to be made of exactly the manifests that were checked, and only then is the digest given the release tag. Nothing is reachable under that tag until all of that has passed. For each platform entry the check confirms that the image config agrees with the index and that the ELF header of every binary names the advertised architecture. It looks at busybox from the base image as well as the two binaries the build produces, so a final stage built for the wrong platform is caught even if the binaries copied into it are right. The check inspects the artifact rather than running it, because running it cannot detect this on an amd64 runner: the amd64 binaries inside a mislabelled arm64 entry execute natively, so a smoke test passes.
The builder installed alpine-sdk, a meta package that pulls in a full C toolchain across 46 packages and roughly 290 MiB, on every build. It was presumably added for cgo, but the Makefile builds with CGO_ENABLED=0 and the only cgo sensitive dependency, modernc.org/sqlite, is pure Go, so no C compiler is involved. Both binaries in the published images are statically linked. The packages the build actually needs, git and make, are already named explicitly on the same line, so only the meta package is removed. Both platforms still build and produce the same statically linked binaries.
GODEBUG=netdns=cgo was set in the builder stage, and a stage inherits no environment from an earlier one, so it never reached the image that is actually shipped: the config of both the published image and a locally built one lists PATH and nothing else. The comment above it described a runtime concern, so the intent was clearly the final stage. Moving it there would not help either. The binaries are built with CGO_ENABLED=0 and are statically linked, so the cgo resolver is not compiled in and the pure Go resolver is used whatever GODEBUG asks for. Honouring the original intent would mean building with cgo enabled, which is a much larger change and one nothing appears to need.
Module mode has been the default since Go 1.16 and the image builds with Go 1.26, so setting GO111MODULE=on has no effect. Its own comment said it was there "until this becomes the default", which happened nine releases ago. The Makefile also sets it on the go install line, so nothing depends on it being in the environment.
BuildKit reports a FromAsCasing warning on every build for each stage that spells the keyword as lowercase "as". With the two obsolete ENV lines gone, these were the last two warnings the build printed; "docker buildx build --check" is now clean.
The workflow passed checkout=<tag> as a build argument, but nothing has ever read it: this Dockerfile takes no such argument. It appears to have been copied from lnd, whose Dockerfile does take one because it clones the repository inside the build. The value would add nothing either way. The default context that build-push-action uses is the repository git URL with the pushed ref appended, so the tag being built is already pinned by the context.
shellcheck reports SC2086 twice for this workflow, for the unquoted redirection target in the Set env step. The variable holds a path chosen by the runner, so word splitting has never bitten in practice, but these were the only two findings actionlint reported for the file, which is now clean.
The images published so far report no commit at all: running "loopd --version" from the released image prints "commit= commit_hash=". The Makefile fills those in from git describe and git rev-parse, but BuildKit's git context clones without keeping the .git directory, so every one of those shell calls returned an empty string. Passing BUILDKIT_CONTEXT_KEEP_GIT_DIR=1 keeps the directory in the context. Verified against the real context for the v0.35.0-beta tag: without the argument .git is absent and git describe fails, with it git describe returns v0.35.0-beta. The .git directory only ever exists in the builder stage, so the published image does not grow.
This Dockerfile started as a copy of lnd's and several comments still describe that image rather than this one. One of them names the wrong project, one names ports that do not do what it says, and one refers to an entrypoint that is not copied because this image has none. The ports are corrected from "server, rpc" to what loopd actually listens on in the order they are exposed: 8081 is the REST listener and 11010 the gRPC one, per the defaults in loopd/config.go. The comment above the last stage is also extended to mention the CA certificates it installs alongside bash.
The builder passed both --no-cache and --update to apk. The latter is an abbreviation of --update-cache, itself an alias for --cache-max-age 0, which has nothing to act on once --no-cache has ruled out a persistent index cache. Installing git and make with and without it in the same base image yields byte identical package sets. The final stage of this file already uses --no-cache on its own, so this also makes the two stages consistent.
starius
marked this pull request as ready for review
August 28, 2026 04:15
bhandras
approved these changes
Aug 28, 2026
hieblmi
reviewed
Aug 28, 2026
Collaborator
There was a problem hiding this comment.
Thanks for fixing this! There are two findings:
Ancestor whiteouts do not remove discovered binaries. In check_multiarch_image.py, .wh.bin becomes wh.bin because lstrip("./") removes the leading dot. Additionally, deleting or replacing /bin does not clear /bin/* from found. I reproduced an image whose final layer deletes /bin; the validator still reported all three binaries and exited successfully.
Unexpected platforms are promoted without validation. The loop at lines 394–400 checks only platforms named by --platforms, but writes every manifest to --digests-out. I reproduced an extra linux/386 manifest containing amd64 binaries; it was included in the digest list while the validator exited successfully. Either reject extra platforms or run check_entry for every non-attestation entry.
GustavoStingelin
left a comment
GustavoStingelin
left a comment
Contributor
There was a problem hiding this comment.
looking good, just the same findings from heebs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1211 and other related issues.
The published
linux/arm64images have never held arm64 code. The Dockerfile pinned both of its stages to${BUILDPLATFORM}, so every platform buildx was asked for got built on the release runner's amd64 architecture and then labelled arm64, which buildx reports as a perfectly successful build. Runningloopdfrom one of those images on ARM hardware fails withexec format error. The final stage now resolves to the target platform, and the builder stage, which stays native so the Go toolchain is never emulated, cross-compiles to it.Nothing in the release workflow ever looked at what it was publishing, which is why this went unnoticed across every release since the workflow was added. The build now emits a local OCI layout alongside an untagged push, the layout is checked for binaries whose ELF headers match the architecture each manifest entry advertises, the staged image is confirmed to consist of exactly the manifests that were checked, and only then does that digest get the release tag. Note that a smoke test could not have caught this, since amd64 binaries in a mislabelled arm64 image run natively on an amd64 runner.
Released images also reported no build metadata, printing
commit= commit_hash=, because BuildKit's git context drops the.gitdirectory that the Makefile reads to stamp it in. Keeping that directory restores the commit inloopd --version.Pull Request Checklist
docs/release-notes/release-notes-next.md, or apply theno-changeloglabel (required by CI)