build: build the firmware with CMake - #1967
Conversation
The steps that turn resources into the firmware's tables, headers and pbpack were written as waf task classes, so nothing else could call them. Move the logic into tools/resources/generators.py, as plain functions over paths, and leave the task classes as the thin wrappers they should have been. The SDK build, which keeps using waf, gets the new module bundled alongside the waftools it already ships. Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
Build performance, measured against wafI profiled the build and pushed three commits on top of the migration. All
Against waf that is 31 % off a clean build, 5× off a no-op and 2× off the The firmware is unchanged by all three commits: same Where the time was goingClean build: 63 % compiling firmware, 18 % generating resources, 15 % What the three commits do
Left on the tableThe remaining 63 % is header fan-out — 57 % of every compile is the |
92a7457 to
d2f8a2c
Compare
Replace the waf firmware build with CMake, driven by Ninja. The model
is the one waf had grown into, which was itself borrowed from Zephyr, so
the shape of a directory's build file is unchanged: declare a library,
give it sources, gate the optional pieces on Kconfig symbols.
pbl_library()
pbl_library_sources(service.c)
pbl_library_sources_ifdef(CONFIG_MAG correction.c)
What the build does is unchanged too. Kconfig still resolves the board's
configuration into .config and autoconf.h; the resource pipeline, the
generated tables and headers, the linker script hooks, the C library
selection and the image and bundling steps all keep their behaviour, and
the firmware they produce is the same: for qemu_flint, byte-identical
system resources, and an ELF with the same symbols at the same sizes,
differing only in padding inside .text.
The generators the build shells out to move from waf task classes to
plain command line tools under tools/cmake/, and the helpers that were
never waf-specific (board parsing, the pblboot header, SLE) move out of
tools/waf/ next to the rest of the tooling.
waf stays for what it still builds: the unit tests, out of their own
build directory, and the SDK packaging (`./pbl waf sdk`).
The object file names change with the build system, and the sf32lb52x
ramfunc fragment selects the code that has to run from RAM by matching
them: waf wrote bf0_hal_mpi.c.1.o, CMake writes bf0_hal_mpi.c.obj. Its
patterns end wild so they match either.
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
The firmware workflows call ./pbl, which now drives CMake; the test workflow keeps waf and follows the unit tests into build-test/. Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
Rewrite the build system page around CMake and the pbl_library API, point the testing and SDK pages at their new build directory and commands, and add CMake to the nix development shell. Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
Every one of the board's several hundred resources was a build rule of its own, and each rule was a Python process that spent more time starting up than generating. 120 resources cost 9.8 s of CPU one process at a time and 0.5 s in a single process; the interpreter and its imports are 77 ms of the roughly 100 ms a resource takes. Group them by type -- so a process still imports only the one generator module it needs -- and build them sixteen at a time. That is 40 rules instead of 581, small enough that they still spread across the cores, and takes about 45 s of CPU out of a clean build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
Two build outputs nobody asks for most of the time. -g3 keeps every macro definition in the debug info, which costs about 7% of the compile and grew the ELF from 17 to 24 MB, most of it a 3.4 MB .debug_macro section that also slows the link down. -Wl,-Map with --cref writes tens of megabytes and is about a fifth of the link; the only thing that reads it is tools/analyze_fw_static_memory_usage.py. Both are now Kconfig symbols, off by default. The firmware itself does not change: same loadable sections, same symbols at the same addresses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
The version header is regenerated on every build, so that a new commit always reaches the firmware, and `git describe --dirty` was 0.24 s of the 0.31 s that took. All of it is the walk into the submodules: the same check that ignores their working trees is 0.02 s. Ask git for the tag and the dirty flag separately, and let the flag ignore uncommitted edits inside a submodule -- a submodule moved to another commit is still a change, and still counts. Combining the commit and timestamp lookups into one process saves another call. A no-op build goes from 0.72 s to 0.46 s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
The firmware needs CMake and Ninja, which the v6 image does not carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
They were built from execute_process() while CMake was still configuring, which is not where a compile belongs: it needs a host compiler, has nothing to do with resolving the configuration, and any failure aborts the configure rather than the build. waf ran them as a build task; this puts them back there, as the custom command that produces mc.xs.c and mc.resources.c. Configuring qemu_emery with CONFIG_SHELL_SDK=y drops from minutes to 1.6 s as a result, since the host tools are no longer built to answer a question nobody asked. Moving them also means the headers mcconfig writes are no longer there before the first compile, so the generated set -- mc.defines.h and the rest, which xsHost.h pulls in -- hangs off pbl_generated_headers, which every library already waits for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
set(ENV{PATH}) only reaches what CMake runs while it is configuring. The
generators shell out to the toolchain by name -- the native SDK asks for
arm-none-eabi-gcc, the stored apps' metadata injection for
arm-none-eabi-nm -- so on a machine where the toolchain is not already on
PATH, which is every CI runner, they failed as soon as they ran as build
steps rather than configure ones.
Every generator now runs behind `cmake -E env PATH=...`, with the PATH the
SDK locator resolved at configure time. The steps that still run while
configuring are left alone: they already inherit it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
A Pebble app is a single blob: the firmware loads it into app RAM and runs it there, so pebble_app.ld gives it one rwx region and everything lands in one LOAD segment. binutils 2.39 warns about that. Splitting the segment would be worse than the warning -- ld pads between LOAD segments, and the flat image objcopy produces is what the firmware loads -- so tell ld the permissions are deliberate. The option is only passed when the linker knows it, and the app binary is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
A <path> whose d is only a move draws nothing, and the icon set has 104 of them across 49 files -- stray anchor points the drawing tools leave behind. Skipping them loses no ink, so the generator printed a hundred lines of "No points in parsed path" per build, with no file named and nothing to act on. Say it only under --verbose, and say which path it was. The diagnostics that do mean dropped artwork -- unsupported elements, unrecognized circles -- still print unconditionally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
check-size exists to fail the build when the image no longer fits; the running total it printed on every build was noise. The size limit is still enforced, and still says how far over it is when it trips. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
FW_FLASH_SIZE was the whole 1 MiB, but the image is linked from FLASH_OFFSET, 32 KiB in, past the bootloader -- so the FLASH region ran to 0x108000, past the end of the part, and the linker would accept an image 32 KiB larger than fits. Every other board already subtracts what it reserves. The link now reports 992 KiB, and the region is the one the image is flashed into. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
The linker enforces the same limit from the FLASH region and fails the link with the overflow, so checking the size of the image afterwards only repeated the answer -- and printed it on every build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
The last copy of the limits, and the same reasoning: the linker will not produce an image that does not fit its FLASH region, so re-measuring the binary before flashing it answered a question already settled. The exception it raised was never caught either, so an oversize image would have come out as a traceback. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
Interrupting a build unwound through the subprocess wait() it landed in and printed a traceback of Python internals, which says nothing about what was interrupted. Ctrl+C reaches the whole process group, so ninja has already stopped and said so; report 128 + SIGINT, as a shell expects from an interrupted command, and let its message be the last word. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
Replaces the waf firmware build with CMake, driven by Ninja.
What changes
The build model is the one waf had grown into, which was itself borrowed
from Zephyr, so the shape of a directory's build file is unchanged —
declare a library, give it sources, gate the optional pieces on Kconfig:
What the build does is unchanged too. Kconfig still resolves the board's
configuration into
.configandautoconf.h; the resource pipeline, thegenerated tables and headers, the linker script hooks, the C library
selection and the image and bundling steps all keep their behaviour.
The generators the build shells out to move from waf task classes to plain
command line tools under
tools/cmake/, and the helpers that were neverwaf-specific (board parsing, the pblboot header, SLE) move out of
tools/waf/next to the rest of the tooling.waf stays for what it still builds: the unit tests, out of their own build
directory, and the SDK packaging (
./pbl waf sdk).docs/development/build_system.mdis rewritten around the new API.Verification
qemu_flint: byte-identical system resources, and an ELF with the samesymbols at the same sizes, differing only in padding inside
.text.asterix: identical.text/.bss/.log_stringssizes and anidentical symbol table.
Why CMake and not Meson
A full Meson port was written and benchmarked against this branch before
settling. Same board, same machine, ccache off:
Both emit Ninja, so the build itself is a wash. The gaps are all
configure-time, and the deciding factor was expressiveness: Meson has no
user-defined functions, so
pbl_library()becomes a dict literal the rootassembles later, which added ~950 lines to the per-directory files
contributors actually touch. It also has no per-source compile flags
(
__FILE_NAME_LEGACY__needed a compiler wrapper, which rules outccache), no globbing and no
CONFIGURE_DEPENDS, cannot read a file out ofthe build directory, cannot name a subdirectory in a
custom_targetoutput, and has one flat variable scope shared across
subdir().