Skip to content

unix: keep the micropython patch applied so a submodule checkout can't break the build - #728

Open
hitechhayekian wants to merge 1 commit into
Coldcard:masterfrom
hitechhayekian:mpy-patch-durability
Open

unix: keep the micropython patch applied so a submodule checkout can't break the build#728
hitechhayekian wants to merge 1 commit into
Coldcard:masterfrom
hitechhayekian:mpy-patch-durability

Conversation

@hitechhayekian

Copy link
Copy Markdown
Contributor

Thanks for contributing to COLDCARD source code, please be as descriptive as possible.

By submitting this work you agree to the Individual Contributor License Agreement (CLA)


Summary

macos-mpy.patch and ubuntu24_mpy.patch are applied by hand into the external/micropython
working tree and are never committed anywhere. Anything that checks that submodule out drops them
silently — and the next build then fails hundreds of lines into the compile, on -Werror
diagnostics that point at untouched upstream code in py/. Nothing in that output mentions the
patch, the submodule, or the README step that was undone.

This PR makes the patch self-asserting. unix/Makefile gains an idempotent mpy-patch target
that all and tools depend on, so a build re-applies the patch when it has gone missing and
does nothing when it is already there.

Follow-up to #712, which is where the current clang-21 suppressions in macos-mpy.patch came
from. Nothing here changes firmware behaviour on device: the diff is one make target plus the
README lines that describe it. stm32/ is untouched.

Environment where this was reproduced and fixed

OS macOS 26.5.2 (Darwin 25.5.0), Apple Silicon
Compiler Apple clang 21.0.0
Make GNU Make 3.81 (the stock macOS one)
Git 2.50.1
Python 3.13.5

The failure

The patch exists only as an uncommitted modification to two files in the submodule:

$ git -C external/micropython status --short
 M mpy-cross/Makefile
 M ports/unix/Makefile

git submodule update, a git checkout in that tree, git submodule foreach git checkout ., or
re-syncing submodules after a branch change in the parent all revert it without comment. From
there cd unix && make setup dies like this:

CC ../../py/objfun.c
../../py/objfun.c:147:5: error: variable 'n_info' set but not used [-Werror,-Wunused-but-set-variable]
  147 |     MP_BC_PRELUDE_SIZE_DECODE(code_info);
      |     ^
../../py/bc.h:165:12: note: expanded from macro 'MP_BC_PRELUDE_SIZE_DECODE'
  165 |     size_t n_info, n_cell; \
      |            ^

12 errors in that one file, from variables declared by MP_BC_PRELUDE_SIZE_DECODE,
MP_BC_PRELUDE_SIG_DECODE and DECODE_CODESTATE_SIZE — macros that deliberately decode every
prelude field while each caller reads only some. That is exactly the benign-by-design class #712
catalogued and suppressed; the suppression just isn't there any more.

What makes this cost real time is that the evidence points the wrong way. The errors name
py/objfun.c and py/bc.h, which are upstream files nobody edited and which are not broken. The
build has already emitted a few hundred lines of CC ... by that point, so the failure looks like
it belongs to the code being compiled rather than to a setup step that silently came undone.

What the fix does

# Which micropython patch this host needs, if any. Override to use another, ie.
# on Ubuntu 24.04:   make mpy-patch MPY_PATCH=ubuntu24_mpy.patch
MPY_PATCH_Darwin = macos-mpy.patch
MPY_PATCH = $(MPY_PATCH_$(shell uname -s))

mpy-patch reverse-checks before applying, so it is a no-op when the patch is already in place
and it never double-applies:

mpy-patch:
ifneq ($(MPY_PATCH),)
	@cd $(MPY_TOP) && \
	  if git apply --reverse --check $(CC_TOP)/$(MPY_PATCH) 2>/dev/null; then \
	    true; \
	  ...

all and tools now depend on it. setup picks it up through its existing $(MAKE) tools,
which matters for ordering: setup runs git submodule update --init inside ports/unix first,
and the patch is re-asserted after that, not before.

Three things worth a look during review:

  • It fails closed. If neither direction applies — someone has their own edits in that tree, or
    a half-applied patch — it does not guess. It prints what it could not do and stops with a
    non-zero status, instead of proceeding into the confusing compile failure above.
  • Linux behaviour is deliberately unchanged. MPY_PATCH is only defaulted for Darwin, so on
    Linux the target is an empty recipe and nothing is auto-applied. Ubuntu 24.04 users get the same
    single manual step they have now, just idempotent, via the documented
    make -C unix mpy-patch MPY_PATCH=ubuntu24_mpy.patch. I have no Ubuntu 24.04 host to test
    auto-application on, and the README notes the patch is only wanted on 24.04, so guessing the
    distro from the Makefile seemed worse than leaving it explicit. Adding
    MPY_PATCH_Linux = ubuntu24_mpy.patch is the one-line change if you would rather it be automatic.
  • make setup still re-applies on macOS even if you never invoke mpy-patch yourself, so an
    existing clone that has lost the patch heals on the next build with no new instructions.

The README changes just replace the manual pushd/git apply/popd block with the target. The
macOS ordering is unchanged and still correct: the patch touches mpy-cross/Makefile too, so it
has to land before make -C external/micropython/mpy-cross.

Testing

All on the environment above.

Idempotency — from a clean submodule, three runs in a row:

$ git -C external/micropython status --short      # (empty: patch is gone)
$ make -C unix mpy-patch
applied macos-mpy.patch
$ make -C unix mpy-patch                          # silent
$ make -C unix mpy-patch                          # silent
$ git -C external/micropython status --short
 M mpy-cross/Makefile
 M ports/unix/Makefile

Fails closed on an unexpected tree — reverted the patch, then deleted the CWARN line it
anchors on:

$ make -C unix mpy-patch
error: patch failed: ports/unix/Makefile:40
error: ports/unix/Makefile: patch does not apply
ERROR: cannot apply macos-mpy.patch; micropython tree is not in a state I understand.
       'cd ../external/micropython && git checkout .' then retry, if you have no work there.
make: *** [mpy-patch] Error 1

Self-heal — with everything already built, reverted the patch and ran a plain make:

$ git -C external/micropython checkout -- ports/unix/Makefile mpy-cross/Makefile
$ git -C external/micropython status --short      # (empty: patch is gone)
$ make -C unix
applied macos-mpy.patch
cd ../external/micropython/ports/unix && make -j 4 VARIANT=coldcard-mpy ... DEBUG=1
Including User C Module from .../external/c-modules/aes256ctr
...
$ git -C external/micropython status --short
 M mpy-cross/Makefile
 M ports/unix/Makefile

And the non-Darwin path, where MPY_PATCH is empty, is a clean no-op rather than an error:

$ make -C unix mpy-patch MPY_PATCH=
make: Nothing to be done for `mpy-patch'.
$ echo $?
0

Full build from a clean, unpatched submodule, following the README as changed here:

$ git -C external/micropython status --short      # (empty: fresh submodule checkout)
$ make -C unix mpy-patch
applied macos-mpy.patch
$ make -C external/micropython/mpy-cross
...
LINK mpy-cross
$ cd unix && make setup && make ngu-setup && make
...
LINK coldcard-mpy

All five steps completed; error: appears 0 times in the full build log. The resulting
coldcard-mpy is a working arm64 binary, and the vanilla unix/micropython that make setup
produces still runs:

$ ./micropython -c 'import sys; print(sys.version, sys.implementation)'
3.4.0 (name='micropython', version=(1, 13, 0), mpy=773)

Notes

  • unix/Makefile has no .PHONY declarations today and I did not add one, to keep the diff to
    the change itself. There is no file named mpy-patch, so the target behaves.
  • Verified against the stock macOS GNU Make 3.81, not just a newer brew make — the ifneq
    around the recipe and $(shell uname -s) both work there.
  • testing/ was not run as a suite, same reason as macOS fixes pip install and unix simulator build on current toolchains #712: python-secp256k1 wants a locally
    compiled native libsecp256k1, which is out of scope here.

…t break the build

macos-mpy.patch and ubuntu24_mpy.patch are applied by hand into the
external/micropython working tree and never committed anywhere, so any
`git submodule update`, checkout, or clean of that tree silently reverts
them.

The next build then fails hundreds of lines into the compile with -Werror
diagnostics against py/objfun.c and py/bc.h -- upstream files nobody
touched and which are not broken -- and nothing in the output mentions the
patch or the README step that came undone.

Add an idempotent `mpy-patch` target that reverse-checks before applying,
and make `all` and `tools` depend on it, so a build re-applies the patch
when it has gone missing and does nothing when it is already there.
`setup` inherits it through its existing `$(MAKE) tools`, which keeps the
ordering right: its `git submodule update --init` runs first, and the
patch is re-asserted afterwards.

If neither direction applies -- local edits, or a half-applied patch --
it stops with a clear message rather than guessing.

MPY_PATCH is only defaulted on Darwin, so Linux behaviour is unchanged;
Ubuntu 24.04 keeps its single documented step, now idempotent, via
`make -C unix mpy-patch MPY_PATCH=ubuntu24_mpy.patch`.

Follow-up to Coldcard#712.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@scgbckbone scgbckbone left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to remove those patch files for long time... what do you think about #731 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants