Skip to content

Ship prebuilt abi3 R-tree wheels so users don't need a C compiler - #57

Merged
tlambert03 merged 15 commits into
funkelab:mainfrom
tlambert03:prebuilt-abi3-proto
Aug 28, 2026
Merged

Ship prebuilt abi3 R-tree wheels so users don't need a C compiler#57
tlambert03 merged 15 commits into
funkelab:mainfrom
tlambert03:prebuilt-abi3-proto

Conversation

@tlambert03

@tlambert03 tlambert03 commented Jul 29, 2026

Copy link
Copy Markdown
Member

Closes #56 for the R-tree half.

spatial_graph JIT-compiles its extensions at runtime via witty, so every end
user needs a working C compiler. That's what's been breaking Windows users of
motile_tracker (via tracksdata).

The graph half genuinely can't be prebuilt — node/edge attribute dtypes are only
known when you construct the graph. The R-tree is different: its compile
parameters are just an item dtype, a coordinate dtype, the dimensionality, and
the tree class. That's a small, enumerable set, so those variants are now
compiled ahead of time and shipped in the wheels. witty remains as the
fallback for anything not prebuilt, and for the graph.

This does it inside spatial_graph rather than extracting a standalone
rtree-nd package, since the split turned out not to be necessary for the goal.

What ships

Wheels are cp311-abi3 — one per platform, covering every supported
CPython, verified end to end (a wheel built on 3.11 is installed and tested on
3.12/3.13/3.14 in CI). 8 wheels per release: manylinux + musllinux × x86_64 /
aarch64, win_amd64 + win32, macOS x86_64 + arm64. ~2.7 MB each, 32 prebuilt
modules:

classes PointRTree, LineRTree
item dtypes int64, uint64 (int64[2] / uint64[2] for LineRTree)
coord dtypes float32, float64
dimensions 2, 3, 4, 5

tracksdata only ever constructs PointRTree(item_dtype="int64", coord_dtype="float32", dims=N) with N of 2–4, so it is fully covered.

How it works

RTree.__init__ looks up a prebuilt module first and falls back to witty if
there isn't one:

  • _rtree/_codegen.py — the variant matrix and the pyx renderer. setup.py and
    the JIT path both call build_wrapper(), so the two can't drift.
  • _rtree/_naming.py — deterministic module names, e.g.
    rtree_int64_float_d3_df3f6198. Dtypes are canonicalised through
    DType.base_c_type (so int/int64 and float32/float collapse to one
    module) and the digest covers the C/pyx a subclass injects — so an RTree
    subclass with custom code can never be served a module built from different
    code, and falls through to JIT instead.
  • _rtree/_prebuilt/ — populated at build time; empty in a source checkout.

The compiled modules have no numpy C-API dependency (the wrappers use plain
buffer-protocol memoryviews), so the wheels need no numpy ABI pinning.

Why 3.11, and why 3.10 had to go

Py_buffer and the PyBUF_* flags only entered the limited API in 3.11
before that they live in cpython/object.h, which is excluded under
Py_LIMITED_API. Cython's typed memoryviews compile straight down to
PyObject_GetBuffer/PyBuffer_Release, so 3.11 is a hard floor for a
stable-ABI build.

This is the one user-visible consequence: requires-python is now >=3.11,
so 3.10 users will stay on the last release. (3.10 reaches EOL in October.)
Everything else should be invisible — same API, same behaviour, witty and
CT3 are still unconditional dependencies. What changes is that they no longer
get invoked for the common cases, which is what actually required a compiler.

Other changes

  • Build backend hatchling → setuptools. Hatchling has no first-party
    C-extension support, so this otherwise meant hand-driving build_ext from a
    custom hook. hatch-vcssetuptools-scm is like-for-like.
  • cibuildwheel for release wheels. uv build alone emits linux_x86_64,
    which PyPI rejects, so the previous deploy path could not have shipped
    compiled wheels at all.
  • CI tests the built wheel, not an editable install (uv sync --no-editable), and SPATIAL_GRAPH_REQUIRE_PREBUILT=1 makes a build that
    silently degrades to pure Python fail instead of going green.
  • import spatial_graph no longer pulls in the compilation toolchain: ~35 ms,
    down from ~88 ms
    .
  • Sdist installs still work without a compiler — setup.py probes for one and
    falls back to a pure-Python wheel that JIT-compiles on first use, exactly as
    before.

Testing

tests/test_prebuilt.py blocks witty.compile_cython and constructs all 32
declared variants, asserting the real property directly: prebuilt variants
must not invoke the compiler
. That runs on every OS and Python in the matrix.
Plus: dtype aliases share a module, unlisted combinations fall back to JIT, and
subclasses with custom C aren't served a prebuilt module.

Not included (deliberate)

  • int32/uint32 item dtypes — those still JIT.
  • The cross-version abi3 check runs against the Linux artifact only; other
    platforms' wheels are tested on 3.11 by cibuildwheel.
  • No [jit] extra — witty/CT3 stay required, they're just not invoked.

Follow-up

The conda-forge feedstock will keep working via the JIT path, but won't deliver
the benefit until it's converted from noarch to per-platform builds.

@codecov-commenter

codecov-commenter commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.71%. Comparing base (caab135) to head (06df3db).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #57      +/-   ##
==========================================
+ Coverage   95.11%   95.71%   +0.59%     
==========================================
  Files          12       14       +2     
  Lines         389      443      +54     
==========================================
+ Hits          370      424      +54     
  Misses         19       19              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed-hq

codspeed-hq Bot commented Jul 29, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 29.94%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 5 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
test_roi_query_performance[100000] 1.9 ms 1.4 ms +29.94%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing tlambert03:prebuilt-abi3-proto (06df3db) with main (e655613)

Open in CodSpeed

tlambert03 and others added 13 commits July 29, 2026 15:33
…piler probe, README

- guard setup() so cythonize's worker processes don't re-run the build
- declare rtree.c/h + config.h as Extension depends, so edits to the vendored
  C core actually rebuild the prebuilt modules instead of silently going stale
- probe with #include <Python.h> (via get_python_inc) so a box with cc but no
  dev headers is caught before the real build, and guard the distutils import
- package-data patterns needed a src/ prefix to match the vendored C at all
- rewrite README's Cross-Platform Support section; it still told users they
  needed a compiler
- reword comments that implied witty/Cheetah were optional: they remain
  install-time deps, they're just never invoked on the prebuilt path
…r __init__

- license = "MIT" + license-files, drop the deprecated classifier
- skip rendering/cythonizing 32 variants for metadata-only commands, matched on
  recognized command names so 'sdist bdist_wheel' still builds
- SPATIAL_GRAPH_NO_PREBUILT=0 disabled prebuilding, since any non-empty string
  is truthy; both readers now share env_flag() so they cannot diverge
- the lazy __init__ broke the API docs: griffe inspects __dict__, so the seven
  deferred names vanished (527 -> 414 documented symbols). Import witty and
  Cheetah lazily inside _graph.graph_base instead, which keeps the 35ms import
  and restores full docs
Two follow-ups from review.

setup.py imported `spatial_graph._rtree._naming` at module scope (after
inserting src/ on sys.path), which pulled the whole package -- and with
the eager __init__, the graph half too -- into every setuptools command.
A single module-scope third-party import anywhere in the package would
then break the entire build system, `sdist` and `egg_info` included,
with a traceback pointing at the package rather than at setup.py.
Verified: adding `import witty` to graph_base.py made `build --sdist`
fail at get_requires_for_build_sdist.

The import now happens only where the package is genuinely needed, via
`_src_on_path()`: in `prebuilt_extensions()`, and in `should_prebuild()`
after the `metadata_only()` early return. Metadata-only commands no
longer import the package at all -- `spatial_graph` stays out of
sys.modules -- while `env_enabled` remains shared with the runtime, so
the build and the lookup still cannot read the same variable
differently.

tests/test_prebuilt.py was the one consumer not converted to
`env_enabled`, so SPATIAL_GRAPH_REQUIRE_PREBUILT=0 would skip
prebuilding at build time yet still assert it at test time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he probe

- can_compile()'s comment justified get_python_inc with a venv/sysconfig
  mechanism that does not exist; replaced with the real reason, and the probe
  now also passes the platform-specific include dir as build_ext does, since
  that is where pyconfig.h can live and a false negative there silently emits
  a pure-Python wheel
- _naming.py claimed installs may have neither Cheetah nor witty; both are
  unconditional dependencies, and it contradicted _codegen.py
- README implied LineRTree is prebuilt for int64/uint64 items; its items are
  node pairs, so the prebuilt set is int64[2]/uint64[2]
…-less install

Blocking witty.compile_cython states the actual property -- prebuilt variants
must not invoke the compiler -- in the environment users really get, since
witty is an unconditional dependency and will always be installed. The old CI
step instead simulated its absence, which can never happen, and needed
--no-deps/--no-index gymnastics plus a second install to undo them.

The check now lives in the test suite, so it runs on every OS and Python in
the test matrix rather than only in one Linux job, and it covers all 32
variants. test-abi3-wheel keeps its remaining job: proving one cp311 wheel we
actually publish runs on 3.11-3.14.
Installing by package name would have resolved spatial-graph from PyPI: our
version is a dev release and resolvers exclude pre-releases by default, so
'uv pip install --find-links wheelhouse spatial-graph' silently installs the
last real release (verified: it picked 0.0.7). Naming the manylinux wheel is
both simpler than pinning the parsed version and immune to that.

deploy has never executed, and only ever will on a tag. Give it a checkout
rather than assume action-gh-release works without one, and name build-wheels
in needs instead of relying on test-abi3-wheel to pull it in.
@tlambert03 tlambert03 changed the title Prebuilt rtree abi3 prototype Ship prebuilt abi3 R-tree wheels so users don't need a C compiler Aug 28, 2026
@tlambert03
tlambert03 merged commit 7d289b0 into funkelab:main Aug 28, 2026
29 checks passed
@tlambert03
tlambert03 deleted the prebuilt-abi3-proto branch August 28, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extract the R-tree into a standalone, prebuilt-wheel package

2 participants