Ship prebuilt abi3 R-tree wheels so users don't need a C compiler - #57
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #57 +/- ##
==========================================
+ Coverage 95.11% 95.71% +0.59%
==========================================
Files 12 14 +2
Lines 389 443 +54
==========================================
+ Hits 370 424 +54
Misses 19 19 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will improve performance by 29.94%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | test_roi_query_performance[100000] |
1.9 ms | 1.4 ms | +29.94% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing tlambert03:prebuilt-abi3-proto (06df3db) with main (e655613)
…piler probe, README - guard setup() so cythonize's worker processes don't re-run the build - declare rtree.c/h + config.h as Extension depends, so edits to the vendored C core actually rebuild the prebuilt modules instead of silently going stale - probe with #include <Python.h> (via get_python_inc) so a box with cc but no dev headers is caught before the real build, and guard the distutils import - package-data patterns needed a src/ prefix to match the vendored C at all - rewrite README's Cross-Platform Support section; it still told users they needed a compiler - reword comments that implied witty/Cheetah were optional: they remain install-time deps, they're just never invoked on the prebuilt path
…r __init__ - license = "MIT" + license-files, drop the deprecated classifier - skip rendering/cythonizing 32 variants for metadata-only commands, matched on recognized command names so 'sdist bdist_wheel' still builds - SPATIAL_GRAPH_NO_PREBUILT=0 disabled prebuilding, since any non-empty string is truthy; both readers now share env_flag() so they cannot diverge - the lazy __init__ broke the API docs: griffe inspects __dict__, so the seven deferred names vanished (527 -> 414 documented symbols). Import witty and Cheetah lazily inside _graph.graph_base instead, which keeps the 35ms import and restores full docs
Two follow-ups from review. setup.py imported `spatial_graph._rtree._naming` at module scope (after inserting src/ on sys.path), which pulled the whole package -- and with the eager __init__, the graph half too -- into every setuptools command. A single module-scope third-party import anywhere in the package would then break the entire build system, `sdist` and `egg_info` included, with a traceback pointing at the package rather than at setup.py. Verified: adding `import witty` to graph_base.py made `build --sdist` fail at get_requires_for_build_sdist. The import now happens only where the package is genuinely needed, via `_src_on_path()`: in `prebuilt_extensions()`, and in `should_prebuild()` after the `metadata_only()` early return. Metadata-only commands no longer import the package at all -- `spatial_graph` stays out of sys.modules -- while `env_enabled` remains shared with the runtime, so the build and the lookup still cannot read the same variable differently. tests/test_prebuilt.py was the one consumer not converted to `env_enabled`, so SPATIAL_GRAPH_REQUIRE_PREBUILT=0 would skip prebuilding at build time yet still assert it at test time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he probe - can_compile()'s comment justified get_python_inc with a venv/sysconfig mechanism that does not exist; replaced with the real reason, and the probe now also passes the platform-specific include dir as build_ext does, since that is where pyconfig.h can live and a false negative there silently emits a pure-Python wheel - _naming.py claimed installs may have neither Cheetah nor witty; both are unconditional dependencies, and it contradicted _codegen.py - README implied LineRTree is prebuilt for int64/uint64 items; its items are node pairs, so the prebuilt set is int64[2]/uint64[2]
…-less install Blocking witty.compile_cython states the actual property -- prebuilt variants must not invoke the compiler -- in the environment users really get, since witty is an unconditional dependency and will always be installed. The old CI step instead simulated its absence, which can never happen, and needed --no-deps/--no-index gymnastics plus a second install to undo them. The check now lives in the test suite, so it runs on every OS and Python in the test matrix rather than only in one Linux job, and it covers all 32 variants. test-abi3-wheel keeps its remaining job: proving one cp311 wheel we actually publish runs on 3.11-3.14.
Installing by package name would have resolved spatial-graph from PyPI: our version is a dev release and resolvers exclude pre-releases by default, so 'uv pip install --find-links wheelhouse spatial-graph' silently installs the last real release (verified: it picked 0.0.7). Naming the manylinux wheel is both simpler than pinning the parsed version and immune to that. deploy has never executed, and only ever will on a tag. Give it a checkout rather than assume action-gh-release works without one, and name build-wheels in needs instead of relying on test-abi3-wheel to pull it in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #56 for the R-tree half.
spatial_graphJIT-compiles its extensions at runtime viawitty, so every enduser needs a working C compiler. That's what's been breaking Windows users of
motile_tracker(viatracksdata).The graph half genuinely can't be prebuilt — node/edge attribute dtypes are only
known when you construct the graph. The R-tree is different: its compile
parameters are just an item dtype, a coordinate dtype, the dimensionality, and
the tree class. That's a small, enumerable set, so those variants are now
compiled ahead of time and shipped in the wheels.
wittyremains as thefallback for anything not prebuilt, and for the graph.
This does it inside
spatial_graphrather than extracting a standalonertree-ndpackage, since the split turned out not to be necessary for the goal.What ships
Wheels are
cp311-abi3— one per platform, covering every supportedCPython, verified end to end (a wheel built on 3.11 is installed and tested on
3.12/3.13/3.14 in CI). 8 wheels per release: manylinux + musllinux × x86_64 /
aarch64, win_amd64 + win32, macOS x86_64 + arm64. ~2.7 MB each, 32 prebuilt
modules:
PointRTree,LineRTreeint64,uint64(int64[2]/uint64[2]forLineRTree)float32,float64tracksdataonly ever constructsPointRTree(item_dtype="int64", coord_dtype="float32", dims=N)with N of 2–4, so it is fully covered.How it works
RTree.__init__looks up a prebuilt module first and falls back towittyifthere isn't one:
_rtree/_codegen.py— the variant matrix and the pyx renderer.setup.pyandthe JIT path both call
build_wrapper(), so the two can't drift._rtree/_naming.py— deterministic module names, e.g.rtree_int64_float_d3_df3f6198. Dtypes are canonicalised throughDType.base_c_type(soint/int64andfloat32/floatcollapse to onemodule) and the digest covers the C/pyx a subclass injects — so an
RTreesubclass with custom code can never be served a module built from different
code, and falls through to JIT instead.
_rtree/_prebuilt/— populated at build time; empty in a source checkout.The compiled modules have no numpy C-API dependency (the wrappers use plain
buffer-protocol memoryviews), so the wheels need no numpy ABI pinning.
Why 3.11, and why 3.10 had to go
Py_bufferand thePyBUF_*flags only entered the limited API in 3.11 —before that they live in
cpython/object.h, which is excluded underPy_LIMITED_API. Cython's typed memoryviews compile straight down toPyObject_GetBuffer/PyBuffer_Release, so 3.11 is a hard floor for astable-ABI build.
This is the one user-visible consequence:
requires-pythonis now>=3.11,so 3.10 users will stay on the last release. (3.10 reaches EOL in October.)
Everything else should be invisible — same API, same behaviour,
wittyandCT3are still unconditional dependencies. What changes is that they no longerget invoked for the common cases, which is what actually required a compiler.
Other changes
C-extension support, so this otherwise meant hand-driving
build_extfrom acustom hook.
hatch-vcs→setuptools-scmis like-for-like.uv buildalone emitslinux_x86_64,which PyPI rejects, so the previous deploy path could not have shipped
compiled wheels at all.
uv sync --no-editable), andSPATIAL_GRAPH_REQUIRE_PREBUILT=1makes a build thatsilently degrades to pure Python fail instead of going green.
import spatial_graphno longer pulls in the compilation toolchain: ~35 ms,down from ~88 ms.
setup.pyprobes for one andfalls back to a pure-Python wheel that JIT-compiles on first use, exactly as
before.
Testing
tests/test_prebuilt.pyblockswitty.compile_cythonand constructs all 32declared variants, asserting the real property directly: prebuilt variants
must not invoke the compiler. That runs on every OS and Python in the matrix.
Plus: dtype aliases share a module, unlisted combinations fall back to JIT, and
subclasses with custom C aren't served a prebuilt module.
Not included (deliberate)
int32/uint32item dtypes — those still JIT.platforms' wheels are tested on 3.11 by cibuildwheel.
[jit]extra —witty/CT3stay required, they're just not invoked.Follow-up
The conda-forge feedstock will keep working via the JIT path, but won't deliver
the benefit until it's converted from
noarchto per-platform builds.