Skip to content

feat(linked): support TVM FFI providers - #945

Merged
voltjia merged 1 commit into
masterfrom
feat/linked-tvm-ffi-transport
Aug 24, 2026
Merged

feat(linked): support TVM FFI providers#945
voltjia merged 1 commit into
masterfrom
feat/linked-tvm-ffi-transport

Conversation

@voltjia

@voltjia voltjia commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add a tvm_ffi linked-operator transport with .cu adapters, PEP 440 package-version checks, installed include discovery, and explicit DSO dependencies.
  • Partition Torch and TVM-FFI adapter sources in the generated CMake manifest, and retain declared dependency DSOs with a scoped --no-as-needed.
  • Document the transport contract and cover version, source selection, include discovery, dependency validation, and manifest generation.

Motivation

This is the transport-only prerequisite for reducing #930 to a FlashInfer provider PR. It intentionally contains no FlashInfer provider implementation, sampling operator changes, or third-party binaries.

Type of Change

  • feat - new feature / new operator / new platform
  • fix - bug fix
  • perf - performance improvement (no behavioral change)
  • refactor - code restructuring without behavior change
  • test - adding or fixing tests only
  • docs - documentation only
  • build / ci - build system or CI configuration
  • chore - tooling, formatting, or other non-code changes
  • Breaking change (requires a ! in the Conventional Commits prefix or a BREAKING CHANGE: footer)

Platforms Affected

  • CPU (WITH_CPU)
  • NVIDIA (WITH_NVIDIA)
  • Iluvatar (WITH_ILUVATAR)
  • MetaX (WITH_METAX)
  • Cambricon (WITH_CAMBRICON)
  • Moore (WITH_MOORE)
  • Ascend (WITH_ASCEND)
  • PyTorch C++ bindings (WITH_TORCH)
  • Build system / CMake / CI
  • Python bindings / user-facing API

Smoke Test Result

The real integration check overlaid the follow-up FlashInfer provider files only in an isolated NVIDIA validation tree. Those files are not present in this PR.

python3 -m pytest tests/test_resolve_linked_ops.py -q
Running 37 items in this shard
..................................... [100%]
37 passed in 0.94s

python3 -m ruff check scripts/resolve_linked_ops.py tests/test_resolve_linked_ops.py
All checks passed!

cmake -S /src -B /work/cmake-build-linked   -DCMAKE_BUILD_TYPE=Release   -DINFINI_RT_ROOT=/opt/infinirt   -DAUTO_DETECT_DEVICES=OFF   -DAUTO_DETECT_BACKENDS=OFF   -DWITH_CPU=OFF   -DWITH_NVIDIA=ON   -DWITH_LINKED=ON   -DWITH_TORCH=OFF   -DGENERATE_PYTHON_BINDINGS=OFF   -DINFINI_OPS_OPS=/work/ops.json   -DCMAKE_CUDA_ARCHITECTURES=80
-- Operator selection: /work/ops.json
-- Resolved 1 linked operator source(s).
-- Configuring done
-- Generating done

cmake --build /work/cmake-build-linked --target infiniops --parallel 8
[100%] Built target infiniops

readelf -d libinfiniops.so
NEEDED: sampling.so
NEEDED: libtvm_ffi.so
RUNPATH: flashinfer sampling directory; tvm_ffi/lib; InfiniRT; CUDA

ldd -r libinfiniops.so
sampling.so => /opt/linked-site/flashinfer_jit_cache/jit_cache/sampling/sampling.so
libtvm_ffi.so => /opt/linked-site/tvm_ffi/lib/libtvm_ffi.so
(no missing libraries or unresolved symbols)

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
NVIDIA Yes TVM-FFI integration build passed Resolver tests passed; ELF dependencies and RUNPATH verified
Iluvatar No N/A - not affected Hosted CI pending
MetaX Yes Hosted CI pending Existing linked Torch source partition is shared
Cambricon Yes Hosted CI pending Existing linked Torch source partition is shared
Moore Yes Hosted CI pending Existing linked Torch source partition is shared
Ascend No N/A - not affected Hosted CI pending

Benchmark / Performance Impact

N/A. This PR changes configure-time resolution and build wiring only.

Notes for Reviewers

  • link_libraries entries are retained independently of manifest sort order, so --as-needed cannot discard a declared dependency.
  • include_glob is applied for any linked library that declares one.
  • WITH_TORCH=OFF disables generated ATen implementations, but the existing WITH_LINKED setup still requires an installed torch package for its shared Python/C++ ABI configuration; the documentation now states this explicitly.
  • External DSOs remain environment-bound dependencies and are not bundled into the wheel.
  • The follow-up to this PR will reduce feat(linked): add FlashInfer sampling provider #930 to the FlashInfer slot-16 provider only.
  • The validation target was infiniops. The default all-target build also reached and built infiniops, then unrelated examples failed against the reused older InfiniRT prefix because its runtime copy API no longer matches current examples.

@voltjia
voltjia force-pushed the feat/linked-tvm-ffi-transport branch from bbcb26f to d3199f9 Compare August 24, 2026 04:03
@voltjia
voltjia marked this pull request as ready for review August 24, 2026 06:12
@voltjia
voltjia requested a review from a team August 24, 2026 06:12
@voltjia
voltjia merged commit caa8987 into master Aug 24, 2026
12 of 20 checks passed
@voltjia
voltjia deleted the feat/linked-tvm-ffi-transport branch August 24, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant