tensor-parallel

Here are 17 public repositories matching this topic...

harleyszhang / lite_llama

A light llama-like llm inference framework based on the triton kernel.

python3 attention continues llm llm-inference llama3 continuous-batching flash-attention-3 triton-kernels qwen3-moe qwen3-vl tensor-parallel

Updated Aug 26, 2026
Python

lna-lab / gemma4-12b-vllm-sm120

Star

Reproducible recipe: serve abliterated Gemma-4-12B (gemma4_unified) at 50-118 tok/s on no-NVLink Blackwell (SM120) via vLLM nightly + ModelOpt FP8/NVFP4 + MTP spec-decode.

quantization gemma blackwell fp8 vllm speculative-decoding sm120 nvfp4 abliterated gemma-4 modelopt tensor-parallel

Updated Jun 7, 2026
Python

Reederey87 / dgx-spark-2x-deepseek-v4-flash

Star

Reproducible kit to deploy DeepSeek-V4-Flash-DSpark on a 2× NVIDIA DGX Spark (GB10) cluster: vLLM TP=2 over QSFP 200GbE, NVFP4 KV, DSpark speculative decoding, 1M context, systemd self-heal. Apache-2.0.

cuda rdma nccl vllm llm-inference deepseek gb10 nvfp4 dgx-spark grace-blackwell deepseek-v4-flash tensor-parallel

Updated Aug 24, 2026
Python

idonati / spark-vllm-docker-festr2

Star

Patches + recipe to deploy festr2/MiMo-V2.5-Pro-NVFP4-MXFP8-attn-TP8 on 8-node DGX Spark sm_121 (Ray + vLLM, TP=8). Fixes the fused-qkv loader bug that mis-slotted Q values as K/V on 7 of 8 ranks.

moe ray quantization mimo huggingface vllm gb10 nvfp4 dgx-spark mxfp8 sm121 tensor-parallel

Updated May 19, 2026
Python

chriswagner-ai / intel-arc-b70-vllm-multi-gpu

Star

Field-tested guide: multi-GPU vLLM tensor-parallel (TP=2/TP=4) on Intel Arc Pro B70 (Battlemage BMG-G31, Xe2) on Linux. Driver setup (xe force_probe=e223), bare-metal vLLM + oneAPI 2025.3, the compute-runtime multi-root USM + triton-xpu init_devices fixes, FP8/int4-AutoRound quant, root-cause error reports. AI-agent readable (AGENTS.md).

Updated Jun 13, 2026
Shell

Zhang-Wen-chao / mini-megatron

Star

A ~800-line PyTorch implementation of Megatron-LM's TP + PP + DP + AMP. 1.6-2x faster than Megatron-Core on 125M models.

distributed-systems pytorch distributed-training megatron tensor-parallel pipeline-parallel

Updated Aug 25, 2026
Python

behindther8326 / deepseek-v4-deploy

Star

Deploy DeepSeek-V4-Flash-0731 on dual NVIDIA RTX PRO 6000 Blackwell GPUs with vLLM PR #41834 (jasl fork) and DSpark speculative decoding, achieving ~200-227 tok/s in no-overseas-network environments.

benchmarking inference nvidia rdma b200 mixture-of-experts tensor-parallelism long-context vllm llm-inference deepseek llm-benchmark nvfp4 dgx-spark deepseek-v4 tensor-parallel asus-gx10

Updated Aug 25, 2026

bakunyoav-a11y / gemma4-12b-vllm-sm120

Star

Serve an abliterated Gemma-4-12B at high speeds on Blackwell GPUs without NVLink using vLLM, FP8 quantization, and MTP speculative decoding.

quantization gemma blackwell fp8 vllm speculative-decoding sm120 nvfp4 abliterated modelopt tensor-parallel

Updated Aug 25, 2026
Python

123123213weqw / dual-v100-llama.cpp

Star

Reproducible llama.cpp kernel and runtime optimization lab for dual NVIDIA Tesla V100 GPUs (SM70)

cuda inference llama-cpp speculative-decoding tesla-v100 sm70 tensor-parallel

Updated Aug 18, 2026
Shell

www-ai-rs / gb10-deepseek-v4-flash

Star

Two-node DGX Spark operator tooling and measurement harness for DeepSeek-V4-Flash

benchmarks mixture-of-experts vllm local-llm llm-inference deepseek self-hosted-ai gb10 dgx-spark tensor-parallel

Updated Aug 7, 2026
Python

guyiicn / deepseek-v4-flash-2x-gb10

Star

Serve DeepSeek-V4-Flash-0731 on 2x NVIDIA GB10 / DGX Spark (sm_121) with vLLM TP=2 over 200GbE RoCE — runbook, community survey, tuning, and an L1-L3 benchmark suite.

rdma vllm local-llm llm-inference deepseek gb10 dgx-spark deepseek-v4-flash tensor-parallel

Updated Aug 27, 2026
Shell

Red-Weasel / machx-inference-engine

Star

From-scratch C++/SYCL LLM inference engine for Intel Arc (B-series) — 8+ architectures, tensor-parallel, beats llama.cpp on Arc B70

gpu quantization sycl inference-engine oneapi llm intel-arc gguf battlemage tensor-parallel

Updated Jul 3, 2026
C++

botAGI / dgx-spark-llm-benchmarks

Star

Measured LLM benchmarks for NVIDIA DGX Spark (GB10): DeepSeek-V4-Flash 284B MoE on a TP=2 pair over 200G RoCE — tok/s by profile and concurrency, 1M-token context curve, the MoE backend flag, monitoring traps. Every number links to raw runs.

benchmark nvidia llm vllm local-llm deepseek gb10 llm-benchmark dgx-spark tensor-parallel

Updated Aug 20, 2026

Reederey87 / inkling-small-nvfp4-2x-dgx-spark

Star

Deployment kit: Thinking Machines Inkling-Small-NVFP4 (276B MoE) on 2x NVIDIA DGX Spark — TP=2 over QSFP, vLLM 0.26, FA4 SM120 kernel fixes, full docs + bench harness. C1 21.4 / C8 93.9 tok/s measured.

deployment blackwell thinking-machines vllm llm-inference nvfp4 dgx-spark tensor-parallel

Updated Aug 2, 2026
Python

Git-er-done-son / Maya_llama_server_launcher

Star

Lightweight terminal launcher and auto-optimizer for llm models using llama.cpp with hardware detection, tensor sharding, benchmarks, presets, context tuning, and OpenAI-compatible serving.

cuda openapi tensor multi-gpu multi-gpus multi-gpu-inference tensor-parallelism llm llms llama-cpp local-llm llm-inference llm-tools local-ai llama-cpp-python gguf llm-agents openclaw tensor-parallel

Updated Aug 11, 2026
Python

Reederey87 / glm53-flash-nvfp4-2x-dgx-spark

Star

Run GLM-5.3-Flash-NVFP4 (320B/18B active, NoPE sparse MLA, 131K ctx) across two NVIDIA DGX Sparks at TP=2. Adds the FlashInfer (32,2176) sparse-MLA specialization that TP=2 needs. Tested on real hardware.

self-hosted glm sparse-attention vllm llm-inference flashinfer gb10 nvfp4 dgx-spark tensor-parallel

Updated Aug 27, 2026
Python

teilomillet / kvfit

Star

Will a Hugging Face LLM fit your GPU or DGX? Plan weights, KV cache or recurrent state, context, TP/DP, and measured calibration

python cli capacity-planning gpu-memory model-serving huggingface kv-cache llm long-context vllm sglang gb10 vram-calculator dgx-spark tensor-parallel inference-memory

Updated Jul 22, 2026
Python

Improve this page

Add a description, image, and links to the tensor-parallel topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the tensor-parallel topic, visit your repo's landing page and select "manage topics."

Learn more