moe-stream

Memory-efficient inference for small mixture-of-experts models on consumer hardware.

Features

• LRU expert cache
• Memory-mapped weight loading
• Top-2 routing
• Aux-loss-free router balancing
• Streaming from NVMe
• CPU and GPU support

Stack

• PyTorch
• NumPy

Quick start

pip install -e .
python -m moestream.inference --model ./checkpoints/model
python -m moestream.benchmark --model ./checkpoints/model

Project structure

src/
└── moestream/
    ├── __init__.py
    └── cache/
tests/
├── __init__.py
└── test_smoke.py

License

MIT