Ferrox is a stateless, horizontally-scalable LLM API gateway written in Rust. It exposes an OpenAI-compatible API and routes requests to Anthropic, OpenAI, Gemini, and AWS Bedrock.
- OpenAI-compatible API - drop-in replacement; no client-side changes needed
- Multi-provider routing - round-robin, weighted, failover, and random strategies
- Fallback chains - automatic failover to backup providers on failure
- Circuit breakers - lock-free, per-provider; prevents cascading failures
- Rate limiting - per-key token bucket; burst-aware
- Virtual keys - issue scoped API keys with per-model access control
- Streaming - full SSE pass-through for all providers
- Observability - Prometheus metrics, structured JSON logs, OpenTelemetry tracing
flowchart LR
Client -->|Bearer token| Auth
Auth -->|RequestContext| Router
subgraph Ferrox
Auth[Auth Middleware]
Router[Model Router]
CB[Circuit Breaker]
RateLimit[Rate Limiter]
Auth --> RateLimit
Router --> CB
end
CB -->|primary| Anthropic
CB -->|primary| OpenAI
CB -->|primary| Gemini
CB -->|fallback| Bedrock[AWS Bedrock]
Ferrox --> Prometheus
Ferrox --> OTLP[OTLP Collector]
Download the latest release for your platform from GitHub Releases:
# macOS (Apple Silicon)
curl -L https://github.com/shaharia-lab/ferrox/releases/latest/download/ferrox-aarch64-apple-darwin.tar.gz | tar xz
sudo mv ferrox /usr/local/bin/
# macOS (Intel)
curl -L https://github.com/shaharia-lab/ferrox/releases/latest/download/ferrox-x86_64-apple-darwin.tar.gz | tar xz
sudo mv ferrox /usr/local/bin/
# Linux (x86_64)
curl -L https://github.com/shaharia-lab/ferrox/releases/latest/download/ferrox-x86_64-unknown-linux-gnu.tar.gz | tar xz
sudo mv ferrox /usr/local/bin/
# Linux (ARM64)
curl -L https://github.com/shaharia-lab/ferrox/releases/latest/download/ferrox-aarch64-unknown-linux-gnu.tar.gz | tar xz
sudo mv ferrox /usr/local/bin/brew install shaharia-lab/tap/ferroxTo install a specific version:
brew install shaharia-lab/tap/ferrox@1.0.0docker pull ghcr.io/shaharia-lab/ferrox:latestOr with Docker Compose (includes full LGTM observability stack and control plane):
docker compose up| Service | URL |
|---|---|
| Ferrox gateway | http://localhost:8080 |
| Control plane admin UI | http://localhost:9090 |
| Grafana dashboards | http://localhost:3000 (admin / admin) |
# Prerequisites: Rust 1.74+, protobuf-compiler
# Ubuntu/Debian: sudo apt install protobuf-compiler
# macOS: brew install protobuf
git clone https://github.com/shaharia-lab/ferrox
cd ferrox
cargo build --release
# Binary at: ./target/release/ferrox# 1. Copy the minimal config (pre-configured with sensible defaults)
cp ferrox/config/config_minimal.yaml config/local.yaml
# 2. Set your API keys
export ANTHROPIC_API_KEY=sk-ant-...
export OPENAI_API_KEY=sk-...
# 3. Run
LLM_PROXY_CONFIG=config/local.yaml ferrox# 1. Download the minimal config
curl -Lo local.yaml https://raw.githubusercontent.com/shaharia-lab/ferrox/main/ferrox/config/config_minimal.yaml
# 2. Run (API keys passed as env vars)
docker run -p 8080:8080 \
-e ANTHROPIC_API_KEY=sk-ant-... \
-v $(pwd)/local.yaml:/etc/ferrox/config.yaml \
ghcr.io/shaharia-lab/ferrox:latest \
--config /etc/ferrox/config.yamlSend a request:
curl http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer sk-local-dev" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet",
"messages": [{"role": "user", "content": "Hello"}]
}'| Guide | Description |
|---|---|
| Quick Start | Get running in 5 minutes |
| Configuration | Full config reference |
| Providers | Anthropic, OpenAI, Gemini, Bedrock setup |
| GLM & Kimi in Claude Code | Use Z.AI / Moonshot coding plans from Claude Code |
| Routing | Strategies, failover, circuit breakers |
| Virtual Keys | Auth, rate limits, model access |
| API Reference | Endpoints and request/response formats |
| Observability | Metrics, tracing, logging |
| Guide | Description |
|---|---|
| Architecture | System design and request flow |
| Development | Build, test, contribute |
| Deployment | Docker, control plane, admin UI |
MIT. See LICENSE.