feat(bench): add reproducible serving performance harness

[tuxevil]

Summary

• Add benchmarks/bench_serving.py, a reproducible client-wall performance harness for an already-running FreeToken server.
• Calibrate prompt sizes through /v1/messages/count_tokens.
• Measure streamed /v1/chat/completions TTFT, decode throughput, effective prefill throughput, and total latency.
• Capture model, server, hardware, KV/MoE/Mamba cache geometry, MoE residency, and cache budget.
• Benchmark fresh, identical-prefix, and small-suffix prefix-cache behavior.
• Add CPU-only unit tests and usage documentation.

Motivation

FreeToken currently has serving benchmarks, but no small, generic harness that clearly separates client-observed performance from engine-internal telemetry.

This harness records measurements at the HTTP client boundary. Fields prefixed with client_wall_ include queueing, prefill, scheduling, detokenization, and SSE delivery.

Design

The harness:

1. Validates that the server is ready.
2. Collects metadata from /health, /v1/models, /v1/cache/status, and /v1/stats.
3. Performs configurable warmup runs.
4. Calibrates deterministic prompts to approximately 512, 1024, 2048, and 4096 tokens.
5. Measures fresh prefill performance.
6. Measures steady-state single-stream decode performance.
7. Measures prefix-cache reuse through an ordered fresh → identical → small_suffix sequence.
8. Writes machine-readable schema version 1 JSON and prints a human-readable summary.

The benchmark uses explicit greedy sampling with thinking disabled so runtime measurements are not affected by reasoning nondeterminism.

Decode throughput follows the existing serving benchmark convention:

(completion_tokens - 1) / (last_token_time - first_token_time)

Scope

This first version intentionally:

• connects to an existing server;
• measures one serving configuration at a time;
• uses standard-library HTTP/SSE handling;
• does not add new server endpoints;
• does not start or stop the server;
• does not run concurrency or backend sweeps;
• does not parse engine logs;
• does not perform behavioral evaluation.

These capabilities can build on the schema in later PRs.

Validation

• PYTHONPATH=python:. uv run --no-project --with pytest pytest -q tests/benchmarks/test_bench_serving.py

  • 13 tests passed

• uv run --no-project --with ruff ruff check benchmarks/bench_serving.py tests/benchmarks/test_bench_serving.py

• uv run --no-project --with ruff ruff format --check benchmarks/bench_serving.py tests/benchmarks/test_bench_serving.py

• python -m py_compile benchmarks/bench_serving.py

• git diff --check

The full repository test suite was not collected in this environment because FreeToken's runtime dependencies, including PyTorch, are not installed.