Add the paper's benchmark artifact and PDF

[anorouzii]

The CLINC150 and BANKING77 numbers in the FlowRoute paper live only in the arXiv source bundle. Nothing in this repository backs them, so a reader — or an automated reproducibility check — has nothing to verify against.

What this adds

• experiments/, copied verbatim from the arXiv bundle: run_benchmark.py, pinned requirements.txt, environment-lock.txt, and results/benchmark_results.json + ranking_summary.csv with complete per-seed metrics, selected thresholds, model revision, dataset URLs and checksums.
• paper/FlowRoute_Ali_Norouzi.pdf.
• A Paper experiments section in the README.

All six experiments/ files match the SHA-256 digests in the bundle's SOURCE_SHA256SUMS.txt, so what is committed here is byte-identical to what produced the paper.

Independently verified

Every CLINC150 value in the paper reproduces from a reimplementation written against the paper text, on different hardware (Apple Silicon vs the paper's AMD EPYC 9V74) and a different torch version:

Row	Paper	Reproduced
Dense description only, top-1 / R@5	65.38 / 88.47	65.38 / 88.47
FlowRoute k=10, top-1	85.01 ± 0.45	85.01 ± 0.45
FlowRoute k=10, R@5	98.00 ± 0.13	98.00 ± 0.13
Word/char TF-IDF k=10, top-1	80.71 ± 0.35	80.71 ± 0.35
In-scope coverage	64.01 ± 2.86	64.01 ± 2.86
Selective error	4.60 ± 0.99	4.60 ± 0.99
OOS false-positive rate	2.92 ± 0.63	2.92 ± 0.63

The threshold search independently selected the identical (α, τ, γ) for all five seeds. Separately, all Table 1 and Table 2 cells for both datasets were checked against benchmark_results.json: zero mismatches.

Note recorded in the README

The scoring rule benchmarked in the paper is not the one flowroute.backends.HuggingFaceRetriever implements. The paper embeds each workflow description and each approved example separately and takes the maximum example similarity; the shipped retriever embeds one concatenated capability string per contract via serialize_capability. Measured on the same catalogs, the shipped retriever reaches 83.78 ± 0.75 top-1 at k=10 against the paper's 85.01 ± 0.45 — 2.7 SD below. The README now says so explicitly so the two are not mistaken for each other.

Closing that gap (a scoring path that keeps description and example embeddings separate) is worth doing, but is left to a separate change.

🤖 Generated with Claude Code