Context
Competitive analysis of LiteLLM, agentgateway, Portkey, Envoy AI GW, Kong AI GW and Bifrost shows the recurring, most-damaging failure class is protocol infidelity: HTTP 200 + valid streaming + visible text, but the semantics of the agent turn are corrupted — dropped tool_use arguments, lost reasoning/thinking, stripped cache_control, dropped images, mangled SSE ordering. The gateway looks up while the agent is broken.
This is a moving target: this cycle alone we shipped fixes for exactly this class — images dropped on the Responses path (#476), reasoning items not replayed (#465), tool-call salvage (tool_salvage.rs). Every provider protocol change can reintroduce one.
What we already have
src/features/harness/ — record & replay ("sandwich") driver + mock backend.tests/enterprise/scenario_*.rs — failover, budget, DLP, routing scenarios.- cucumber E2E.
Gap
No published, per-release conformance matrix asserting semantic fidelity, only HTTP success. The analysis frames grob's credible claim as N certified routes × M agent clients × published conformance tests rather than "100+ providers".
Proposal
- A
docs/reference/conformance.md matrix: {Anthropic /v1/messages, OpenAI /responses, /chat/completions, MCP, ...} × {Claude Code, Codex, ...} with the loss dimensions each pair is asserted to preserve (tool args, SSE order, reasoning, cache_control, images, provider error detail).
- Turn the harness scenarios into a per-tag gate that asserts semantic equality, not just 200: tool arguments intact, SSE event order, reasoning round-trip, images forwarded, provider error preserved.
- Ship it as the marketing-credible artifact ("6 certified routes × 5 clients, tests public").
Priority
High. This is the single place the analysis finds a real process gap (the code architecture is already right); it directly protects the #476/#465 class from silent regression.
Context
Competitive analysis of LiteLLM, agentgateway, Portkey, Envoy AI GW, Kong AI GW and Bifrost shows the recurring, most-damaging failure class is protocol infidelity:
HTTP 200 + valid streaming + visible text, but the semantics of the agent turn are corrupted — droppedtool_usearguments, lost reasoning/thinking, strippedcache_control, dropped images, mangled SSE ordering. The gateway looks up while the agent is broken.This is a moving target: this cycle alone we shipped fixes for exactly this class — images dropped on the Responses path (#476), reasoning items not replayed (#465), tool-call salvage (
tool_salvage.rs). Every provider protocol change can reintroduce one.What we already have
src/features/harness/— record & replay ("sandwich") driver + mock backend.tests/enterprise/scenario_*.rs— failover, budget, DLP, routing scenarios.Gap
No published, per-release conformance matrix asserting semantic fidelity, only HTTP success. The analysis frames grob's credible claim as
N certified routes × M agent clients × published conformance testsrather than "100+ providers".Proposal
docs/reference/conformance.mdmatrix:{Anthropic /v1/messages, OpenAI /responses, /chat/completions, MCP, ...} × {Claude Code, Codex, ...}with the loss dimensions each pair is asserted to preserve (tool args, SSE order, reasoning, cache_control, images, provider error detail).Priority
High. This is the single place the analysis finds a real process gap (the code architecture is already right); it directly protects the #476/#465 class from silent regression.