Augur

A cost-first FinOps gate for AI agents. Know the bill before you ship.

Augur runs your AI agent against a representative workload before production, projects its unit economics at scale (cost per request / per user / per tenant / per month), and fails the build if it blows your budget.

The thesis: today you find out an agent is financially unsustainable after you ship and the bill arrives. Augur catches it in CI.

$ augur gate
# Augur cost report
**Verdict: ❌ FAIL** — 1 of 3 budget check(s) over limit

| check          | limit      | projected  | status |
|----------------|-----------:|-----------:|:------:|
| $/request p95  | $0.030000  | $0.043125  |   ❌   |
| $/tenant/month | $500.00    | $360.34    |   ✅   |
| monthly bill   | $20000.00  | $18016.88  |   ✅   |

$ echo $?
1   # the build fails

---

Honest positioning (read this before judging it)

This is not a virgin space, and pretending otherwise would be dishonest. What already exists:

• Token calculators (LiteLLM's calculator, dozens of tiktoken-based ones). You type average tokens and volume by hand. Useless for agents — you can't guess an agent's token usage, because one user request fans out into many LLM calls with retries and tool loops.
• Eval / CI platforms (Braintrust, Confident AI, Maxim, Promptfoo, Langfuse, …). They run your agent against a dataset in CI and gate the merge, and several attach token cost per call. But cost is a second-class citizen — they are quality gates (right tool? hallucination? grounded?) with a cost number bolted on the side.
• Predictive research (e.g. PreflightLLMCost) — narrow, focused on predicting output length. Not a productized gate.

What makes Augur different (the defensible wedge)

1. Cost-first, not cost-as-byproduct. The primary output is unit economics, not a quality score.
2. Projects to production scale. Not "this run cost $0.04" — "at your projected volume this is $X/tenant/month, which breaks your budget."
3. Models agentic cost drivers from real runs. It runs the agent N times and captures the cost distribution (retries, tool-loop fan-out, variance), not a single naive token estimate.
4. Complements eval gates; doesn't replace them. Quality gate: "is it good?" Augur: "can we afford it?"

Honest reality check: as a business this gets crushed by any eval incumbent adding a cost tab — there's no commercial moat. This is built as a portfolio piece: a clean, pure-Go systems tool (recording proxy, empirical distributions, bootstrap projection, CI gate) that shows real depth.

---

How it works

scenarios.yaml ─┐
                ├─► augur run ──drives──► your agent ──LLM calls──► augur proxy ──► provider
                │                              (sets X-Augur-* headers)  │
                │                                              writes cost trace
                │                                                        ▼
                │                                                  trace.jsonl
pricing.yaml ───┤                                                        │
traffic.yaml ───┼────────────────────────────► augur project ◄───────────┘
budget.yaml  ───┘                                     │  (distributions × traffic)
                                                       ▼
                                              augur gate ──► report.md / report.json + exit 0/1

The proxy captures calls at the HTTP layer (you point your agent's base_url at it), so Augur is framework- and language-agnostic and records the real call graph — retries and fan-out included — without you rewriting the agent. It speaks three wire dialects natively — OpenAI, Anthropic (/v1/messages, including the cache-read/cache-write token split), and Gemini (usageMetadata) — so you point it at whichever provider your agent already uses.

---

Quickstart

Build:

go build -o augur .

1. Make your agent honour the contract

augur run injects four environment variables per invocation. Your agent's only job is to read them — typically one line of OpenAI-client config:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["AUGUR_BASE_URL"],            # point at the proxy
    default_headers={
        "X-Augur-Scenario-Id": os.environ["AUGUR_SCENARIO_ID"],
        "X-Augur-Run-Id":      os.environ["AUGUR_RUN_ID"],
    },
)
prompt = os.environ["AUGUR_INPUT"]                    # the scenario input
# ... run your agent as usual ...

2. Describe your scenarios, traffic, and budget

See examples/ for documented templates:

• scenarios.yaml — representative inputs + the agent command + how many times to run each (the projection is only as good as these inputs).
• traffic.yaml — production assumptions (users, requests/user/day, tenants, scenario mix).
• budget.yaml — the thresholds that fail the build.
• pricing.yaml — a dated price snapshot (ships with the repo).

3. Run the pipeline

# Drive the agent N times per scenario through the recording proxy.
augur run --scenarios scenarios.yaml --upstream https://api.openai.com

# Eyeball the observed per-scenario cost distribution.
augur aggregate --trace trace.jsonl

# Project to production scale with confidence intervals.
augur project --traffic traffic.yaml

# Gate it: writes report.md / report.json, exits non-zero if over budget.
augur gate --traffic traffic.yaml --budget budget.yaml

augur gate is the one you wire into CI.

Tuning for CI

• Parallel scenarios. augur run --concurrency N (alias -j N) runs up to N agent invocations at once. With ~20 repetitions over network-bound LLM calls this cuts wall-clock time sharply; the proxy and trace ledger are concurrency-safe, and each invocation's stdout/stderr is flushed as one block so parallel output never interleaves. Defaults to 1 (sequential).
• Per-call timeout. augur run --timeout 5m / augur proxy --timeout 5m bounds how long the proxy waits on the provider (default 10m). A single call can override it with the X-Augur-Timeout header (a Go duration such as 300s) — handy for slow reasoning models (o1/o3/r1). The deadline is a context deadline, so a client that disconnects cancels the upstream call immediately (no orphaned tokens). --timeout 0 disables the proxy-imposed deadline.
• Model-alias fallback. augur aggregate --normalize-models / augur gate --normalize-models prices a model that is absent from the snapshot via its longest dash-delimited prefix (e.g. gpt-4o-2024-08-06 → gpt-4o), so a provider bumping a dated suffix doesn't fail the build with unknown model. It is opt-in and prints a warning naming every model it normalized, because a silent fallback can mis-bill.

Record once, replay for free

Running the agent against the real provider on every CI push spends real tokens. Record the responses once, then replay them — augur run --replay re-executes the agent against the recorded responses and regenerates the trace without contacting the provider:

# Once (locally or nightly), against the real provider:
augur run --scenarios scenarios.yaml --record cassette.jsonl

# In CI, on every push — zero tokens spent:
augur run --scenarios scenarios.yaml --replay cassette.jsonl --trace trace.jsonl
augur gate --traffic traffic.yaml --budget budget.yaml

Because replay re-runs the agent (not just the trace), a cost regression from an agent code change still surfaces — exercised against the old responses, for free. Commit cassette.jsonl alongside your scenarios. (A call the agent makes that wasn't recorded is a replay miss — reported loudly, so divergence from the recording never passes silently.)

What-if sensitivity analysis

project and gate take what-if multipliers that re-cost the recorded trace under hypothetical agentic cost drivers — no agent re-run, no tokens:

# "What if retries climb 30%, sub-agent fan-out adds 50% more calls, and
#  context grows to 2× as conversations lengthen?"
augur project --retry-rate 0.3 --fanout 1.5 --context-growth 2

# Gate against a pessimistic scenario, not just today's happy path:
augur gate --context-growth 1.5 --budget budget.yaml

Retries and fan-out scale the whole call (more calls); context growth inflates only the prompt side, not the completion — so the model reflects which driver moved, not a flat fudge factor.

Predict cost without running (Python sidecar)

The what-if knobs above re-cost a recorded trace, but they hold output length fixed. The one thing you genuinely can't guess for an agent is completion length — and it's where the cost surprise lives. The optional sidecar/ (Python) learns output length from a recorded trace, then projects cost for inputs you never ran:

# Learn output_tokens ≈ a + b·input_tokens per model, from one recorded run.
python -m augur_predict fit --trace trace.jsonl --out model.json

# "What if prompts grow 1.5× as conversations lengthen?" — predict, don't run.
python -m augur_predict emit-trace --model model.json --out grown.jsonl --input-scale 1.5

# The predicted trace flows through the normal gate — no tokens spent.
augur gate --trace grown.jsonl --traffic traffic.yaml --budget budget.yaml

Coupling is the trace file and nothing else — no RPC, no shared library. Unlike --context-growth, the sidecar feeds the larger prompt back through the output model, so a bigger ask predicts a longer answer, not just a costlier prompt. The default fit is an honest linear baseline (it reports R² and falls back to the mean when the signal is weak); --dist quantile fits conditional quantiles directly to model the right-skewed tail the gate's p95 lives in, and --run-correlation widens the per-run spread to match correlated runs. See sidecar/README.md.

Self-hosted models (TCO)

For a model you run yourself there's no per-token API price — you pay for an instance by the hour. Describe the deployment in tco.yaml (instance $/hr, serving throughput, utilization) and Augur derives the effective $/Mtok:

augur tco --tco tco.yaml          # show the derived effective $/Mtok
augur gate --tco tco.yaml ...     # cost the trace against self-hosted pricing

--tco is accepted by aggregate, project, and gate as an alternative to --pricing. Utilization is the honest part: you pay for the box 24/7 but it's rarely saturated, and a half-idle instance doubles the effective token price.

GitHub Action

Augur ships as a composite action that runs the gate on a pull request, posts the report as a comment, and fails the check if the projection is over budget. Combined with a committed cassette, the PR check spends no tokens:

# .github/workflows/cost-gate.yml
name: Cost gate
on: pull_request
jobs:
  augur:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write   # to post the report comment
    steps:
      - uses: actions/checkout@v4
      - uses: Cro22/augur@v1.1.0   # pin a release tag → prebuilt binary
        with:
          cassette: cassette.jsonl   # replay → zero tokens
          traffic: traffic.yaml
          budget: budget.yaml

No Go toolchain required. When you pin the action to a release tag (@v1.1.0), it downloads the matching prebuilt binary from the release and checksum-verifies it — so a Python/TS/Node agent repo runs the gate without a Go build. Pin to a branch or SHA (@master) and it transparently falls back to building from source; force either mode with the version input (source, or a specific tag). Binaries are produced by GoReleaser on every v* tag (release workflow).

See action.yml for all inputs and examples/github-workflow.yml for a fuller example.

Cutting a release (maintainers): git tag v1.2.0 && git push origin v1.2.0 triggers GoReleaser to cross-compile (linux/darwin/windows × amd64/arm64) and attach the binaries + checksums.txt to the GitHub release.

---

Design decisions

• Gate on p95, not the mean. The cost surprise lives in the tail — long outputs, retry storms, tool loops. The mean hides exactly the failure mode Augur exists to catch. Conservative by design.
• Observation via a proxy, not an SDK wrapper. Works with any agent that can set a base URL; captures the real call graph.
• The trace records tokens, not dollars. Cost is computed downstream, so the same trace can be re-priced against a different snapshot without re-running the agent.
• Pricing is an updatable, dated snapshot. Provider prices drift constantly; pricing.yaml says when it was captured. Do not assume it's current.

Honest limitations

• Streaming usage. For OpenAI streaming, the provider only emits a usage block when the request sets stream_options.include_usage. Augur injects that flag automatically (disable with --inject-usage=false), so streamed calls are captured exactly rather than as a zero-token row — the trade-off is one extra usage-only SSE chunk the agent receives (benign for standard clients). Anthropic and Gemini report usage on every streamed response with no opt-in.
• Multipliers. Augur reports calls per run and splits it into retries per run — a call whose request body is byte-identical to an earlier one in the same run, the observable signature of a client-library retry — vs the rest. It still does not isolate sub-agent fan-out from sequential tool-loop steps: both are distinct-body calls, and call concurrency isn't visible at the HTTP layer.
• Cache-write pricing is modelled from cache_write in pricing.yaml (Anthropic's ~1.25x input premium on cache_creation_input_tokens). It is a dated snapshot like every other price — verify it against the current page.
• Running the agent in CI spends real tokens. Keep the scenario set small and runs modest, or record once and replay (--record/--replay) so CI pushes spend nothing.

Status

v1 complete — all milestones implemented and tested (pure Go, only external dependency is gopkg.in/yaml.v3):

Hito 0	single-call cost computation
Hito 1	OpenAI-compatible recording proxy (streaming + non-streaming)
Hito 2	scenario runner + per-scenario aggregation
Hito 3	projection engine with bootstrap confidence intervals
Hito 4	budget gate + Markdown/JSON report + CI exit codes
Hito 5	record/replay cassette (--record/--replay), what-if knobs (--retry-rate/--fanout/--context-growth), self-hosted TCO mode (augur tco, --tco), GitHub Action (action.yml), Python output-length prediction sidecar (sidecar/)

Every SPEC milestone and stretch is implemented. The Go core is pure Go (only external dependency gopkg.in/yaml.v3); the optional prediction sidecar is Python (numpy + scipy), coupled to the core through the trace file alone.

See SPEC.md for the full design.

Dogfooded on a real agent

Augur has been run against a real, non-trivial agent — the Insights Agent from its sibling project CloudOracle (a LangGraph supervisor multi-agent with tool calls and guardrails), not just synthetic traces. CloudOracle is FinOps for cloud (runtime); Augur is FinOps for AI agents (pre-prod) — dogfooding one on the other closes the loop.

Because that agent talks to its model natively through LangChain (no OpenAI base_url), the capture used the SPEC's documented proxy fallback (ADR D1): a LangChain callback shim that writes Augur's trace schema from the usage every call reports — without editing CloudOracle. Full walkthrough and harness: examples/cloudoracle/.

(The proxy now speaks Anthropic's /v1/messages natively, so an Anthropic agent that can set a base_url — e.g. LangChain's ChatAnthropic(base_url=…) — can take the HTTP path directly and skip the shim; the shim remains the fallback for frameworks that don't expose a base URL at all.)

What it found (Claude Haiku 4.5, 20 runs): gate PASS at $/request p95 $0.0198 (budget $0.02). The headline is the find-savings scenario — its p95 cost is 2.3× its median, driven by a call-count tail (5 → 13 calls/run when the savings specialist's tool loop keeps going). That agentic cost driver, observed on a real agent, is exactly what Augur exists to catch; gating on the mean would hide it.

Dogfooding also surfaced real findings about the agent — it isn't provider-portable (its LLM judge sends a system-only message Anthropic rejects but Gemini tolerates), Claude Haiku hallucinates dollar figures that trip the agent's grounding check, and the sidecar's input→output signal is weak there (output length is driven by call role, not prompt size). Details in the example README.

Naming

Augur — a seer who reads omens to foretell what's coming; here, an agent's production cost before it ships. (There is an unrelated crypto project of the same name; this is a personal portfolio repo in a different domain.)

License

Apache 2.0 — see LICENSE.