PREFLIGHT

An AI risk gate for payment-recovery agents. Razorpay AI Buildathon — Track 02, AI Risk Manager.

Live Demo · 5-minute Pitch · Proof · Control room · Architecture · Evidence

PREFLIGHT sits between a recovery agent and the customer. Before a proposed message is sent or a recovery action is taken, it checks whether that intervention is still true and still appropriate against authoritative current state — and blocks it if it is not.

The model identifies what must be true.

Provider state tells us what is true.

Deterministic policy decides what may happen.

---

At a glance

Loss class	A stale recovery message sends a customer who has already paid back into a payment flow — duplicate-collection risk, refund and support load, lost trust.
Razorpay integration	Real. A live read-only GET against Razorpay Test Mode returned captured/paid, and the stale message was blocked against it → §2
Measured evidence	On latent mutable-state dependency detection: model recall 0.889 vs repaired deterministic baseline 0.587 — +0.302, 95% CI [0.137, 0.469]. Post-hoc segmentation-repaired diagnostic, not a confirmatory result → §6
What broke	Our first number was +0.556. A self-audit found a segmentation defect in our own baseline that had been inflating it. We publish the smaller number → §8
What is unproven	The pre-registered confirmatory corpus was not run; achieved power was 0.62 against our own 0.15 threshold; under the stricter CI reading the effect does not clear it → §9
Verify it	uv sync && uv run pytest → 558 Python tests, no API key, no credentials, no network

---

1. The failure this prevents

A recovery agent reasons from an event. "Payment failed at 12:03" arrives, and it drafts something correct-for-that-moment:

Hi Meera, your payment for order order_DEMO0001 didn't go through. Complete your payment here: https://rzp.io/i/…

While that message sits in a queue, the customer pays on another device. The message is now false, and the directive in it is worse than false.

If the stale recovery still goes out, the customer is sent back into a payment flow even though the merchant already has the money — creating duplicate-collection risk, refund and support work, and loss of trust.

The hard part is not that state changes. That is obvious. The hard part is that the dependence is latent: nothing in "Complete your payment here" names a payment status, so there is no keyword to filter on. The sentence is only appropriate if payment_requires_completion currently holds, and it never says so.

In our frozen 48-message pilot, 43 of 48 messages carried at least one latent mutable-state dependency — because the template that recovers a payment is the template that presupposes one is owed.

---

2. Proof: a real Razorpay Test Mode run

Not a mock. A live read-only GET against api.razorpay.com/v1, against a Razorpay Test Mode payment whose provider state was captured / paid. Test Mode transactions are test transactions — no real money moved here, and none was ever meant to. What this run establishes is that the gate reads authoritative provider state over the real API and decides against it.

In the real-provider path, Razorpay supplies that state. The public sandbox does not: it runs on controlled fixture worlds, and RazorpayTestModeProvider is unreachable from it (§4b).

AGENT PROPOSES     "Complete your payment here:"        (recorded gpt-5.6-terra output)

RAZORPAY SAYS      payment.status = captured            (live Test Mode read)
                   order.status   = paid
                   pay_TO7sMXmu6KtSwu / order_TO7OzdK9bO0h46 / 10000 INR minor

PREFLIGHT          payment_not_captured  VIOLATED
                   order_not_paid        VIOLATED

VERDICT            BLOCK
                   execution = SUPPRESSED

Evidence packet provenance — three layers, each labelled for what it actually was:

"provenance": {
  "semantic_analysis": "RECORDED_TERRA_CASSETTE",
  "state":             "RAZORPAY_TEST_MODE",
  "verdict":           "DETERMINISTIC_ENGINE"
}

• Semantic analysis — replayed, not live. Verbatim gpt-5.6-terra output from the frozen semantic-classifier-v1, replayed from data/product/terra_cassette.json. No model call was made during this run.
• Provider state — live. A read-only GET against Razorpay Test Mode, with rzp_test_ credentials supplied through the environment only.
• Verdict — deterministic. Plain Python in engine.py. No model output reaches it as authority.

Committed at results/product/real_provider_run.json. Only GET was issued. No customer communication was sent, no write of any kind was made, and no money moved — an ALLOW would have executed as SIMULATED. Credentials were supplied through the environment only and appear nowhere in the packet, the logs, or this repository.

---

3. Architecture in 30 seconds

              Agent proposal
                    |
                    v
        SEMANTIC ANALYSIS  (LLM)          "What does this intervention depend on?"
                    |
                    v
        AUTHORITATIVE RAZORPAY STATE      "What is actually true right now?"
                    |
                    v
        DETERMINISTIC CONTRACTS           "What may happen?"
        + PREDICATE RESOLUTION
                    |
                    v
        ALLOW / BLOCK / ESCALATE / UNVERIFIABLE
                    |
                    v
        Evidence packet  +  action ledger

The LLM does NOT determine payment truth, and does NOT produce the financial verdict. The LLM DOES identify the semantic dependencies a customer-facing intervention carries.

---

4. Quickstart

Try it live: https://preflight-gate6.vercel.app/ — control room · proof

The demo backend runs on Render's free tier. The first request after it has been idle can take up to about a minute to wake the service; the interface keeps the request alive and reports WAKING DECISION SERVICE while it does. Requests after that are fast.

Locally, the whole product runs with no API key, no Razorpay credentials, and no network.

uv sync
uv run python -m preflight.product.cli scenario stale-payment   # BLOCK
uv run python -m preflight.product.cli scenario valid-recovery  # ALLOW
uv run python -m preflight.product.cli scenario all
uv run streamlit run src/preflight/product/ui.py                # UI

Public presentation and programmatic adapter:

uv run uvicorn preflight.api.app:app --reload --port 8000     # bounded public API

cd frontend
npm install
set NEXT_PUBLIC_PREFLIGHT_API_URL=http://localhost:8000        # Windows cmd
npm run dev                                                   # public Next.js interface

The public API has two endpoints. /api/evaluate accepts only stale-payment and valid-recovery and replays recorded Terra output. /api/sandbox accepts a visitor's own message and calls the frozen classifier live (§4b) — it is default-off, rate limited, and still fixture-backed. Neither exposes payment lookup, provider selection, model selection, or cross-request durable idempotency. Set PREFLIGHT_CORS_ORIGINS to a comma-separated list of exact frontend origins in deployed environments; wildcard origins are refused.

Production deployment is split deliberately:

• Vercel: set the project root to frontend/, set NEXT_PUBLIC_PREFLIGHT_API_URL=https://YOUR-BACKEND, then deploy the Next.js project.
• Render / Uvicorn: use render.yaml, set PREFLIGHT_CORS_ORIGINS=https://YOUR-FRONTEND to the exact Vercel origin, and start with uv run uvicorn preflight.api.app:app --host 0.0.0.0 --port $PORT --proxy-headers --forwarded-allow-ips="*" --workers 1.

Render's load balancer is the only public path to the service's bound port, so Uvicorn is the explicit trusted-proxy boundary. The application uses Uvicorn's canonical request.client address and does not parse caller-supplied forwarding headers itself.

NEXT_PUBLIC_PREFLIGHT_API_URL is a public URL, not a secret. No Razorpay key, OpenAI key, private token, or internal credential belongs in any NEXT_PUBLIC_* variable. The deployed API remains fixture-only for provider state: no Razorpay credentials and no caller-supplied payment IDs. OPENAI_API_KEY is server-side only and never reaches the browser.

Interface roles are deliberately separate:

• CLI -- reproducible engineering proof.
• Streamlit -- internal inspection and evidence interface.
• FastAPI -- bounded programmatic interface over the frozen Python core.
• Next.js -- public product and submission interface.

With real recorded gpt-5.6-terra output (7 verbatim responses, committed):

uv run python -m preflight.product.cli scenario stale-payment --analyzer cassette

With live Razorpay Test Mode (read-only; needs your own rzp_test_* credentials):

export RAZORPAY_KEY_ID=rzp_test_...      # environment only; never committed, never logged
export RAZORPAY_KEY_SECRET=...

uv run python -m preflight.product.cli razorpay-read pay_XXXXXXXXXXXX
uv run python -m preflight.product.cli scenario stale-payment \
    --provider razorpay-test --payment-id pay_XXXXXXXXXXXX --analyzer cassette

4b. Try your own message

The public control surface has two modes. Guided demo replays the recorded classifier output for the two preset scenarios. Try your own message sends what you type to the frozen semantic-classifier-v1 live, and judges it against one of two already-tested fixture worlds.

POST /api/sandbox
{ "message": "Complete your payment here.", "state_profile": "already_completed" }

Two fields, and the schema forbids a third. The browser cannot choose the provider, the payment identifiers, the action, the analyzer, or any model parameter — a body carrying payment_id, provider or model is rejected with 422 rather than ignored.

already_completed	the stale-payment fixture — payment captured, order paid
still_unresolved	the valid-recovery fixture — payment failed, order attempted
action	always SEND_RECOVERY_MESSAGE
provider	always the fixture; RazorpayTestModeProvider is unreachable from this endpoint
analyzer	always live Terra; there is no stub fallback

If the classifier cannot run — no key, a transport failure, an unparseable reply — the endpoint returns 502 and no verdict. It never substitutes a weaker analyzer and labels the result as Terra. The live path is limited to 3 requests per minute per client, 5 admitted attempts per client per UTC day, and 50 admitted attempts globally per UTC day. The message is capped at 500 characters, is never logged, and nothing is persisted (evidence_dir=None, in-memory ledger).

Daily quota is consumed only after request validation, the sandbox kill switch, and the minute limiter all pass. Admission is counted immediately before Terra is called. An admitted attempt therefore remains counted if Terra fails because provider resources may already have been consumed; the service does not retry. Invalid/blank/oversized requests, forbidden fields, invalid profiles, disabled-sandbox requests, and minute-limit rejections do not consume daily quota.

Daily counters roll over on the UTC date. They are intentionally in-memory and process-local: they reset on a Render restart or redeploy, and separate workers would have separate counters. The current deployment uses one Uvicorn process on one instance.

Deployment environment

Variable	Where	Value
OPENAI_API_KEY	Render only, secret	server-side secret required only for the live custom-message sandbox
PREFLIGHT_SANDBOX_ENABLED	Render	true (default is false — the kill switch)
PREFLIGHT_SANDBOX_PER_IP_DAILY_LIMIT	Render	5
PREFLIGHT_SANDBOX_GLOBAL_DAILY_LIMIT	Render	50
PREFLIGHT_CORS_ORIGINS	Render	the exact frontend origin
NEXT_PUBLIC_PREFLIGHT_API_URL	Vercel	the Render service URL

The frontend receives no OpenAI secret. OPENAI_API_KEY is read from os.environ server-side and appears in no response body, no log line, and no NEXT_PUBLIC_* variable. Never add it to Vercel.

5. Reproduce everything

Offline reproduction — the whole core, no credentials of any kind:

uv sync
uv run pytest                        # 558 Python tests, fully offline
uv run preflight verify              # 37 checks: freeze hashes, contamination, no credentials
uv run python scripts/score_task_b.py               # the frozen pilot evaluation
uv run python scripts/baseline_integrity_audit.py   # the audit that found F012

cd frontend
npm install
npm test                             # 41 frontend tests
npm run lint
npm run build

Deterministic throughout: fixture provider, fixed scenarios, seeded bootstrap (BOOTSTRAP_SEED = 20260827), pinned matcher threshold (IOU_THRESHOLD = 0.50).

A reviewer never needs a paid API key to verify this repository. Three paths, kept deliberately separate:

Path	Needs	Covers
Offline reproduction	nothing	engine, contracts, resolver, providers, ledger, the frozen evaluation and its audits, the frontend — everything above
Live Terra	your own OPENAI_API_KEY	re-running the semantic classifier instead of replaying the committed cassette (§4b)
Razorpay Test Mode	your own rzp_test_* credentials	re-running the read-only provider path of §2

---

6. Evaluation

Headline — POST-HOC SEGMENTATION-REPAIRED DIAGNOSTIC

Frozen 48-message held-out pilot · 336 human-labelled semantic units · 63 LATENT + MUTABLE_CURRENT_STATE units.

system	precision	recall	F1	unsafe-message recall
deterministic baseline (rules-v2-diag-seg)	0.514	0.587	0.548	0.767
semantic-classifier-v1 (gpt-5.6-terra)	0.675	0.889	0.767	0.953

Recall difference +0.302, 95% CI [0.137, 0.469] (percentile cluster bootstrap over messages), paired discordance 21 : 2, McNemar exact p = 6.6e-05.

Four disclosures belong with those numbers, every time:

• This repaired comparison is POST-HOC DIAGNOSTIC. The segmentation repair was made after unblinding. It is not a pre-registered result.
• The planned fresh confirmatory corpus was not executed before submission. docs/POOLING_DECISION.md bars the pilot from primary inference; the fresh final corpus (69 messages required against a capacity of 86) was never generated.
• Achieved pilot power against the pre-registered +0.15 threshold was 0.62 (0.46 under the double-p10 sensitivity), against a pre-registered target of 0.80.
• The pre-registered CI / materiality decision rule was ambiguous. It never states whether +0.15 applies to the point estimate or to a CI bound, and §9 K2/K3 — the only place a numeric CI rule would have lived — read TBD and were never completed. Under the stricter lower-CI-bound > +0.15 interpretation, 0.137 does not clear +0.15.

The original frozen comparison

system	precision	recall	F1	unsafe-message recall
rules-baseline-v2 (frozen comparator)	0.284	0.333	0.307	0.419
semantic-classifier-v1	0.675	0.889	0.767	0.953

Difference +0.556, 95% CI [0.375, 0.726], discordance 38 : 3.

F012 later established that the frozen baseline segmenter violated the written ontology and inflated the apparent model advantage. +0.556 is disclosed, never headlined.

Full audit: results/prereg_audit.md · results/baseline_integrity_audit.md

---

7. The AI / non-AI boundary

For "Complete your payment here." the frozen classifier returns:

semantic_unit_type = DIRECTIVE
dependency_type    = MUTABLE_CURRENT_STATE
expression_mode    = LATENT
predicate          = requires_completion
grounding_target   = payment.status

Deterministic code then asks the provider whether that predicate holds, and deterministic code emits the verdict. Five properties make this a boundary rather than a slogan:

• Regime A runs before Regime B is read. A violated precondition short-circuits to BLOCK without consulting the classifier at all.
• The classifier's output type has no verdict field. SemanticAnalysis structurally cannot carry a decision.
• Financial resolution requires two exact gates: a grounding_target that is an exact member of GROUNDING_TARGETS, and an exact human-reviewed predicate valid for that target. Both are dict lookups on casefolded strings — no fuzzy matching, no embeddings, no LLM-assisted mapping, no substring guessing, and no semantic default inferred from a target alone. Anything else escalates to a human.
• Aliases are reviewed, not harvested. Appearing in model output is not a reason to map a predicate. Eleven were rejected, each with its reasoning recorded in resolver.REJECTED_ALIASES.
• confidence is never consulted. A confident classifier claim about a captured payment still loses to the provider.

Adversarial tests hold this down: a classifier that returns nothing, returns garbage, raises, or insists nothing depends on state cannot reach ALLOW once the payment is captured.

Per-component accounting: AI_JUDGMENT_LEDGER.md.

---

8. What broke

Every failure is recorded as it happened in FAILURE_LEDGER.md. Four matter, and the first one changed the headline number.

F012 — the baseline segmenter kept bare URLs inside semantic units, contrary to ONTOLOGY_V2.md §8 rule 7. Recovery directives containing links were segmented too broadly, so IoU matching turned real rule detections into unmatched misses. 38 baseline spans in the pilot carry a bare URL; when the frozen rule baseline — the deterministic comparator, not the semantic LLM — was handed the gold span instead, it classified 27 of those 35 unmatched spans correctly. The misses were in segmentation, not in the rules. The original comparator was preserved, a segmentation-only diagnostic repair was created, and the model advantage fell from +55.6pp to +30.2pp. The smaller number is now the public headline.

F011 — a cost optimisation leaked sibling context into the AI's Task A run, giving the model context the ablation existed to remove. Caught mid-flight, before any metric existed; the run was invalidated and quarantined, and its errors were deliberately left unread.

F013 — the resolver let a supported grounding target supply a financial meaning by itself. An unknown model predicate on a known field inherited that field's default check. Removed: resolution now requires both an exact target and an exact reviewed predicate.

F009 / F010 — a predeclared length correction prevented a 2.3× capacity error, and a lost timing event was not reconstructed after the annotator had already seen the message.

F011 and F012 were independent defects and both biased the evaluation toward the AI hypothesis. They were discovered because we explicitly audited the evaluation apparatus for failure. We corrected or invalidated them when found; we cannot establish that all residual sources of bias have been identified.

---

9. Limitations

False-positive cost

A false positive means PREFLIGHT blocks a legitimate recovery intervention. The cost is delayed or lost recovery, plus unnecessary human review. No rupee value is claimed.

• The pre-registered confirmatory test was never run. Every comparative figure here is computed on data the frozen plan classifies as sizing and descriptive data, at 0.62 achieved power against the pre-registered +0.15 threshold.
• Our own materiality criterion is ambiguous, and one reading fails. Point estimate +0.302 clears +0.15; CI lower bound 0.137 does not. Both are reported.
• Only gpt-5.6-terra was evaluated as a semantic classifier. No other model was compared.
• The pilot contained 0 AMBIGUOUS and 0 UNVERIFIABLE human labels. The ESCALATE and UNVERIFIABLE pathways are demonstrated as engineering behaviour and are not empirically validated. No ambiguous cases were manufactured.
• The production base rate is unknown. Test Mode does not estimate production frequency.
• Action contracts and predicate aliases require human authoring. Two action contracts, four canonical predicates, two grounding targets, and eighteen reviewed predicate mappings (fourteen of them aliases). Anything outside escalates — safe, but it caps coverage.
• The corpus is synthetic, English-first, generated under a frozen policy from public Razorpay documentation. It is not merchant traffic.
• The real Test Mode run proves integration, not prevalence. One payment, read-only. It does not show how often this happens in production.
• The pilot is not claimed to have been incapable of contamination: the classifier author had seen aggregate pilot prevalence before classifier freeze, though prompt development was restricted to pre-pilot ontology, annotation-guide and calibration materials.

Full account: LIMITATIONS.md.

---

10. Repository map

Path	What it is
frontend/	Public presentation. Next.js, React, TypeScript, Tailwind CSS
src/preflight/api/	Thin FastAPI adapter over the frozen product core
src/preflight/product/	The product. Engine, contracts, resolver, providers, ledger, CLI, UI
tests/	558 Python tests; test_product_*.py are engine, mutations, grounding, Razorpay, sandbox, UI, cassette
data/product/terra_cassette.json	7 verbatim gpt-5.6-terra responses
results/product/real_provider_run.json	The live Razorpay Test Mode evidence packet
docs/SUBMISSION_VIDEO_RUNBOOK.md	Exact demo sequence
research/README.md	Index to the evaluation and its audits
FAILURE_LEDGER.md · LIMITATIONS.md	What broke · what this cannot establish

Product module map

Module	Role
contracts.py	Regime A. Human-authored preconditions per action type. No model.
regime_b.py	Regime B. Extracts dependencies. Cannot express a verdict.
resolver.py	The two-gate boundary and the closed predicate vocabulary.
engine.py	The only place a verdict is produced.
ledger.py	Operation identity and idempotency (SQLite).
providers/	Fixture provider, and read-only Razorpay Test Mode.

---

11. Research and evidence

The product sits on a pre-registered falsification experiment whose job was to find out whether the semantic layer deserved to exist at all — baseline first, labels second, predictions third. Start at research/README.md.

Safety

No real customer messages. No production credentials — a key id that is not rzp_test_* is refused in code before any request, with no override. No money movement. No writes to any merchant system: the Razorpay provider has no method that creates, captures, refunds or notifies. No credentials committed — enforced by preflight verify.

Status

Decision core frozen 2026-08-29 at commit 3390885. Engine, contracts, resolver, providers, scenarios, classifier, baselines, datasets and evaluation results are unchanged since that commit. Everything after it is public surface: the FastAPI adapter, the Next.js frontend, the Streamlit presentation layer, and this README.

Submission surfaces finalized at commit feb5365 — the current main.

558 Python tests · 41 frontend tests · preflight verify 37/37 · ruff check clean · mypy clean on src/preflight/api and src/preflight/product.

Two caveats belong on that last line. Repository-wide mypy still reports 15 pre-existing errors across src/preflight/baseline_rules_v2_diag.py, scripts/baseline_integrity_audit.py and tests/test_product_razorpay.py. Separately, ruff format --check would re-wrap 11 files — cosmetic line wrapping only, which is why ruff check still passes. Both live in product, adapter, research and test code that is frozen for submission, and neither was fixed by editing that code to make this sentence shorter.