sequential-testing

Runs ~20 options strategies against live market data in shadow mode, records every hypothetical fill under worst/base/optimistic assumptions, and grades each with anytime-valid e-processes. Places no orders.

Sequential testing for efficient and reliable comparison of stochastic algorithms.

Sequential Hypothesis Testing with e-Values and p-Values

Which model is really behind your API relay or agent IDE? Behavioral fingerprinting + anytime-valid sequential tests (FPR<=1%). LLMs can't be random - measured on 9 frontier models.

Fixed Sequence Multiple Testing Procedures

Walds Sequential Probability Ratio Testing (SPRT) statistical evaluator for LLM and AI agent comparison in Ruby.

Sequential A/B Testing Framework in Python

Bayesian multi-armed bandits for continuous prompt experimentation: Thompson sampling routes traffic to the best prompt variant and a stopping rule promotes a winner without a fixed-N A/B test. Zero dependencies, TypeScript-first.

Bayesian A/B Testing for Proportions

A/B testing engine: frequentist + Bayesian + CUPED variance reduction + peeking-safe sequential testing

A/B test analysis that is correct by default, informative by design, and composable by construction. 88 classes, 60+ paper references.

Anytime-valid A/B test analyzer that holds the false-positive rate under 1.5% while you peek at the dashboard continuously, where naive fixed-horizon testing leaks to 23% at 15 looks. mSPRT with confidence sequences, CUPED variance reduction (50% on the demo, SE 0.223 to 0.135), a peeking guard, and a reproducible A/A simulation.

Ad campaign performance evaluation using AB Testing

A solution for simulating realistic exoplanet transit light curves and training a classifier to detect exoplanet transits

Always-valid sequential A/B testing engine: mSPRT confidence sequences make peeking safe by construction, CUPED cuts variance up to 49%, SRM gates bad data. Built-in adversarial peeking harness proves the claim: naive daily peeking hit 27.5% false positives in 2,000 simulations; this engine held 1.7%. All numbers reproducible from committed seeds.

Design and analysis of A/B experiments: power, tests, bootstrap, SRM and a sequential peeking correction - every method validated by simulation.

[Alpha / WIP] Agent-first CLI for Bayesian and sequential A/B/n experiment decisions with JSON, CSV, XLSX, and text inputs.

Typed toolkit for A/B experiment analysis: power analysis, CUPED variance reduction, always-valid sequential testing (mSPRT), Welch/z analysis with CIs - property-based tests (hypothesis) and a worked forecasting-rollout case study.

Automated canary control plane for LLM configs: shadow testing, sequential inference (SPRT/CUSUM), drift monitoring

Causal bandit orchestration platform for real-time adaptive experimentation, sequential testing, and interference-aware decisioning.