selective-classification

A list of papers that studies out-of-distribution (OOD) detection and misclassification detection (MisD)

PyTorch implementation of our ECCV 2022 paper "Rethinking Confidence Calibration for Failure Prediction"

A decision-safety lab for loan approval: trains a baseline classifier, calibrates probabilities (ECE/Brier), sweeps confidence thresholds to build a coverage, quality frontier and outputs a defensible abstention policy (auto-decide vs review). Includes a Streamlit dashboard for report cards, triage UI, and data quality checks.

Longform article reframing abstention (reject option / selective prediction) as product design, not model weakness. Covers coverage as a KPI, calibration as a prerequisite, threshold selection under review capacity and risk, queue/UX design for human-in-the-loop workflows, and anti-patterns that break safety in production.

Decision-safe evaluation + Streamlit dashboard for AI vs Human vs Post-Edited AI text detection. Generates a reliability report card (Accuracy, Macro F1, ECE, Brier), calibration plots, confidence histograms, and a coverage-vs-performance abstention curve. Recommends an operating threshold for human-review routing.

GenPark AI Agent Skill - Decomposes total prediction uncertainty into epistemic (model knowledge void) vs aleatoric (inherent task ambiguity) components via Shannon mutual information.

GenPark AI Agent Skill - Post-processing confidence calibration using temperature scaling, expected calibration error (ECE), and Brier score evaluation.

Promises and Pitfalls of Threshold-based Auto-labeling (NeurIPS 2023, Spotlight)

GenPark AI Agent Skill - Distribution-free conformal prediction sets with rigorous marginal coverage guarantees, calibrated non-conformity scores, and split conformal inference.

GenPark AI Agent Skill - Risk-coverage trade-off optimization; controls agent abstention, fallback escalation, and human-in-the-loop triggers under dynamic risk budgets.

GenPark AI Agent Skill - Detects LLM hallucinations and confabulations by clustering sampled responses into semantic equivalence classes and computing semantic entropy.

GenPark AI Agent Skill - Risk-coverage trade-off optimization; controls agent abstention, fallback escalation, and human-in-the-loop triggers under dynamic risk budgets.

GenPark AI Agent Skill - Distribution-free conformal prediction sets with rigorous marginal coverage guarantees, calibrated non-conformity scores, and split conformal inference.

GenPark AI Agent Skill - Post-processing confidence calibration using temperature scaling, expected calibration error (ECE), and Brier score evaluation.

GenPark AI Agent Skill - Detects LLM hallucinations and confabulations by clustering sampled responses into semantic equivalence classes and computing semantic entropy.

GenPark AI Agent Skill - Decomposes total prediction uncertainty into epistemic (model knowledge void) vs aleatoric (inherent task ambiguity) components via Shannon mutual information.

Code for "Budgeted Classification with Rejection: An Evolutionary Method with Multiple Objectives", at IEEE CEC 2022

Robust Selective Classification of Skin Lesions with Asymmetric Costs

More human oversight can make an AI agent less safe. Headroom is a human-in-the-loop firewall for coding agents that measures when to trust the human: oversight as resource allocation, not just classification.

Reliable and Efficient Selective Machine Learning