diff --git a/README.md b/README.md index 38846bc..22e2e14 100644 --- a/README.md +++ b/README.md @@ -243,6 +243,7 @@ curl -X POST http://localhost:8000/api/predict \ - Calibration may drift under distribution shift. - Risk probabilities are model estimates, not financial-loss probabilities. - The online feature-store layer is outside the current research scope. +- Network features are strong (see ablation), which raises the question of whether they transfer to accounts absent from training. See [Unseen-Entity Generalization](docs/research/unseen_entity_generalization.md) for a dedicated experiment on this. ## Dataset Reference diff --git a/docs/assets/unseen_entity_generalization.png b/docs/assets/unseen_entity_generalization.png new file mode 100644 index 0000000..a34ec8e Binary files /dev/null and b/docs/assets/unseen_entity_generalization.png differ diff --git a/docs/assets/unseen_entity_results.json b/docs/assets/unseen_entity_results.json new file mode 100644 index 0000000..28d2e21 --- /dev/null +++ b/docs/assets/unseen_entity_results.json @@ -0,0 +1,43 @@ +{ + "alert_rate": 0.005, + "accounts_seen_in_training": 693879, + "standard_out_of_time": { + "label": "Standard out-of-time (full test set)", + "transactions": 1420913, + "positives": 1694, + "prevalence": 0.0011921912179000402, + "metrics": { + "pr_auc": 0.9859286355765932, + "roc_auc": 0.9998338278230887, + "precision_at_alert_rate": 0.23673469387755103, + "recall_at_alert_rate": 0.9929161747343566, + "lift_at_alert_rate": 198.5710767896297 + } + }, + "both_parties_seen": { + "label": "Both sender and receiver seen during training", + "transactions": 716838, + "positives": 1569, + "prevalence": 0.002188779054681811, + "metrics": { + "pr_auc": 0.9917797464630843, + "roc_auc": 0.9998019722862765, + "precision_at_alert_rate": 0.4345885634588563, + "recall_at_alert_rate": 0.992989165073295, + "lift_at_alert_rate": 198.55296153774358 + } + }, + "unseen_entity": { + "label": "At least one party unseen during training", + "transactions": 704075, + "positives": 125, + "prevalence": 0.00017753790434257713, + "metrics": { + "pr_auc": 0.7912056121187283, + "roc_auc": 0.9997998835144541, + "precision_at_alert_rate": 0.03521726782164158, + "recall_at_alert_rate": 0.992, + "lift_at_alert_rate": 198.36478273217836 + } + } +} diff --git a/docs/research/unseen_entity_generalization.md b/docs/research/unseen_entity_generalization.md new file mode 100644 index 0000000..e571761 --- /dev/null +++ b/docs/research/unseen_entity_generalization.md @@ -0,0 +1,75 @@ +# Unseen-Entity Generalization + +## Question + +The main results show PR-AUC jumping from 0.0898 (transaction features only) to +0.9435 once network features (lifetime counts, degree, counterparty HHI) are +added. That's a large amount of signal riding on account history. This asks a +narrower question than the standard out-of-time test: + +Does performance hold when the sender or receiver was never observed during +training, or is it propped up by history that only exists for accounts the +model has already seen? + +## Method + +No retraining. Reuses the artifact from the standard temporal split and +partitions the same held-out test set (1,420,913 transactions) by whether +both parties appeared in the training window (693,879 accounts): + +```bash +python scripts/unseen_entity_evaluation.py +``` + +## Results + +| Partition | Transactions | Positives | Prevalence | PR-AUC | ROC-AUC | Recall @ 0.5% | Lift @ 0.5% | +|---|---|---|---|---|---|---|---| +| Standard out-of-time (full test set) | 1,420,913 | 1,694 | 0.119% | 0.9859 | 0.9998 | 99.29% | 198.6x | +| Both parties seen in training | 716,838 | 1,569 | 0.219% | 0.9918 | 0.9998 | 99.30% | 198.6x | +| At least one party unseen | 704,075 | 125 | 0.018% | 0.7912 | 0.9998 | 99.20% | 198.4x | + +![Generalization to Unseen Entities](../assets/unseen_entity_generalization.png) + +## Interpretation + +PR-AUC and precision both drop sharply for the unseen-entity partition +(0.99 → 0.79, precision 0.43 → 0.035). Taken alone, that looks like the network +features fail to transfer. But three other numbers move by less than noise +across all three partitions: + +- **ROC-AUC**: 0.9998 in every partition — invariant to class balance. +- **Recall @ 0.5% alert budget**: ~99.2-99.3% in every partition. +- **Lift over base rate**: ~198x in every partition. + +PR-AUC's baseline is the positive prevalence itself, and the unseen-entity +partition has an 8x lower prevalence (0.018% vs 0.219%) than the seen-pair +partition. A drop in PR-AUC is exactly what a stable ranking model produces +when handed a rarer-positive subset — it is not, on its own, evidence of +degraded discrimination. Lift and ROC-AUC are the metrics that control for +prevalence, and neither moves. + +**Conclusion**: within this dataset, the model's ability to rank suspicious +above non-suspicious transactions does not measurably degrade for previously +unseen accounts. The apparent collapse in PR-AUC/precision is a base-rate +artifact of the partition, not a generalization failure. + +The lower prevalence itself is worth noting as a property of SAML-D rather +than of the model: its suspicious typologies (structuring, layering, +fan-in/out rings) are built from a recurring cast of actors, so a genuinely +cold-start account is simply less likely to be labeled suspicious in this +synthetic dataset. That is a plausible property of real laundering rings too +(sustained actors, not one-off), but it means this test cannot fully separate +"the model doesn't know what to do with new accounts" from "new accounts are +inherently rarer positives here." + +## What this doesn't test + +Network features (out-degree, counterparty HHI) are computed per-transaction +from each account's own prior activity, so a first-appearance account +naturally gets a short, sparse history — a realistic cold-start, not a data +leak. What it doesn't test is whether an entire *cluster* of colluding +accounts, never seen together during training, would be caught — that needs a +connected-component holdout (partition the transaction graph so related +account clusters don't span train/test) rather than an individual-account +holdout. Left as a further extension. diff --git a/scripts/unseen_entity_evaluation.py b/scripts/unseen_entity_evaluation.py new file mode 100644 index 0000000..269ba81 --- /dev/null +++ b/scripts/unseen_entity_evaluation.py @@ -0,0 +1,160 @@ +"""Unseen-entity generalization: does performance hold when sender or +receiver was never observed during training, or is it driven by +account-history features (lifetime counts, degree, HHI) that only work +for accounts the model has already seen? + +No retraining: reuses the artifact trained on the standard temporal split +and evaluates it on three partitions of the same held-out test set. +""" + +import json +import sys +from pathlib import Path + +import joblib +import matplotlib.pyplot as plt +import numpy as np +import pandas as pd + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(PROJECT_ROOT)) + +from src.evaluation.metrics import evaluate_model, precision_recall_at_alert_rate +from src.logging_config import setup_logging +from src.models.train import MODEL_FEATURES, TARGET, temporal_split + +logger = setup_logging(__name__) + +ARTIFACT_PATH = PROJECT_ROOT / "artifacts/risk_model.joblib" +FEATURE_PATH = PROJECT_ROOT / "data/processed/transactions_features.parquet" +OUTPUT_PATH = PROJECT_ROOT / "reports/unseen_entity_results.json" +FIGURE_PATH = PROJECT_ROOT / "reports/figures/unseen_entity_generalization.png" +ALERT_RATE = 0.005 + + +def summarize(mask: np.ndarray, y_test: np.ndarray, calibrated_prob: np.ndarray, label: str) -> dict: + subset_y = y_test[mask] + subset_prob = calibrated_prob[mask] + + positives = int(subset_y.sum()) + result = { + "label": label, + "transactions": int(mask.sum()), + "positives": positives, + "prevalence": float(subset_y.mean()) if len(subset_y) else None, + "metrics": None, + } + + if len(subset_y) == 0 or positives == 0 or positives == len(subset_y): + logger.warning( + f"{label}: insufficient class variance ({positives} positives / " + f"{len(subset_y)} rows) -- metrics skipped" + ) + return result + + metrics = evaluate_model(subset_y, subset_prob) + budget = precision_recall_at_alert_rate(subset_y, subset_prob, alert_rate=ALERT_RATE) + result["metrics"] = { + "pr_auc": metrics["pr_auc"], + "roc_auc": metrics["roc_auc"], + "precision_at_alert_rate": budget["precision"], + "recall_at_alert_rate": budget["recall"], + "lift_at_alert_rate": budget["lift"], + } + return result + + +def main(): + logger.info(f"Loading artifact from {ARTIFACT_PATH}...") + artifact = joblib.load(ARTIFACT_PATH) + preprocessor = artifact["preprocessor"] + model = artifact["model"] + calibrator = artifact["calibrator"] + + logger.info(f"Loading features from {FEATURE_PATH}...") + df = pd.read_parquet(FEATURE_PATH) + + train, _, _, test = temporal_split(df) + + seen_accounts = set(train["Sender_account"]) | set(train["Receiver_account"]) + logger.info(f"Accounts observed during training: {len(seen_accounts):,}") + + sender_seen = test["Sender_account"].isin(seen_accounts).to_numpy() + receiver_seen = test["Receiver_account"].isin(seen_accounts).to_numpy() + both_seen = sender_seen & receiver_seen + unseen_entity = ~both_seen + + logger.info( + f"Test set: {len(test):,} transactions | " + f"{both_seen.sum():,} both-parties-seen ({both_seen.mean():.2%}) | " + f"{unseen_entity.sum():,} involve an unseen account ({unseen_entity.mean():.2%})" + ) + + X_test_processed = preprocessor.transform(test[MODEL_FEATURES]) + raw_prob = model.predict_proba(X_test_processed)[:, 1] + calibrated_prob = calibrator.predict(raw_prob) + y_test = test[TARGET].to_numpy() + + standard = summarize(np.ones(len(y_test), dtype=bool), y_test, calibrated_prob, "Standard out-of-time (full test set)") + seen_result = summarize(both_seen, y_test, calibrated_prob, "Both sender and receiver seen during training") + unseen_result = summarize(unseen_entity, y_test, calibrated_prob, "At least one party unseen during training") + + results = { + "alert_rate": ALERT_RATE, + "accounts_seen_in_training": len(seen_accounts), + "standard_out_of_time": standard, + "both_parties_seen": seen_result, + "unseen_entity": unseen_result, + } + + logger.info("=" * 70) + logger.info("UNSEEN-ENTITY GENERALIZATION RESULTS") + logger.info("=" * 70) + for key in ("standard_out_of_time", "both_parties_seen", "unseen_entity"): + r = results[key] + logger.info(f"\n{r['label']}:") + prevalence = f"{r['prevalence']:.4%}" if r["prevalence"] is not None else "n/a" + logger.info(f" Transactions: {r['transactions']:,} | Positives: {r['positives']:,} | Prevalence: {prevalence}") + if r["metrics"]: + for metric_key, value in r["metrics"].items(): + logger.info(f" {metric_key:30s}: {value:.6f}") + + OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True) + OUTPUT_PATH.write_text(json.dumps(results, indent=2) + "\n") + logger.info(f"\nResults written to {OUTPUT_PATH.relative_to(PROJECT_ROOT)}") + + chart_labels = { + "standard_out_of_time": "Standard\nOut-of-Time", + "both_parties_seen": "Both Parties\nSeen", + "unseen_entity": "Unseen\nEntity", + } + labels = [] + pr_aucs = [] + for key in ("standard_out_of_time", "both_parties_seen", "unseen_entity"): + r = results[key] + if r["metrics"] is None: + continue + labels.append(chart_labels[key]) + pr_aucs.append(r["metrics"]["pr_auc"]) + + if pr_aucs: + FIGURE_PATH.parent.mkdir(parents=True, exist_ok=True) + fig, ax = plt.subplots(figsize=(10, 6)) + colors = ["#06b6d4", "#10b981", "#f59e0b"] + bars = ax.bar(labels, pr_aucs, color=colors[: len(labels)], edgecolor="black", linewidth=1.5) + ax.set_ylabel("PR-AUC Score", fontsize=12, fontweight="bold") + ax.set_title("Generalization to Unseen Entities", fontsize=14, fontweight="bold", pad=20) + ax.set_ylim([0, 1.0]) + ax.grid(axis="y", alpha=0.3) + for bar in bars: + height = bar.get_height() + ax.text(bar.get_x() + bar.get_width() / 2, height, f"{height:.4f}", + ha="center", va="bottom", fontsize=10, fontweight="bold") + fig.tight_layout() + fig.savefig(FIGURE_PATH, dpi=150, bbox_inches="tight") + plt.close(fig) + logger.info(f"Saved comparison chart to {FIGURE_PATH.relative_to(PROJECT_ROOT)}") + + +if __name__ == "__main__": + main()