research: Add unseen-entity generalization experiment

README.md

@@ -243,6 +243,7 @@ curl -X POST http://localhost:8000/api/predict \
 - Calibration may drift under distribution shift.
 - Risk probabilities are model estimates, not financial-loss probabilities.
 - The online feature-store layer is outside the current research scope.
+- Network features are strong (see ablation), which raises the question of whether they transfer to accounts absent from training. See [Unseen-Entity Generalization](docs/research/unseen_entity_generalization.md) for a dedicated experiment on this.
 
 ## Dataset Reference
 

docs/assets/unseen_entity_results.json

@@ -0,0 +1,43 @@
+{
+  "alert_rate": 0.005,
+  "accounts_seen_in_training": 693879,
+  "standard_out_of_time": {
+    "label": "Standard out-of-time (full test set)",
+    "transactions": 1420913,
+    "positives": 1694,
+    "prevalence": 0.0011921912179000402,
+    "metrics": {
+      "pr_auc": 0.9859286355765932,
+      "roc_auc": 0.9998338278230887,
+      "precision_at_alert_rate": 0.23673469387755103,
+      "recall_at_alert_rate": 0.9929161747343566,
+      "lift_at_alert_rate": 198.5710767896297
+    }
+  },
+  "both_parties_seen": {
+    "label": "Both sender and receiver seen during training",
+    "transactions": 716838,
+    "positives": 1569,
+    "prevalence": 0.002188779054681811,
+    "metrics": {
+      "pr_auc": 0.9917797464630843,
+      "roc_auc": 0.9998019722862765,
+      "precision_at_alert_rate": 0.4345885634588563,
+      "recall_at_alert_rate": 0.992989165073295,
+      "lift_at_alert_rate": 198.55296153774358
+    }
+  },
+  "unseen_entity": {
+    "label": "At least one party unseen during training",
+    "transactions": 704075,
+    "positives": 125,
+    "prevalence": 0.00017753790434257713,
+    "metrics": {
+      "pr_auc": 0.7912056121187283,
+      "roc_auc": 0.9997998835144541,
+      "precision_at_alert_rate": 0.03521726782164158,
+      "recall_at_alert_rate": 0.992,
+      "lift_at_alert_rate": 198.36478273217836
+    }
+  }
+}

docs/research/unseen_entity_generalization.md

@@ -0,0 +1,75 @@
+# Unseen-Entity Generalization
+
+## Question
+
+The main results show PR-AUC jumping from 0.0898 (transaction features only) to
+0.9435 once network features (lifetime counts, degree, counterparty HHI) are
+added. That's a large amount of signal riding on account history. This asks a
+narrower question than the standard out-of-time test:
+
+Does performance hold when the sender or receiver was never observed during
+training, or is it propped up by history that only exists for accounts the
+model has already seen?
+
+## Method
+
+No retraining. Reuses the artifact from the standard temporal split and
+partitions the same held-out test set (1,420,913 transactions) by whether
+both parties appeared in the training window (693,879 accounts):
+
+```bash
+python scripts/unseen_entity_evaluation.py
+```
+
+## Results
+
+| Partition | Transactions | Positives | Prevalence | PR-AUC | ROC-AUC | Recall @ 0.5% | Lift @ 0.5% |
+|---|---|---|---|---|---|---|---|
+| Standard out-of-time (full test set) | 1,420,913 | 1,694 | 0.119% | 0.9859 | 0.9998 | 99.29% | 198.6x |
+| Both parties seen in training | 716,838 | 1,569 | 0.219% | 0.9918 | 0.9998 | 99.30% | 198.6x |
+| At least one party unseen | 704,075 | 125 | 0.018% | 0.7912 | 0.9998 | 99.20% | 198.4x |
+
+![Generalization to Unseen Entities](../assets/unseen_entity_generalization.png)
+
+## Interpretation
+
+PR-AUC and precision both drop sharply for the unseen-entity partition
+(0.99 → 0.79, precision 0.43 → 0.035). Taken alone, that looks like the network
+features fail to transfer. But three other numbers move by less than noise
+across all three partitions:
+
+- **ROC-AUC**: 0.9998 in every partition — invariant to class balance.
+- **Recall @ 0.5% alert budget**: ~99.2-99.3% in every partition.
+- **Lift over base rate**: ~198x in every partition.
+
+PR-AUC's baseline is the positive prevalence itself, and the unseen-entity
+partition has an 8x lower prevalence (0.018% vs 0.219%) than the seen-pair
+partition. A drop in PR-AUC is exactly what a stable ranking model produces
+when handed a rarer-positive subset — it is not, on its own, evidence of
+degraded discrimination. Lift and ROC-AUC are the metrics that control for
+prevalence, and neither moves.
+
+**Conclusion**: within this dataset, the model's ability to rank suspicious
+above non-suspicious transactions does not measurably degrade for previously
+unseen accounts. The apparent collapse in PR-AUC/precision is a base-rate
+artifact of the partition, not a generalization failure.
+
+The lower prevalence itself is worth noting as a property of SAML-D rather
+than of the model: its suspicious typologies (structuring, layering,
+fan-in/out rings) are built from a recurring cast of actors, so a genuinely
+cold-start account is simply less likely to be labeled suspicious in this
+synthetic dataset. That is a plausible property of real laundering rings too
+(sustained actors, not one-off), but it means this test cannot fully separate
+"the model doesn't know what to do with new accounts" from "new accounts are
+inherently rarer positives here."
+
+## What this doesn't test
+
+Network features (out-degree, counterparty HHI) are computed per-transaction
+from each account's own prior activity, so a first-appearance account
+naturally gets a short, sparse history — a realistic cold-start, not a data
+leak. What it doesn't test is whether an entire *cluster* of colluding
+accounts, never seen together during training, would be caught — that needs a
+connected-component holdout (partition the transaction graph so related
+account clusters don't span train/test) rather than an individual-account
+holdout. Left as a further extension.

scripts/unseen_entity_evaluation.py

@@ -0,0 +1,160 @@
+"""Unseen-entity generalization: does performance hold when sender or
+receiver was never observed during training, or is it driven by
+account-history features (lifetime counts, degree, HHI) that only work
+for accounts the model has already seen?
+
+No retraining: reuses the artifact trained on the standard temporal split
+and evaluates it on three partitions of the same held-out test set.
+"""
+
+import json
+import sys
+from pathlib import Path
+
+import joblib
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+
+PROJECT_ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(PROJECT_ROOT))
+
+from src.evaluation.metrics import evaluate_model, precision_recall_at_alert_rate
+from src.logging_config import setup_logging
+from src.models.train import MODEL_FEATURES, TARGET, temporal_split
+
+logger = setup_logging(__name__)
+
+ARTIFACT_PATH = PROJECT_ROOT / "artifacts/risk_model.joblib"
+FEATURE_PATH = PROJECT_ROOT / "data/processed/transactions_features.parquet"
+OUTPUT_PATH = PROJECT_ROOT / "reports/unseen_entity_results.json"
+FIGURE_PATH = PROJECT_ROOT / "reports/figures/unseen_entity_generalization.png"
+ALERT_RATE = 0.005
+
+
+def summarize(mask: np.ndarray, y_test: np.ndarray, calibrated_prob: np.ndarray, label: str) -> dict:
+    subset_y = y_test[mask]
+    subset_prob = calibrated_prob[mask]
+
+    positives = int(subset_y.sum())
+    result = {
+        "label": label,
+        "transactions": int(mask.sum()),
+        "positives": positives,
+        "prevalence": float(subset_y.mean()) if len(subset_y) else None,
+        "metrics": None,
+    }
+
+    if len(subset_y) == 0 or positives == 0 or positives == len(subset_y):
+        logger.warning(
+            f"{label}: insufficient class variance ({positives} positives / "
+            f"{len(subset_y)} rows) -- metrics skipped"
+        )
+        return result
+
+    metrics = evaluate_model(subset_y, subset_prob)
+    budget = precision_recall_at_alert_rate(subset_y, subset_prob, alert_rate=ALERT_RATE)
+    result["metrics"] = {
+        "pr_auc": metrics["pr_auc"],
+        "roc_auc": metrics["roc_auc"],
+        "precision_at_alert_rate": budget["precision"],
+        "recall_at_alert_rate": budget["recall"],
+        "lift_at_alert_rate": budget["lift"],
+    }
+    return result
+
+
+def main():
+    logger.info(f"Loading artifact from {ARTIFACT_PATH}...")
+    artifact = joblib.load(ARTIFACT_PATH)
+    preprocessor = artifact["preprocessor"]
+    model = artifact["model"]
+    calibrator = artifact["calibrator"]
+
+    logger.info(f"Loading features from {FEATURE_PATH}...")
+    df = pd.read_parquet(FEATURE_PATH)
+
+    train, _, _, test = temporal_split(df)
+
+    seen_accounts = set(train["Sender_account"]) | set(train["Receiver_account"])
+    logger.info(f"Accounts observed during training: {len(seen_accounts):,}")
+
+    sender_seen = test["Sender_account"].isin(seen_accounts).to_numpy()
+    receiver_seen = test["Receiver_account"].isin(seen_accounts).to_numpy()
+    both_seen = sender_seen & receiver_seen
+    unseen_entity = ~both_seen
+
+    logger.info(
+        f"Test set: {len(test):,} transactions | "
+        f"{both_seen.sum():,} both-parties-seen ({both_seen.mean():.2%}) | "
+        f"{unseen_entity.sum():,} involve an unseen account ({unseen_entity.mean():.2%})"
+    )
+
+    X_test_processed = preprocessor.transform(test[MODEL_FEATURES])
+    raw_prob = model.predict_proba(X_test_processed)[:, 1]
+    calibrated_prob = calibrator.predict(raw_prob)
+    y_test = test[TARGET].to_numpy()
+
+    standard = summarize(np.ones(len(y_test), dtype=bool), y_test, calibrated_prob, "Standard out-of-time (full test set)")
+    seen_result = summarize(both_seen, y_test, calibrated_prob, "Both sender and receiver seen during training")
+    unseen_result = summarize(unseen_entity, y_test, calibrated_prob, "At least one party unseen during training")
+
+    results = {
+        "alert_rate": ALERT_RATE,
+        "accounts_seen_in_training": len(seen_accounts),
+        "standard_out_of_time": standard,
+        "both_parties_seen": seen_result,
+        "unseen_entity": unseen_result,
+    }
+
+    logger.info("=" * 70)
+    logger.info("UNSEEN-ENTITY GENERALIZATION RESULTS")
+    logger.info("=" * 70)
+    for key in ("standard_out_of_time", "both_parties_seen", "unseen_entity"):
+        r = results[key]
+        logger.info(f"\n{r['label']}:")
+        prevalence = f"{r['prevalence']:.4%}" if r["prevalence"] is not None else "n/a"
+        logger.info(f"  Transactions: {r['transactions']:,} | Positives: {r['positives']:,} | Prevalence: {prevalence}")
+        if r["metrics"]:
+            for metric_key, value in r["metrics"].items():
+                logger.info(f"  {metric_key:30s}: {value:.6f}")
+
+    OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True)
+    OUTPUT_PATH.write_text(json.dumps(results, indent=2) + "\n")
+    logger.info(f"\nResults written to {OUTPUT_PATH.relative_to(PROJECT_ROOT)}")
+
+    chart_labels = {
+        "standard_out_of_time": "Standard\nOut-of-Time",
+        "both_parties_seen": "Both Parties\nSeen",
+        "unseen_entity": "Unseen\nEntity",
+    }
+    labels = []
+    pr_aucs = []
+    for key in ("standard_out_of_time", "both_parties_seen", "unseen_entity"):
+        r = results[key]
+        if r["metrics"] is None:
+            continue
+        labels.append(chart_labels[key])
+        pr_aucs.append(r["metrics"]["pr_auc"])
+
+    if pr_aucs:
+        FIGURE_PATH.parent.mkdir(parents=True, exist_ok=True)
+        fig, ax = plt.subplots(figsize=(10, 6))
+        colors = ["#06b6d4", "#10b981", "#f59e0b"]
+        bars = ax.bar(labels, pr_aucs, color=colors[: len(labels)], edgecolor="black", linewidth=1.5)
+        ax.set_ylabel("PR-AUC Score", fontsize=12, fontweight="bold")
+        ax.set_title("Generalization to Unseen Entities", fontsize=14, fontweight="bold", pad=20)
+        ax.set_ylim([0, 1.0])
+        ax.grid(axis="y", alpha=0.3)
+        for bar in bars:
+            height = bar.get_height()
+            ax.text(bar.get_x() + bar.get_width() / 2, height, f"{height:.4f}",
+                    ha="center", va="bottom", fontsize=10, fontweight="bold")
+        fig.tight_layout()
+        fig.savefig(FIGURE_PATH, dpi=150, bbox_inches="tight")
+        plt.close(fig)
+        logger.info(f"Saved comparison chart to {FIGURE_PATH.relative_to(PROJECT_ROOT)}")
+
+
+if __name__ == "__main__":
+    main()