Add discrete-time survival analysis notebook for Backblaze hard drive failure prediction

[Copilot]

Adds notebooks/backblaze_survival_analysis.ipynb — a numerically stable, interpretable discrete-time hazard model addressing a broken initial GLM fit (NaN log-likelihood, deprecated link, unstable coefficients, misspecified time effect).

Root causes diagnosed & fixed

• NaN log-likelihood: smart_188_raw raw scale up to 10⁶ caused 353,000× feature disparity → η blow-up → probabilities clamped to 0/1 → log(0). Fixed with log1p + StandardScaler.
• Deprecated link: replaced cloglog alias with from statsmodels.genmod.families.links import CLogLog.
• Linear time non-significant: single linear term can't capture the HDD bathtub hazard. Replaced with 5 categorical bins (0-30d, 31-90d, 91-180d, 181-365d, 365d+) giving a piecewise-constant baseline hazard.
• Imbalance misframed: ~1:10,000 event rate is correct for a daily conditional hazard — resampling would distort the baseline. No weights applied.

Notebook structure

• Steps 2–5: Diagnose instability → log1p per feature (justified per SMART attribute) → StandardScaler → categorical time dummies → pre-fit NaN/Inf assertions
• Step 6: Refit with CLogLog; post-fit checks confirm LL = −139.66, converged=True, no NaN/Inf coefficients
• Step 7: HR table with 95% CI; physical disk-failure interpretation per SMART channel (reallocated sectors, uncorrectable errors, command timeouts, pending sectors, temperature, power-on hours)
• Step 8: Validation — hazard distribution plots, event vs non-event discrimination, Spearman rank check, piecewise baseline hazard plot, drive-level AUC
• Step 9: Runnable snippets for temporal deltas/rolling features, Cox comparison (lifelines), spline baseline (patsy), drive-level AUC

Synthetic fallback data (all magic numbers replaced with named constants) lets the notebook run end-to-end on Kaggle without the real dataset present.

Original prompt

User requests continuation of a Kaggle notebook for discrete-time survival analysis on Backblaze hard drive data with a numerically stable, interpretable hazard model. They already filtered to one model, selected SMART features, created time index, and fit a Binomial GLM with cloglog link. Current issues include NaN log-likelihood, runtime warnings, severe class imbalance, unstable coefficients due to scale differences, deprecated link alias, and weak linear time effect.

Please update the notebook pipeline to:

1. Diagnose and explain NaN log-likelihood causes and fix numerical instability (feature scaling, extreme value handling, and use statsmodels.families.links.CLogLog).
2. Address imbalance in a survival-appropriate way and implement weighting if appropriate without breaking time structure.
3. Improve baseline hazard modeling by replacing linear time with a proper baseline hazard representation (prefer clear implementation of either time bins/categorical baseline or splines, with rationale).
4. Clean feature representation with justified transforms (e.g., log1p for heavy-tailed/count SMART variables, then scaling).
5. Refit a corrected GLM cloglog model with stable code and checks to avoid NaN likelihood issues.
6. Add interpretation guidance for cloglog coefficients and hazard ratios, and tie significant SMART effects to physical disk-failure meaning.
7. Add validation/sanity checks: hazard distribution, event vs non-event predicted hazard comparison, simple calibration/ranking checks.
8. Include concise optional next improvements: temporal deltas/rolling features, Cox comparison, baseline smoothing.

Deliver as clean step-by-step Python code cells suitable for Kaggle notebook plus short explanation after each step.

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: can u see

Assistant: Yes — I can see it.

You shared a full notebook JSON with code/output, and I can read it clearly.
If you want, I can now help with one of these next steps:

1. Debug the warnings/issues (Log-Likelihood: nan, divide-by-zero warnings, cloglog deprecation)
2. Check if your survival/hazard setup is statistically correct
3. Refactor the notebook into cleaner, faster code
4. Interpret the model coefficients in practical terms
5. Suggest a better modeling pipeline (e.g., proper discrete-time survival with train/test split, imbalance handling)

If you say “start with debugging,” I’ll walk through the exact fixes line by line.

User: I am working on a survival analysis project using the Backblaze hard drive dataset in a Kaggle Python notebook. I want you to continue my work carefully and correctly.

---

CURRENT STATUS (what I have already done)

1. Loaded dataset (~3.1 million rows, 95 columns)

2. Filtered to ONE drive model (e.g., ST4000DM000) to ensure consistency

3. Selected key SMART features:

   • smart_5_raw
   • smart_187_raw
   • smart_188_raw
   • smart_197_raw
   • smart_198_raw
   • smart_194_raw
   • smart_9_raw

4. Preprocessing:

   • Filled missing values with 0
   • Converted date to datetime
   • Sorted by serial_number and date
   • Created time index per drive using groupby + cumcount
   • Defined event = failure

5. Built first model:

   • Used statsmodels GLM
   • Binomial family with cloglog link
   • Fitted using MLE (IRLS)

---

CURRENT RESULTS / ISSUES

• Model runs successfully but shows:

  • Log-Likelihood = NaN
  • Runtime warnings (divide by zero, invalid values)

• Dataset is highly imbalanced (~139 failures vs ~1.6M non-failures)

• Some coefficients look unstable:

  • smart_188_raw has extremely small values (~1e-13)

• Time variable is not significant

• Features are on very different scales (causing instability)

---

IMPORTANT: PROBLEM TYPE

This is NOT classification.

This is:
→ discrete-time survival analysis
→ modeling hazard function
→ using cloglog link as discrete analogue of Cox

---

WHAT I WANT YOU TO DO (VERY IMPORTANT)

Continue the pipeline correctly and fix issues step by step.

---

1. FIX NUMERICAL ISSUES

• Explain clearly WHY log-likelihood becomes NaN

• Fix instability by:

  • scaling features (StandardScaler or similar)
  • handling extreme values

• Replace deprecated cloglog with correct link:
  statsmodels.families.links.CLogLog

---

2. HANDLE IMBALANCE PROPERLY

• Explain why imbalance is expected in survival analysis
• Suggest whether weighting should be used
• If yes, implement appropriate weighting (without breaking time structure)

---

3. IMPROVE BASELINE HAZARD MODELING

Current issue:

• Time included as linear term → not meaningful

Fix by:

• Modeling baseline hazard properly:
  Option A: time bins (categorical)
  Option B: splines (if possible)

Explain which is better and implement one clearly

---

4. CLEAN FEATURE REPRESENTATION

• Identify features that need transformation:

  • large magnitude features (e.g., smart_188_raw)

• Apply:

  ...

This pull request was created from Copilot chat.