[SC-17796] Update XGBoost notebook compatibility

[cachafla]

Pull Request Description

What and why?

XGBoost notebooks previously depended on whatever XGBoost version happened to be installed and loaded sample models from pickle or legacy model files. Those formats are not a stable interchange mechanism across XGBoost versions, so notebooks could fail before ValidMind tests ran.

This PR:

• installs validmind[xgboost] in notebooks that use XGBoost
• installs the explainability extra in the three XGBoost notebooks that run SHAPGlobalImportance
• replaces three pickle or legacy sample models with XGBoost JSON model files
• updates the affected notebooks to load the JSON assets
• preserves the champion model's documented early_stopping_rounds and eval_metric parameters
• preserves the existing XGBoost minimum while extending the supported range to >=1.5.2,<3.1, with scikit-learn <1.8 for the XGBoost extras
• adds CI coverage for loading the JSON assets and for the unbounded core dependency set
• refreshes uv.lock

After this change, the notebooks declare their XGBoost and SHAP dependencies and load models using XGBoost's stable JSON model format. Existing clients using supported XGBoost 1.x versions remain eligible for the xgboost and all extras. Broader model-adapter compatibility work remains tracked in SC-17796.

How to test

1. Install the XGBoost extra and test dependencies.

2. Run:

   uv lock --check
   python scripts/ensure_clean_notebooks.py
   python scripts/credentials_check.py
   python -m pytest -q tests/unit_tests/test_xgboost_notebook_assets.py
   python -m pytest -q tests/test_full_suite.py tests/test_full_suite_nb.py tests/unit_tests/model_validation/sklearn/test_ROCCurve.py tests/unit_tests/model_validation/sklearn/test_PrecisionRecallCurve.py

3. Load each converted model with xgb.XGBClassifier().load_model(...):

   • notebooks/quickstart/xgboost_model_champion.json
   • notebooks/use_cases/ongoing_monitoring/xgboost_model.json
   • notebooks/use_cases/validation/xgb_model_champion.json

Local results:

• XGBoost 1.7.6: all three JSON models loaded and 10 focused/full-suite tests passed; the runtime produced pandas deprecation warnings but no failures.
• XGBoost 2.1.4: all three JSON models loaded and 10 focused/full-suite tests passed.
• XGBoost 3.0.5: all three JSON models loaded, the new asset test passed, and both full-suite tests passed.
• A clean wheel install without extras resolved scikit-learn 1.9.0 and passed the complete unit-test entrypoint with Jinja2 installed as test bootstrap tooling.
• The champion model retained early_stopping_rounds=10 and eval_metric=["error", "logloss", "auc"] under XGBoost 1.7.6 and 3.0.5.
• Notebook schema, output, credential, lockfile, pre-commit, and pre-push checks passed.

Full notebook execution was not performed because the notebooks require live ValidMind API credentials.

What needs special review?

• Confirm preserving the existing XGBoost minimum at 1.5.2 while extending the upper range to <3.1 is the appropriate conservative compatibility policy.
• Confirm the scikit-learn <1.8 guard is appropriate for the supported pre-3.1 XGBoost range; the new core CI leg separately tests the latest unbounded core scikit-learn dependency.
• Confirm all notebooks that use XGBoost declare the required extras, including explainability where SHAP tests run.
• Confirm the JSON model migrations preserve the intended sample-model behavior and documented parameters.
• XGBoost 3.1+ SHAP compatibility, native Booster handling, multiclass probability preservation, and XGBRegressor SHAP routing are intentionally outside this PR and remain tracked in SC-17796.

Dependencies, breaking changes, and deployment notes

• No application deployment or environment-variable changes are required.
• The existing XGBoost minimum of 1.5.2 is preserved so clients using XGBoost 1.x are not forced to upgrade solely by this change.
• The upper bound expands from <3 to <3.1, adding XGBoost 3.0.x support.
• XGBoost remains capped below 3.1 because the currently resolved SHAP version cannot parse the XGBoost 3.1+ vector base_score representation.
• The XGBoost extra now constrains scikit-learn below 1.8 to avoid unsupported combinations with the allowed pre-3.1 XGBoost versions.
• The dependency workflow now includes a core-only matrix leg so current unbounded core dependencies remain covered independently of optional-extra constraints.

Release notes

XGBoost example notebooks now install their required dependencies explicitly and use portable JSON model files. Existing XGBoost 1.x compatibility is preserved while adding tested support for XGBoost 2.1.x and 3.0.x.

Checklist

• What and why
• Screenshots or videos (Frontend) — Not applicable
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied — bug
• PR linked to Shortcut — SC-17796
• Unit tests added (Backend) — JSON notebook asset loading coverage added
• Tested locally
• Documentation updated (if required) — Notebook setup and model-loading instructions updated
• Environment variable additions/changes documented (if required) — Not applicable

[juanmleng]

Checked this against XGBoost 1.7.6, 2.1.4 and 3.0.5 and the conversion holds up: all three JSON assets load, predict_proba matches the old files exactly, and feature names and types survive. I left four comments on specific lines — the SHAP extra is the one worth fixing here, the rest optional. In addition:

• Docs copies — the docs site carries its own copies of these three notebooks plus the old .pkl/.model files under site/notebooks/, and nothing syncs them, so the published tutorials will still say joblib.load after this merges. A companion docs PR would keep them in step.
• Asset framing — only quickstart/xgboost_model_champion.pkl was a real pickle; the other two were already XGBoost-native files loaded with load_model, one just misnamed .pkl.
• Description — the body links to internal Shortcut URLs, which usually stay out of this repo since it's public. The [SC-…] title prefix is the normal convention.
• Coverage — nothing in CI loads the three new assets; a short test that loads each one would have caught the scikit-learn 1.8 interaction.

[juanmleng]

Nice - thanks @cachafla ! Just left few small things to consider and one to address regarding shap dependencies.

[cachafla]

Thanks for the thorough review, Juan. I addressed the four inline items:

• Added the explainability extra to all three SHAP notebooks.
• Kept the conservative scikit-learn cap scoped to the XGBoost extras and added an unbounded core scikit-learn compatibility leg across Python 3.9–3.14.
• Restored the constructor-only XGBoost parameters before loading the JSON model.
• Normalized the notebook source indentation.

I also added tests/unit_tests/test_xgboost_notebook_assets.py so CI loads all three migrated assets, updated the PR wording to distinguish the one pickle from the legacy native-model files, and removed internal Shortcut URLs from the public PR body.

The original compatibility failures had two causes: the new core leg initially ran tests requiring optional extras, and the broader matrix had a network-dependent BLEU test that intermittently could not load the remote Hugging Face metric. The core leg now targets the relevant ROC/PR surface, and the BLEU unit test mocks only the external loader. The current compatibility run is at https://github.com/validmind/validmind-library/actions/runs/31643714195 (all six core jobs and 17/18 total jobs green; Python 3.9 max is still running with no failures).

The documentation repo copies under site/notebooks/ remain a separate companion docs change rather than being modified from this library PR.