This article provides complete, runnable Python examples covering binary classification, regression, model persistence and SHAP explanation. The code targets sklearn 1.4+, XGBoost 2.1+, LightGBM 4.0+ and CatBoost 1.2+, and includes comments; it has been run end to end under scikit-learn 1.9.1, XGBoost 3.4.1, LightGBM 4.7.0, CatBoost 1.2.10, shap 0.54.1 and optuna 5.1.0 (Python 3.13).
Version differences to watch
- XGBoost:
early_stopping_rounds,eval_metricandcallbackswere deprecated infit()from 1.6 and removed fromfit()in 2.1, so they belong in the constructor (the code here follows that).- LightGBM: from 4.0,
fit()/train()no longer acceptearly_stopping_roundsorverbose_eval; usecallbacks=[lgb.early_stopping(...)]instead. From 4.7 the sklearn interface gainseval_X/eval_yand markseval_setdeprecated (it still works, with a warning).- scikit-learn:
AdaBoostClassifier’sSAMME.Rwas deprecated in 1.4 and removed in 1.6; thealgorithmparameter itself was deprecated in 1.6 and removed in 1.8, so passingalgorithm=...raisesTypeErrorin new versions.
11.1 A complete binary classification workflow
This example uses sklearn’s make_classification to generate data and walks through the whole path from loading to evaluation and visualization, comparing XGBoost, LightGBM and CatBoost along the way.
| |
11.2 A complete regression workflow
Regression shares most of its code structure with classification; the differences are the loss, the metrics and the plots. This example uses make_regression and adds residual analysis and a learning curve.
| |
11.3 Saving and loading models
In production a trained model has to be persisted to disk for inference, deployment or sharing. Each library offers its own formats.
| |
11.4 Explaining models with SHAP
SHAP (SHapley Additive exPlanations) is a feature-attribution method built on Shapley values from game theory; it gives each prediction the marginal contribution of each feature. XGBoost, LightGBM and CatBoost all integrate well with the SHAP library.
| |
11.5 Feature importance: types and differences
Step 7 of 11.1 drew an XGBoost importance bar chart from feature_importances_. Note that “feature importance” means different things in different libraries, and switching the importance type can reorder the features substantially.
| Library / method | Type | Meaning | Default |
|---|---|---|---|
| XGBoost | weight | Number of times the feature was used to split, across all trees | — |
| XGBoost | gain / total_gain | Average / total gain of that feature’s splits | feature_importances_ for tree models in the sklearn interface uses gain |
| XGBoost | cover / total_cover | Average / total Hessian-weighted sample coverage of that feature’s splits | — |
| LightGBM | split / gain | Times used to split / total gain contributed | the sklearn interface defaults to importance_type='split' |
| CatBoost | PredictionValuesChange and others | See the code in CatBoost | PredictionValuesChange |
| sklearn | permutation importance | Drop in the metric after shuffling one column on held-out data | model agnostic |
A few things to watch:
- Split- and gain-based importance is computed on the training data and doesn’t directly reflect a feature’s contribution to generalization; both scikit-learn’s docs and Strobl et al. (2007) note that this kind of importance favors features with many distinct values (high cardinality, continuous).
- Different types can rank features differently:
weight/splitonly count uses, which often puts features with many small splits on top, whilegainis closer to “contribution to reducing the loss.” Lundberg et al. (2020) also point out consistency problems with these global measures and recommend the mean absolute SHAP value $\frac{1}{n}\sum_{i=1}^{n}\lvert\phi_j(x_i)\rvert$ as a global importance measure (see 11.4). - Compute permutation importance on a validation or test set. When two features are highly correlated, shuffling one still leaves the model access to the information through the other, and both importances are underestimated.
| |
