Boosting performance depends heavily on hyperparameters. Unlike random forests, which work out of the box, boosting’s default settings tend to be aggressive (XGBoost’s max_depth=6, eta=0.3), and getting the best out of it takes systematic tuning. This article lays out a general staged tuning framework, gives concrete parameter advice for each algorithm, and offers fixes for the common traps.
10.1 A general strategy: staged tuning
Boosting has many parameters that interact (the learning_rate/n_estimators trade-off, for one). A blind full grid search is both expensive and prone to landing in a local optimum. Staged tuning — layering the work by importance, refining step by step — works better.
flowchart LR
subgraph Stage1["Stage 1: control model complexity"]
A1["max_depth / num_leaves"] --> B1["min_child_weight / min_data_in_leaf"]
B1 --> C1["gamma / min_split_gain"]
end
subgraph Stage2["Stage 2: sampling and regularization"]
A2["subsample / bagging_fraction"] --> B2["colsample_bytree / feature_fraction"]
B2 --> C2["lambda / l2_leaf_reg"]
C2 --> D2["alpha / lambda_l1"]
end
subgraph Stage3["Stage 3: learning rate and iterations"]
A3["learning_rate"] --> B3["n_estimators"]
B3 --> C3["early_stopping"]
end
Stage1 --> Stage2 --> Stage3
The logic of staged tuning
Stage 1: control the complexity of a single tree
Boosting is a weighted combination of weak learners. If a single weak learner is too complex (a very deep tree, say), the model overfits even with few rounds. So the first step is to limit how much one tree can express:
max_depth(XGBoost, CatBoost) ornum_leaves(LightGBM): cap depth or leaf count. Rules of thumb:- Shallow (
max_depth=3~5): suits noisy data with simple feature interactions; generalizes well - Medium (
max_depth=6~8): suits moderately complex interactions; the range competitions tend to use - Deep (
max_depth=10+): only for large, low-noise data with complex interactions, and only with strong regularization
- Shallow (
min_child_weight(XGBoost: a lower bound on the sum of second derivatives in a child, default 1) ormin_data_in_leaf(LightGBM: minimum samples per leaf, default 20): raising it makes the model more conservative and stops leaves from overfitting a handful of samples. Under squared loss the Hessian is 1, somin_child_weightis just a minimum sample count; under log loss the Hessian is $p(1-p)$, so samples the model is confident about ($p$ near 0 or 1) contribute less. Typical values:min_child_weight=1~10for classification,3~30for regression.gamma/min_split_gain: the minimum loss reduction a split must deliver. Only splits whose gain exceeds it are allowed. Larger means simpler trees. Typical range: 0 to 5.
Stage 2: add randomness to cut variance
Having bounded single-tree complexity, sampling and regularization bring down the ensemble’s variance further:
subsample(row sampling): each round trains the current tree on a random subset of samples. Typical 0.6-0.9. Smaller means more randomness and faster training, but possibly more iterations to converge.colsample_bytree/colsample_bylevel(column sampling): randomly select some features per tree or per level. This decorrelates trees much as random forests do and is especially effective on high-dimensional data. Typical 0.6-1.0. Note XGBoost’scolsample_by*settings compound: each level samples from the features this tree already selected, so withcolsample_bytree=0.8, colsample_bylevel=0.8each level effectively sees about 64% of the features.lambda/reg_lambda(L2): penalizes squared leaf weights, keeping any single leaf output from growing too large. XGBoost defaults to 1, LightGBM’slambda_l2to 0, CatBoost’sl2_leaf_regto 3; raising it smooths the model. Typical 0.1-10.alpha/reg_alpha(L1): penalizes leaf weights in absolute value, shrinking some to exactly 0 (Lasso-like). XGBoost and LightGBM default to 0; CatBoost has no matching leaf-L1 parameter. Typical 0-1. Note this sparsifies leaf outputs, which isn’t the same as feature selection.
Stage 3: tune learning rate and iteration count together
learning_rate ($\eta$) and n_estimators trade off:
where $\eta$ is learning_rate (written as the shrinkage coefficient $\nu$ in Chapter 5) and $M$ is n_estimators.
This is a rough empirical relation, not an exact conservation law. It rests on Friedman’s (2001) observation that a smaller learning rate needs more iterations but usually yields lower test error; ESL §10.12 accordingly recommends a small rate (say $\eta < 0.1$) with early stopping to pick $M$. Concrete guidance (a starting point only — let early stopping decide the actual count):
| learning_rate | Suggested n_estimators | When |
|---|---|---|
| 0.3 | 50-100 | Quick experiments, large datasets |
| 0.1 | 100-500 | Standard starting point |
| 0.05 | 500-1000 | Chasing higher accuracy |
| 0.01 | 1000-5000 | Fine tuning, small datasets |
| 0.001 | 5000+ | Very fine optimization, complex interactions |
In practice, fix learning_rate=0.1, use early stopping to find a suitable n_estimators, then lower the rate and raise n_estimators to the equivalent count early stopping gave, and repeat until performance plateaus.
10.2 Tuning AdaBoost
sklearn’s AdaBoostClassifier has few parameters, but there’s still room.
| Parameter | Meaning | Range | Advice |
|---|---|---|---|
n_estimators | Number of weak classifiers | 50-1000 | Default 50; tune with learning_rate. sklearn’s AdaBoost has no built-in early stopping, so evaluate round by round on a validation set with staged_predict / staged_score and pick the best count |
learning_rate | Shrinkage on each weak classifier’s weight | 0.01-1.0 | The default 1.0 usually works; if overfitting, drop to 0.1-0.5 and raise n_estimators |
estimator | Base learner | Decision stumps / trees | Default DecisionTreeClassifier(max_depth=1); for complex data try max_depth=2~3 |
algorithm | Algorithm type | Removed | SAMME.R deprecated in 1.4 and removed in 1.6; the algorithm parameter deprecated in 1.6 and removed in 1.8. Newer versions use SAMME exclusively and raise an error if you pass this argument |
AdaBoost-specific notes:
Base-learner depth: the theory only requires weak learners slightly better than chance, so stumps (
max_depth=1) are the default and the common choice. But a stump can only express one feature’s main effect and can’t capture interactions; ESL §10.11 notes that for boosting a leaf count $J$ between 4 and 8 usually works well. Somax_depth=2~3is also worth trying via cross-validation, while very deep trees make each round fit too strongly and soak up noise.The learning_rate / n_estimators trade-off: sklearn’s docs state it plainly — lowering
learning_rateusually requires moren_estimatorsto reach a similar level of fitting. There’s no fixed product relation; settle it withstaged_scoreor cross-validation.Noisy data: AdaBoost is very sensitive to noise and outliers. With heavy noise, don’t spend a lot of time tuning it; switch to a gradient boosting method that handles noise better (XGBoost with L1/L2 regularization, say).
10.3 Tuning GBM
sklearn’s GradientBoostingClassifier/GradientBoostingRegressor are richly parameterized, with plenty of room to tune.
| Parameter | Meaning | Range | Advice |
|---|---|---|---|
loss | Loss | Classification: ’log_loss’, ’exponential' | Default ’log_loss’; ’exponential’ reproduces AdaBoost |
| Regression: ‘squared_error’, ‘absolute_error’, ‘huber’, ‘quantile’ | ‘huber’ for noisy data; ‘quantile’ for prediction intervals | ||
learning_rate | Learning rate (shrinkage) | 0.01-0.3 | Start at 0.1; drop to 0.05 for the final fit |
n_estimators | Number of trees | 100-2000 | Tune with learning_rate and early stopping |
subsample | Subsample fraction | 0.5-1.0 | 0.8 is a general starting point; 0.5-0.7 cuts variance noticeably |
max_depth | Maximum depth | 3-8 | One of the most important parameters, default 3; 3-6 is often best, confirmed by cross-validation |
max_features | Feature sampling per tree | ‘sqrt’, ’log2’, 0.5-1.0 | ‘sqrt’ for high-dimensional data; 1.0 for low-dimensional |
min_samples_split | Minimum samples to split | 2-50 | Default 2; raising it guards against overfitting |
min_samples_leaf | Minimum samples per leaf | 1-50 | Default 1; 5+ for regression, 1-5 for classification |
n_iter_no_change | Early-stopping patience | 5-50 | Default None (no early stopping); setting it holds out a validation set of size validation_fraction (default 0.1) from the training data |
Suggested order for GBM:
- Fix
learning_rate=0.1andn_estimators=500(generously large), setn_iter_no_changefor early stopping, and find the best iteration count (check then_estimators_attribute afterwards). - Search
max_depth=[3, 4, 5, 6], the most influential parameter. - Around the best
max_depth, adjustmin_samples_leafandmin_samples_split. - Introduce
subsample=0.8andmax_features='sqrt'(if there are more than 30 features). - Lower
learning_rateto 0.05 or 0.01, scalen_estimatorsup accordingly, and early stop again.
Notes on HistGradientBoosting:
HistGradientBoostingClassifier / HistGradientBoostingRegressor arrived as experimental in sklearn 0.21 and became a stable API in 1.0 (no more enable_hist_gradient_boosting). The docs note it is much faster than GradientBoostingClassifier once you have thousands of samples ($n \ge 10\,000$), so it should be the first choice on large datasets. Its parameters differ somewhat from classic GBM:
max_iter(default 100) instead ofn_estimators- Both
max_leaf_nodes(default 31) andmax_depth(defaultNone, unlimited); complexity is controlled by leaf count by default, much like LightGBM’snum_leaves - Regularization via
l2_regularization(default 0) andmin_samples_leaf(default 20) max_binsdefaults to 255 (also the maximum), with one bin reserved for missing values- Native missing-value support (NaN): each split learns whether missing values go left or right
- Native categorical support (
categorical_features, defaulting to'from_dtype'in recent versions, which detects category columns in a DataFrame) - Built-in early stopping: with
early_stopping='auto'it’s enabled only when there are more than 10,000 samples (or, from 1.7, whenX_val/y_valare passed tofit) - Supports
monotonic_cst(monotonic constraints),interaction_cst(interaction constraints) andclass_weight(1.2+)
10.4 Tuning XGBoost
XGBoost has the most elaborate parameter set, but staged tuning handles it systematically.
| Parameter | Alias | Meaning | Range | Advice |
|---|---|---|---|---|
eta | learning_rate | Learning rate | 0.01-0.3 | Start at 0.1; 0.01-0.05 for the final fit |
max_depth | - | Maximum depth | 3-10 | The key parameter; start at 5-6; 3-4 for small or noisy data |
min_child_weight | - | Minimum Hessian sum in a child | 1-30 | Default 1; 1-5 for classification, 3-10 for regression; larger is more conservative |
gamma | min_split_loss | Minimum split gain | 0-10 | Default 0, meaning any positive-gain split is allowed; raise it to fight overfitting |
lambda | reg_lambda | L2 coefficient | 0.1-10 | Default 1; larger smooths the model |
alpha | reg_alpha | L1 coefficient | 0-1 | Default 0; 0.1-0.5 when sparsity is wanted |
subsample | - | Row sampling fraction | 0.5-1.0 | Default 1; 0.8 is a general start; 0.6 on large data |
colsample_bytree | - | Column sampling per tree | 0.5-1.0 | Default 1; start at 0.8; 0.6-0.8 on high-dimensional data |
colsample_bylevel | - | Column sampling per level | 0.5-1.0 | Default 1; samples from this tree’s chosen features, multiplying with colsample_bytree |
scale_pos_weight | - | Multiplier on positive-class weight | 1-99 | Default 1; for class imbalance the docs suggest sum(negative) / sum(positive) (see trap 4) |
Staged tuning in practice:
Step 1 — a first quick search: fix learning_rate=0.1 and n_estimators=1000, set early_stopping_rounds=50 in the constructor (moved out of fit() in XGBoost 2.1), and find the best iteration count. Search combinations of max_depth=[3, 5, 7] and min_child_weight=[1, 3, 5]. This step establishes the rough complexity range.
Step 2 — refine regularization: around the best max_depth and min_child_weight from step 1, search gamma=[0, 0.1, 0.5, 1, 2], lambda=[0.1, 1, 3] and alpha=[0, 0.1, 0.5]. gamma tends to pay off most when the data are noisy.
Step 3 — tune sampling: search subsample=[0.6, 0.8, 1.0] and colsample_bytree=[0.6, 0.8, 1.0]. Note that subsample < 1.0 lowers variance but slows convergence, so you may need more iterations.
Step 4 — lower the rate and refine: drop learning_rate to 0.05 or 0.01, raise n_estimators following learning_rate * n_estimators ≈ 0.1 * n_estimators_initial, and early stop for the new best count. Repeat the local refinements from steps 2-3.
XGBoost-specific tips:
tree_method='hist'(the default since 2.0): histogram methods are far faster than exact greedy with almost no accuracy cost on large data. For GPU training usedevice='cuda'withtree_method='hist'; the oldtree_method='gpu_hist'was deprecated in 2.0 and removed in 3.1.grow_policy='lossguide': enables LightGBM-style leaf-wise growth (hist/approxonly), usually paired withmax_leaves; it can win accuracy on some datasets but demands stricter regularization.max_bin: histogram bin count, default 256. Raising it to 512 or 1024 can help on data with complex feature distributions, at more memory and compute.
10.5 Tuning LightGBM
Leaf-wise growth makes LightGBM’s parameter set markedly different from XGBoost’s: num_leaves replaces max_depth as the core complexity control.
| Parameter | Meaning | Range | Advice |
|---|---|---|---|
num_leaves | Maximum leaves | 15-150 | The key parameter, default 31; controls model complexity. With max_depth = $d$, set it well below $2^d$ (the official example: 70-80 when $d=7$) |
max_depth | Maximum depth | -1 (unlimited) to 12 | Works with num_leaves; -1 means only num_leaves binds; set it explicitly (6-10, say) |
learning_rate | Learning rate | 0.01-0.3 | Start at 0.1; 0.05-0.01 for refinement |
n_estimators / num_boost_round | Iterations | 100-5000 | Use with early stopping |
feature_fraction | Feature sampling fraction | 0.5-1.0 | Fraction of features sampled per tree; 0.8 is a general start |
bagging_fraction | Data sampling fraction | 0.5-1.0 | Alias subsample; only takes effect with bagging_freq > 0 |
bagging_freq | Sampling frequency | 1-10 | Default 0 (no sampling); resample every bagging_freq rounds; 1 means every round |
min_data_in_leaf | Minimum samples per leaf | 5-100 | Default 20 (alias min_child_samples); the key guard against leaf-wise overfitting |
lambda_l1 | L1 regularization | 0-1 | Default 0 (alias reg_alpha); L1 penalty on leaf weights |
lambda_l2 | L2 regularization | 0-10 | Default 0 (alias reg_lambda); 0.1-1 usually suffices |
min_gain_to_split | Minimum split gain | 0-5 | Corresponds to gamma; default 0 (alias min_split_gain) |
cat_smooth | Categorical smoothing | 1-100 | Default 10; dampens noise in categorical features, especially useful for rare categories |
cat_l2 | L2 on categorical features | 1-100 | Default 10; L2 regularization within categorical splits |
LightGBM-specific notes:
$$ L_{\max} \approx 0.6 \times 2^d $$num_leavesandmax_depthtogether: under leaf-wise growth,num_leavesis the main complexity control. But constraining onlynum_leavescan produce unbalanced trees with some branches very deep and others very shallow. A rough relation:where $L_{\max}$ is
num_leavesand $d$ ismax_depth. This is only a heuristic, grounded in LightGBM’s official tuning example: atmax_depth=7a depth-wise tree already performs well, and settingnum_leavesto $2^7=127$ may overfit, while 70-80 often beats depth-wise. If you setnum_leaveswithout limitingmax_depth, the model can overfit along some paths.Declaring categorical features correctly: pandas DataFrame columns with dtype
categoryare detected automatically under the defaultcategorical_feature='auto'; numpy input needs explicit column indices viacategorical_feature. Category values are cast to int32 (pandas categories use their integer codes), and negative values count as missing, so integer codes should be non-negative. LightGBM partitions categories specifically (one-vs-other when there are at mostmax_cat_to_onehotcategories, default 4), which usually beats one-hot encoding.GOSS parameters (
top_rate,other_rate): gradient-based one-side sampling keeps the samples with the largest absolute gradients (fractiontop_rate, default 0.2) and draws randomly from the rest (fractionother_rate, default 0.1). GOSS is off by default: from 4.0, setdata_sample_strategy='goss'(the oldboosting_type='goss'still runs but is no longer an officialboostingoption). It speeds training further at a slight accuracy cost.force_row_wiseandforce_col_wise: these control how histograms are built on CPU (by row or by column). Both default tofalse, in which case LightGBM tries both and picks the faster. The docs advise col-wise when there are many features, many total bins, many threads (over 20) or memory is tight; row-wise when there are many samples but few bins and few threads (16 or fewer) — though it doubles the Dataset’s memory. Setting it manually skips the test.
10.6 Tuning CatBoost
CatBoost is known for strong defaults, but understanding the parameters still helps in specific situations.
| Parameter | Meaning | Range | Advice |
|---|---|---|---|
iterations | Iterations | 100-5000 | Default 1000 (alias n_estimators); use with early stopping |
depth | Tree depth | 4-10 | Default 6; a symmetric tree doubles its leaves per extra level ($2^{d}$ leaves), so training time grows noticeably with depth |
learning_rate | Learning rate | 0.01-0.3 | With Logloss, MultiClass or RMSE and no leaf_estimation_iterations, leaf_estimation_method or l2_leaf_reg set, chosen from the dataset and iteration count; otherwise 0.03 |
l2_leaf_reg | L2 on leaves | 1-10 | Default 3; raise it against overfitting |
bagging_temperature | Bayesian bootstrap temperature | 0-10 | Default 1; 0 means all sample weights are 1; only applies with bootstrap_type='Bayesian'. On CPU the default for non-multiclass tasks is MVS (subsample=0.8), where you tune subsample instead |
border_count | Bins for numeric features | 32-255 | Alias max_bin; 254 on CPU, 128 on GPU; more is more precise but slower |
random_strength | Randomness in split scores | 0-10 | Default 1 (CPU only); adds zero-mean Gaussian noise to each candidate’s score with variance decaying over training. The parameter multiplies that variance and fights overfitting; the mechanism differs from column sampling |
grow_policy | Growth policy | ‘SymmetricTree’, ‘Depthwise’, ‘Lossguide’ | Default ‘SymmetricTree’; try ‘Lossguide’ for a LightGBM-like behavior |
leaf_estimation_method | Leaf value estimation | ‘Newton’, ‘Gradient’, ‘Exact’ | Default depends on the task: Newton (10 iterations) for classification, Newton (1) for multiclass, Gradient (1) for regression, Exact for Quantile/MAE regression |
cat_features | Categorical features | Column indices or names | Must be declared explicitly; the heart of CatBoost’s advantage |
CatBoost-specific advice:
Start with the defaults: CatBoost’s defaults already perform well on many datasets. Establish a baseline with them, and tune only if results fall short.
Only a few parameters usually need attention: typically
depth(4-8),l2_leaf_reg(1-10) andlearning_rate(withiterations). Adjustrandom_strengthandbagging_temperaturewhen overfitting.Ordered boosting is controlled by
boosting_type(values'Ordered'or'Plain'; there is no parameter namedordered_boostingin CatBoost). The default depends on device and data:'Plain'on CPU; on GPU'Ordered'when there are at most 50,000 samples and the task isn’t multiclass, otherwise'Plain'. The docs say'Ordered'usually gives better quality on small datasets but is slower than'Plain', so small datasets are where it’s worth tryingboosting_type='Ordered'explicitly.GPU training: CatBoost 1.2+ has a substantially faster GPU implementation, especially on large datasets and at greater depths. Just set
task_type='GPU'anddevices='0'.
10.7 Using early stopping
Early stopping is one of the most effective ways to keep boosting from overfitting. The idea: monitor validation loss during training, stop when it hasn’t improved for some number of rounds, and use the round with the lowest validation loss as the final model (whether each library rolls back to the best round automatically is in the table below).
The principle:
Because boosting trains iteratively, training loss falls almost monotonically, while validation loss typically follows a U shape — down and then up. Early stopping halts at the minimum of validation loss and skips the overfitting that later rounds would bring.
$$ t^{*} = \mathop{\arg\min}_{1 \le t \le T} L_{\text{val}}(t), \qquad \text{stop at round } T \text{ if } T - t^{*} \ge p $$where $L_{\text{val}}(t)$ is the loss of the model after $t$ rounds on the validation set and $p$ is the patience. In words: stop when the lowest validation loss hasn’t been beaten in the last $p$ rounds, and take the first $t^{*}$ rounds as the final model.
Early stopping across libraries:
| Library | Parameter | Typical setting | Notes |
|---|---|---|---|
| sklearn GBM | n_iter_no_change + validation_fraction | n_iter_no_change=10, validation_fraction=0.1 | n_iter_no_change defaults to None (no early stopping); the validation split comes from the training data; tol (default 1e-4) sets the improvement threshold; the actual round count is in n_estimators_ |
| sklearn HistGBM | early_stopping + n_iter_no_change | early_stopping='auto', n_iter_no_change=10 | 'auto' enables it only above 10,000 samples (or from 1.7 when X_val/y_val are passed to fit); by default it carves out validation_fraction=0.1 with scoring='loss' |
| XGBoost | early_stopping_rounds (constructor) | early_stopping_rounds=50 | Deprecated in fit() from 1.6 and removed from fit() in 2.1; pass validation data via fit(eval_set=...), using the last entry when there are several; best_iteration records the best round, and the sklearn interface’s predict uses only up to it by default |
| LightGBM | the lgb.early_stopping(50) callback | callbacks=[lgb.early_stopping(50)] | Since 4.0, fit() / train() no longer accept early_stopping_rounds; you can also set early_stopping_round in the parameters (aliases early_stopping_rounds, n_iter_no_change); min_delta sets the minimum improvement; predict uses best_iteration_ by default |
| CatBoost | early_stopping_rounds | early_stopping_rounds=50 | Equivalent to the overfitting detector with od_type='Iter', od_wait=50 (or od_type='IncToDec' with od_pval); with eval_set passed, use_best_model defaults to True and the model keeps only up to the best round |
Best practices:
The validation set must be separate from both training data and the final test set: a set used for early stopping has taken part in model selection, so it can’t report final performance. Under cross-validation, early stop separately inside each fold.
Setting patience:
- 50-100 rounds is common; with a large learning rate (0.1-0.3), 10-30 often suffices
- The smaller the rate, the smaller each round’s improvement, so patience should grow; the smaller and noisier the validation set, the larger it should be, to avoid stopping too soon
- For implementations that roll back to the best round (XGBoost’s sklearn
predict, LightGBM, CatBoost’suse_best_model), excess patience mostly costs compute — the final model won’t include trees after the best round
Early stopping and
n_estimators: setn_estimatorsgenerously (5000, say) and let early stopping decide the real count. That’s more reliable than guessing.Stability across runs: the stopping point depends on randomness (from
subsample, for instance), so fixrandom_stateand average over several runs, or early stop inside cross-validation.
10.8 Cross-validation
Cross-validation (CV) is the gold standard for estimating generalization and selecting hyperparameters. For boosting, pay particular attention to how data are split and how early stopping interacts with the folds.
K-fold cross-validation:
Standard K-fold splits the data into K parts and rotates which part is held out. For boosting:
- Early stop inside each fold: within each fold’s training, split the fold’s training data further into a sub-training set and an early-stopping set. In 5-fold CV, for instance, each fold trains on 80% of the data, holds out 10% of that for early stopping, and fits on the remaining 70%.
- Record the best iteration per fold: folds may disagree on the best
n_estimators. The final count can be the mean or median across folds, or you can simply retrain on all the data. - Aggregate feature importance: each fold yields an importance vector; averaging them across folds gives a steadier final ranking.
Stratified K-fold (classification):
For imbalanced classification, use StratifiedKFold so each fold preserves the overall class proportions. With plain K-fold some folds may lack minority samples, which either breaks training or biases the estimate.
Time series data:
Boosting is often used for time series forecasting (sales, stock prices). Standard random K-fold then causes data leakage (predicting the past with the future). Use time series splitting; if features include lags or rolling-window statistics, TimeSeriesSplit(gap=...) (sklearn 0.24+) leaves a gap between train and validation so windows can’t straddle the boundary:
| |
Group K-fold (grouped data):
When the data have a group structure (several records per user, several visits per patient), all samples of a group must land entirely in train or entirely in validation. Use GroupKFold:
| |
10.9 Common traps and fixes
Trap 1: overfitting
Symptoms: excellent on the training set (AUC around 0.99), poor on validation (AUC around 0.70), a wide gap between them.
Causes:
- Single trees too complex (
max_depthornum_leavestoo large) - Too many iterations with no early stopping
- Regularization too weak (
lambda,alphatoo small) - Little data but high model complexity
- The model memorizing noise or outliers
Fixes:
- Lower
max_depth(XGBoost) ornum_leaves(LightGBM), and lowermax_depthas an extra bound - Use early stopping (
early_stopping_rounds) with a generousn_estimators - Raise
min_child_weight/min_data_in_leafso leaves can’t overfit single samples - Add
subsample(0.6-0.8) andcolsample_bytree(0.6-0.8) to cut variance - Raise
lambda(L2) andalpha(L1) to smooth the model - Clip or Winsorize outliers
Trap 2: underfitting
Symptoms: poor on both training and validation; the model hasn’t captured the patterns.
Causes:
- Single trees too simple (
max_depth=1ornum_leavestoo small) - Learning rate too small, or too few iterations
- Regularization too strong (
lambdaorgammatoo large) - No real relationship between features and target, or insufficient feature engineering
Fixes:
- Raise
max_depthornum_leavesso the model can learn more complex interactions - Raise
learning_rate(from 0.01 to 0.1, say) or increasen_estimators - Lower
gammaandlambdato relax the regularization - Revisit feature engineering: is an important feature missing? Are features scaled and encoded sensibly?
- Check data quality: are the labels right? Are the distributions plausible?
Trap 3: data leakage
Symptoms: excellent offline, sharply worse in production. Or CV scores far above an independent test set.
Causes:
- Target leakage: a feature contains information only available after prediction (using “did the user buy tomorrow” to predict “will the user buy tomorrow”)
- Temporal leakage: predicting the past with the future (failing to use
TimeSeriesSplit) - Preprocessing leakage: standardizing, imputing or target-encoding over the whole dataset before cross-validation, which leaks validation information into training
- Sample leakage: duplicates between train and test (the same user in both)
Fixes:
- The Pipeline rule: wrap every preprocessing step (scaling, encoding, imputation) in a
Pipelineso it’s fitted on each fold’s training data and applied to validation, keeping validation information out - Strict temporal splitting: use
TimeSeriesSplitso training data strictly precede validation data - Careful target encoding: if you use target encoding, compute it per fold from the training data only, never globally in advance
- CatBoost’s Ordered TS: for categorical features, CatBoost’s native support uses only the labels of samples “before” the current one, which markedly reduces the leakage target encoding brings
- Keep the early-stopping set separate from the test set: if the test set is used for early stopping or tuning, its score is no longer unbiased (see the Bayesian optimization example in 10.10)
Trap 4: class imbalance
Symptoms: in binary classification the majority class is predicted well but minority recall is very low (in fraud detection, 99% of transactions are legitimate, so predicting “all legitimate” already gives 99% accuracy).
Fixes:
- Sample weights: each library’s parameter, so the algorithm attends to the minority class:
| Library | Parameter | Notes |
|---|---|---|
| XGBoost | scale_pos_weight | The docs suggest sum(negative) / sum(positive) |
| LightGBM | is_unbalance=True or scale_pos_weight | Pick one, not both; the sklearn interface also accepts class_weight |
| CatBoost | auto_class_weights='Balanced' / 'SqrtBalanced', or scale_pos_weight | Don’t combine with class_weights |
| sklearn | HistGradientBoostingClassifier(class_weight='balanced') (1.2+) | GradientBoostingClassifier has no class_weight; pass weights via fit(sample_weight=...) |
Note that reweighting changes what predicted probabilities mean. XGBoost’s tuning docs point out that if you only care about a ranking metric such as AUC, scale_pos_weight is fine; if you need accurate probabilities, don’t rebalance the data, and instead set max_delta_step to a finite value (1, say) to help convergence. LightGBM’s docs likewise warn that is_unbalance / scale_pos_weight degrade per-sample probability estimates. When probabilities matter, train with weights and then calibrate (sklearn’s CalibratedClassifierCV, for instance).
2. Resampling (only on training folds; validation and test data keep the original distribution — imbalanced-learn’s Pipeline helps enforce this):
- Oversampling (SMOTE): oversample the minority class in the training set
- Undersampling: undersample the majority class (watch for information loss)
- Combined sampling (SMOTEENN, say): oversampling plus undersampling
- Change the metric: replace accuracy with AUC-ROC, AUC-PR (area under the precision-recall curve), F1 or balanced accuracy
- Adjust the threshold: the default 0.5 may be wrong for imbalanced data. Pick the operating threshold that fits the business need (precision versus recall), using the ROC or PR curve
- Focal loss: in extreme imbalance (1:1000 or worse), a custom focal loss down-weights easy samples and concentrates on the hard ones
10.10 Hyperparameter search examples
Grid search
Grid search enumerates every combination in a predefined space, which suits small spaces with ample compute.
| |
Random search
Random search samples a specified number of combinations from the space, which is more efficient than a grid, especially in high dimensions.
| |
Bayesian optimization
Bayesian optimization — more generally, sequential model-based optimization (SMBO) — fits a surrogate model to the results so far and uses an acquisition function to choose the next point to evaluate, which usually finds better parameters than random search for the same number of evaluations. The classic surrogate is a Gaussian process; Optuna’s default TPESampler uses TPE (Tree-structured Parzen Estimator, Bergstra et al., 2011), which splits existing trials into “good” and “bad” groups by objective value, models each with a kernel density estimate $\ell(x)$ and $g(x)$, and prefers parameters with a large ratio $\ell(x)/g(x)$. Optuna (Akiba et al., 2019) is the recommended library.
Bergstra & Bengio (2012) showed that when only a few hyperparameters really matter, random search beats grid search — which is why full grids aren’t recommended when there are many parameters.
| |
The three search strategies compared:
| Method | Pros | Cons | When |
|---|---|---|---|
| Grid search | Exhaustive, so it finds the best point on the grid | Expensive; curse of dimensionality | Small space (2-3 parameters), ample compute |
| Random search | Efficient, explores a wider range | No guarantee of optimality; sparse sampling | Large space, limited budget (10-100 evaluations) |
| Bayesian optimization (TPE, say) | Uses past results to guide sampling, usually beating random search at the same budget | Sequential, so parallelism is limited; degrades if the space is poorly designed | Complex spaces, expensive evaluations (hours per training run) |
10.11 Monotonic and interaction constraints
In credit risk, pricing and medicine, business requirements often impose prior knowledge on the model: “higher income shouldn’t raise default probability,” “a larger house shouldn’t be valued lower.” Left to tuning alone, a model can learn sawtooth relationships that violate common sense in sparse regions. All four libraries support monotonic constraints: 1 means predictions are monotonically non-decreasing in that feature, -1 non-increasing, and 0 unconstrained.
| Library | Monotonic constraint | Interaction constraint | Notes |
|---|---|---|---|
| XGBoost | monotone_constraints | interaction_constraints | Accepts a tuple (in feature order) or a dict keyed by feature name; interaction constraints are nested lists such as [[0, 1], [2, 3, 4]] |
| LightGBM | monotone_constraints | interaction_constraints | Must list all features in order; monotone_constraints_method offers basic (default, fastest), intermediate and advanced |
| CatBoost | monotone_constraints | — | Numeric features only |
| sklearn HistGB | monotonic_cst | interaction_cst | For classification, monotonic constraints apply only to binary problems, acting on the positive-class probability |
Interaction constraints restrict which features may appear on the same root-to-leaf path. [[0, 1], [2, 3, 4]], for instance, allows interaction within features 0 and 1 and within 2, 3 and 4, but not across the two groups. They’re commonly used to improve interpretability, or as regularization when some features are known to act independently.
Constraints usually cost some training fit, but the model matches the business logic better and behaves more sensibly out of distribution. Whether a constraint is worth it should be judged by validation metrics together with business review. The example below imposes a monotonically increasing constraint on $x_0$ over synthetic data and checks predictions from all four libraries:
| |
Run under XGBoost 3.4.1, LightGBM 4.7.0, CatBoost 1.2.10 and scikit-learn 1.9.1, all four models print True.
