3.1 The additive model: boosting’s mathematical base
The mathematical framework of boosting rests on the additive model. An additive model writes a complicated prediction function as a linear combination of simple basis functions, which turns one high-dimensional optimization problem into a series of low-dimensional ones.
3.1.1 The form of an additive model
The general form is
$$f(x) = \sum_{m=1}^M \beta_m b(x; \gamma_m)$$Notation:
| Symbol | Meaning | Notes |
|---|---|---|
| $x$ | Input feature vector | $x \in \mathbb{R}^d$ |
| $M$ | Total number of basis functions | Corresponds to the number of boosting iterations |
| $\beta_m$ | Expansion coefficient of the $m$-th basis function | A scalar weight setting that function’s contribution |
| $b(x; \gamma_m)$ | Basis function (the weak learner) | A simple function characterized by $\gamma_m$ |
| $\gamma_m$ | Parameter vector of the basis function | Split points and leaf values of a decision tree, for instance |
| $f(x)$ | Final prediction function | Weighted sum of all basis functions |
The flexibility of the additive model lies in the choice of $b(x; \gamma)$. In different settings it takes different forms:
- In a single-hidden-layer neural network, $b(x; \gamma) = \sigma(\gamma_0 + \gamma_1^\top x)$, where $\sigma(\cdot)$ is the sigmoid activation and $\gamma = (\gamma_0, \gamma_1)$ parameterizes that hidden unit;
- In wavelet analysis, $b(x; \gamma)$ is a mother wavelet shifted and scaled;
- In boosting, $b(x; \gamma)$ is usually a decision tree (a decision stump, a CART tree), with $\gamma$ encoding the split structure and leaf values.
“The basis function expansions are more generally represented by $f(x) = \sum_{m=1}^M \beta_m b(x; \gamma_m)$ where $\beta_m$ are the expansion coefficients and $b(x; \gamma) \in \mathbb{R}$ are usually simple functions of the multivariate argument $x$, characterized by a set of parameters $\gamma$.” — Lund University Thesis, 2022
3.1.2 The full objective
Given training data $\{(x_i, y_i)\}_{i=1}^N$, the additive model fits its parameters by minimizing a loss:
$$\min_{\{\beta_m, \gamma_m\}_{m=1}^M} \sum_{i=1}^N L\left(y_i, \sum_{m=1}^M \beta_m b(x_i; \gamma_m)\right)$$Here $L(y, f(x))$ is the loss function, measuring the gap between the prediction $f(x)$ and the true value $y$.
Optimizing that joint problem directly is computationally prohibitive, for two reasons:
- The parameter space is high-dimensional. You’d have to optimize $M$ coefficients $\beta_m$ together with $M$ parameter sets $\gamma_m$, which makes for an enormous search space.
- Non-convexity. With complicated basis functions such as decision trees, the joint objective over all parameters is generally non-convex and prone to local optima.
Forward stagewise additive modeling offers a greedy, incremental way around this.
3.2 Forward stagewise additive modeling
3.2.1 The idea
The idea is simple: start from an empty model (or a constant baseline), and at each stage greedily add one new basis function, holding the coefficients and parameters already added fixed and optimizing only the new term.
That converts one hard joint optimization into a sequence of easy one-step optimizations. Each step involves a single $\beta$ and a single $\gamma$, which cuts the complexity dramatically.
“Boosting algorithms can be viewed as an additive model… The approach often used here is Forward Stagewise Optimization, which begins with a baseline of no functions, then incrementally adds to the model by optimizing one weight and one function at a time, keeping previously selected elements fixed.” — PSL_Book, Section 12.5
3.2.2 The algorithm
Algorithm: forward stagewise additive modeling
Initialize: $f_0(x) = 0$ (or a constant baseline, e.g. $f_0(x) = \arg\min_c \sum_{i=1}^N L(y_i, c)$)
For $m = 1$ to $M$ do:
(a) Solve for the optimal basis-function parameters and coefficient:
$$(\beta_m, \gamma_m) = \arg\min_{\beta, \gamma} \sum_{i=1}^N L\left(y_i, f_{m-1}(x_i) + \beta \cdot b(x_i; \gamma)\right)$$(b) Update the model:
$$f_m(x) = f_{m-1}(x) + \beta_m \cdot b(x; \gamma_m)$$Output: $f_M(x) = \sum_{m=1}^M \beta_m b(x; \gamma_m)$
Key properties:
- Greedy. Each stage adds one optimal basis function without revisiting the coefficients or parameters of the ones already in the model. That doesn’t guarantee a global optimum, but it usually works well in practice.
- Adaptive. Each step depends on what came before, accumulated in $f_{m-1}(x)$. Every step optimizes against the current model’s “residual” — under squared loss, step (a) is exactly fitting $\beta b(x; \gamma)$ to the current residual $y_i - f_{m-1}(x_i)$: $$L\left(y_i, f_{m-1}(x_i) + \beta b(x_i; \gamma)\right) = \frac{1}{2}\left(y_i - f_{m-1}(x_i) - \beta b(x_i; \gamma)\right)^2$$
- Stagewise vs. backfitting. Backfitting cycles through all basis functions updating their parameters. Stagewise adds a term once and never touches it again. That one-pass policy makes the algorithm more stable and cheaper to run.
“Boosting adopts the forward stagewise method that solves by sequentially fitting a single weak learner and adding it to the previously fitted terms. The previously fitted terms are not readjusted as new terms are added into the model. This characteristic is commonly called adaptive.” — SOA Delta Boosting Report
3.2.3 Flowchart of forward stagewise modeling
flowchart TD
A["Initialize
f₀(x) = 0"] --> B{"For m = 1 to M"}
B --> C["Starting from the current model
compute the optimal increment"]
C --> D["Solve for this round's
optimal basis function and coefficient"]
D --> E["Add the new basis function
to the additive model"]
E --> F{"M rounds
complete?"}
F -->|No| B
F -->|Yes| G["Output the final model
f_M(x) = Σ β_m·b(x;γ_m)"]
style A fill:#f9f,stroke:#333
style G fill:#bbf,stroke:#333
3.3 What the loss function does in boosting
The loss function sets the direction. It defines how to measure the gap between prediction and truth, and every boosting round tries to shrink it. Different losses give different boosting algorithms and suit different problems.
“The loss function defines how the errors between predicted and actual values are measured. It acts as a guide for the algorithm, helping it understand where it is going wrong and how much correction is needed in the next step. Without a loss function, the model would have no direction in minimizing errors and improving accuracy.” — Datamites, 2025
3.3.1 AdaBoost: exponential loss
AdaBoost can be shown to be equivalent to a forward stagewise algorithm on an additive model under the exponential loss.
The exponential loss AdaBoost minimizes is
$$J(F) = \mathbb{E}\left[e^{-yF(x)}\right]$$where $y \in \{-1, +1\}$ is the class label and $F(x) = \sum_{m=1}^M \alpha_m G_m(x)$ is the additive classifier.
Notation:
| Symbol | Meaning | Notes |
|---|---|---|
| $y$ | True class label | $y \in \{-1, +1\}$ (binary classification) |
| $F(x)$ | Additive classifier | $F(x) = \sum_{m=1}^M \alpha_m G_m(x)$ |
| $G_m(x)$ | The $m$-th weak classifier | Outputs $\{-1, +1\}$ |
| $\alpha_m$ | Weight of the $m$-th classifier | Based on its weighted error rate |
| $\mathbb{E}$ | Expectation operator | Taken over the data distribution |
One important property of exponential loss: the $F(x)$ that minimizes $J(F)$ is exactly the symmetric logistic transform of the class probability,
$$F^*(x) = \frac{1}{2} \log \frac{P(y=1 \mid x)}{P(y=-1 \mid x)}$$(Derivation: for fixed $x$, $\mathbb{E}\left[e^{-yF(x)} \mid x\right] = P(y=1 \mid x)\, e^{-F(x)} + P(y=-1 \mid x)\, e^{F(x)}$; differentiate with respect to $F(x)$ and set to zero.) Conversely, $P(y=1 \mid x) = \frac{1}{1 + e^{-2F^*(x)}}$, so AdaBoost’s real-valued output can be converted into a probability estimate.
“Consider minimizing the criterion $J(F) = E(e^{-yF(x)})$ for estimation of $F(x)$. The function $F(x)$ that minimizes $J(F)$ is the symmetric logistic transform of $P(y=1 \mid x)$.” — Friedman, Hastie & Tibshirani, 2000, The Annals of Statistics, Lemma 1
3.3.2 LogitBoost: log-likelihood loss (binomial deviance)
Elegant as it is in theory, exponential loss isn’t the most robust choice statistically. Friedman, Hastie & Tibshirani (2000) proposed using the binomial log-likelihood loss instead — the negative log-likelihood, also called the binomial deviance.
The binomial log-likelihood itself is
$$\ell(y^*, p(x)) = y^* \log p(x) + (1-y^*) \log(1-p(x)) = -\log\left(1 + e^{-2yF(x)}\right)$$where $y^* = (y+1)/2 \in \{0, 1\}$ is the recoded label and $p(x) = \frac{1}{1+e^{-2F(x)}}$ is the predicted probability. Note that this follows FHT’s (2000) parameterization, in which $F(x)$ is half the log-odds, putting it on the same scale as the exponential loss’s population minimizer $F^*(x)$. Implementations such as sklearn and XGBoost usually let $F(x)$ equal the log-odds directly, in which case $p(x) = \frac{1}{1+e^{-F(x)}}$ and the loss is written $\log\left(1 + e^{-yF(x)}\right)$ with $y \in \{-1,+1\}$ — the two differ only by a factor of 2.
What boosting actually minimizes is the negative log-likelihood, that is, the logistic loss:
$$L(y, F) = -\ell(y^*, p(x)) = \log\left(1 + e^{-2yF(x)}\right)$$The advantages of the log-likelihood loss:
- It corresponds directly to maximum likelihood and so has firmer statistical footing.
- It is more robust to outliers and mislabeled samples than exponential loss: as the margin $yF \to -\infty$, the log loss grows roughly linearly (about $-2yF$), whereas the exponential loss $e^{-yF}$ grows exponentially and punishes badly misclassified samples far too hard.
- It matches the logistic regression loss, which makes it easy to explain.
“The binomial log-likelihood is $l(y^*, p(x)) = y^* \log(p(x)) + (1-y^*) \log(1-p(x)) = -\log(1 + e^{-2yF(x)})$.” — Friedman, Hastie & Tibshirani, 2000, The Annals of Statistics, Eq. 27
3.3.3 Gradient boosting: any differentiable loss
The breakthrough of gradient boosting was to allow any differentiable loss function. Each round, the new base learner is trained to fit the negative gradient of the loss at the current model (the pseudo-residuals):
$$-g_m(x_i) = -\left[\frac{\partial L(y_i, F(x_i))}{\partial F(x_i)}\right]_{F=F_{m-1}}$$The model then moves along the negative gradient:
$$F_m(x) = F_{m-1}(x) + \rho_m h_m(x)$$Here $h_m(x)$ is the base learner fitted to the negative gradient, and $\rho_m$ is the best step size from a line search:
$$\rho_m = \arg\min_{\rho} \sum_{i=1}^N L\left(y_i, F_{m-1}(x_i) + \rho\, h_m(x_i)\right)$$Note that $\rho_m$ and the “learning rate” (shrinkage coefficient $\nu$) are two different things: $\rho_m$ is the optimal step computed from the data each round, while $\nu \in (0, 1]$ is a hyperparameter you set, with the update actually $F_m = F_{m-1} + \nu \rho_m h_m$ (see Chapter 5).
“Gradient Boosting = Gradient Descent + Boosting. Fit an additive model (ensemble) $\sum_t \rho_t h_t(x)$ in a forward stage-wise manner. In each stage, introduce a weak learner to compensate the shortcomings of existing weak learners. In Gradient Boosting, ‘shortcomings’ are identified by gradients.” — Cheng Li, Northeastern University, “A Gentle Introduction to Gradient Boosting”
3.3.4 Common loss functions
The table below collects the losses used most often in boosting and where each applies:
| Problem type | Loss | Form | Used by | Notes |
|---|---|---|---|---|
| Regression | Squared loss | $L(y, F) = \frac{1}{2}(y - F)^2$ | Gradient Boosting (regression) | Sensitive to outliers, cheap to compute |
| Regression | Absolute loss | $L(y, F) = \lvert y - F\rvert$ | Gradient Boosting (regression) | More robust to outliers |
| Regression | Huber loss | $L_\delta(y,F)$ (below) | Gradient Boosting (regression) | Combines the strengths of squared and absolute loss |
| Regression | Quantile loss | $L_\tau(y,F)$ (below) | Gradient Boosting (quantile regression) | Estimates conditional quantiles |
| Classification | Exponential loss | $L(y, F) = e^{-yF}$ | AdaBoost | Heavy penalty on misclassified samples, pushes for large margins |
| Classification | Log loss | $L(y, F) = \log(1 + e^{-2yF})$ | LogitBoost, GBM (classification) | Consistent with maximum likelihood, more robust |
| Classification | Multiclass log loss | $L(\{y_k\}, \{F_k\}) = -\sum_{k=1}^K y_k \log p_k(x)$ | GBM (multiclass) | Handles multiclass problems; $y_k$ is the one-hot label and $p_k$ the softmax probability |
| Ranking | Pairwise logistic loss | $L = \log\left(1 + e^{-\sigma\left(F(x_i) - F(x_j)\right)}\right)$ over pairs with $y_i \succ y_j$ | LambdaMART | For learning to rank (LTR); LambdaMART replaces the raw gradient with “$\lambda$-gradients” scaled by $\lvert \Delta \mathrm{NDCG} \rvert$ |
Huber loss in full:
$$ L_\delta(y,F) = \begin{cases} \frac{1}{2}(y-F)^2, & \lvert y-F\rvert \leq \delta \\ \delta\lvert y-F\rvert - \frac{\delta^2}{2}, & \lvert y-F\rvert > \delta \end{cases} $$Quantile loss in full:
$$ L_\tau(y,F) = \begin{cases} \tau(y-F), & y \geq F \\ (\tau-1)(y-F), & y < F \end{cases} $$where $\tau \in (0, 1)$ is the target quantile (the alpha parameter in sklearn).
3.3.5 Negative gradients, population minimizers and robustness
What gradient boosting actually uses each round is the loss’s negative gradient (the pseudo-residual), while the loss’s population minimizer $F^*(x) = \arg\min_{F(x)} \mathbb{E}\left[L(y, F(x)) \mid x\right]$ determines what quantity the model ends up estimating. The table below lists both for the common losses (see ESL §10.5–10.6 and Table 10.2; for the classification losses $y \in \{-1, +1\}$ and $p(x) = P(y = 1 \mid x)$):
| Loss | Negative gradient $-\partial L / \partial F$ | Population minimizer $F^*(x)$ | Robustness |
|---|---|---|---|
| Squared loss $\frac{1}{2}(y-F)^2$ | $y - F$ | $\mathbb{E}[y \mid x]$ | Poor: the gradient grows linearly with the residual, so outliers dominate |
| Absolute loss $\lvert y-F \rvert$ | $\operatorname{sign}(y - F)$ | $\operatorname{median}(y \mid x)$ | Good: bounded gradient |
| Huber loss $L_\delta$ | $y - F$ (when $\lvert y-F \rvert \leq \delta$); $\delta \operatorname{sign}(y-F)$ (otherwise) | Conditional location parameter in the Huber M-estimation sense (equal to the conditional mean and median when the conditional distribution is symmetric) | Good: behaves like squared loss for small residuals, gradient truncated to $\pm\delta$ for large ones |
| Quantile loss $L_\tau$ | $\tau - \mathbb{I}(y < F)$ | The $\tau$ quantile of $y \mid x$ | Good: bounded gradient |
| Exponential loss $e^{-yF}$ | $y\, e^{-yF}$ | $\frac{1}{2}\log\frac{p(x)}{1-p(x)}$ | Poor: the gradient grows exponentially as the misclassification worsens |
| Log loss $\log\left(1+e^{-2yF}\right)$ | $\frac{2y}{1+e^{2yF}}$ | $\frac{1}{2}\log\frac{p(x)}{1-p(x)}$ | Fair: gradient magnitude never exceeds 2 |
| Multiclass log loss | $y_k - p_k(x)$ (for the $k$-th function $F_k$) | $F_k^*(x) = \log p_k(x) + c(x)$ (softmax only pins things down up to a shared function) | Fair |
A few notes:
- Exponential and log loss share the same population minimizer, half the log-odds, so at the population level they estimate the same quantity. The difference shows up in how much weight they give misclassified samples in finite samples — which is why AdaBoost is more sensitive to mislabeled data (ESL §10.6).
- Every classification loss above can be written as a monotonically decreasing function of the margin $yF(x)$: a positive margin means a correct classification, and the larger it is the more confident the call. All of them are convex surrogates for the 0-1 loss $\mathbb{I}(yF \leq 0)$ (exponential loss, and log loss normalized by $\log 2$, are both upper bounds on the 0-1 loss).
- In Huber loss, Friedman’s (2001) M-TreeBoost sets the threshold $\delta$ each round to the $\alpha$ quantile of the current absolute residuals $\lvert y_i - F_{m-1}(x_i) \rvert$ (sklearn defaults to
alpha=0.9), so it adapts over the course of training.
3.4 Deriving residual = negative gradient under squared loss
Under squared loss, gradient boosting has a particularly clean property: the negative gradient equals the residual. That gives the “fit the residuals” intuition a rigorous foundation.
3.4.1 The derivation
Let the current model be $F_{m-1}(x)$. At round $m$ we want a new base learner $h_m(x)$ and coefficient $\beta$ so that the updated model $F_{m-1}(x) + \beta h_m(x)$ minimizes squared loss.
The squared loss is
$$L(y, F) = \frac{1}{2}(y - F)^2$$For the $i$-th training sample, the partial derivative of the loss with respect to the current prediction $F(x_i)$ is
$$g_m(x_i) = \frac{\partial L(y_i, F(x_i))}{\partial F(x_i)} = \frac{\partial}{\partial F(x_i)} \left[\frac{1}{2}\left(y_i - F(x_i)\right)^2\right]$$Applying the chain rule:
$$g_m(x_i) = \frac{1}{2} \cdot 2 \cdot (y_i - F(x_i)) \cdot (-1) = -(y_i - F(x_i)) = F(x_i) - y_i$$So the negative gradient is
$$-g_m(x_i) = y_i - F_{m-1}(x_i) = r_{im}$$where $r_{im} = y_i - F_{m-1}(x_i)$ is exactly the residual of the $i$-th sample at round $m$.
3.4.2 What it means
The derivation makes a deeper point: under squared loss, “fit the negative gradient” and “fit the residual” are the same thing. That’s why early gradient boosting implementations (the regression version of GBM, say) simply fitted each round’s base learner to the residuals — under squared loss the residual is the negative gradient.
When the loss isn’t squared loss (log loss for classification, for instance), the negative gradient is no longer a plain residual and goes by the name pseudo-residual. We still fit the base learner to it, and that’s precisely what lets gradient boosting handle any differentiable loss.
“For regression with square loss, residual $\Leftrightarrow$ negative gradient. fit $h$ to residual $\Leftrightarrow$ fit $h$ to negative gradient. update $F$ based on residual $\Leftrightarrow$ update $F$ based on negative gradient. So we are actually updating our model using gradient descent!” — Cheng Li, Northeastern University
“With squared error loss, every estimation step is a reestimation on the residuals.” — University of Copenhagen Lecture Notes, 2011
3.5 A unified view of losses and forward stagewise algorithms
Combine the forward stagewise algorithm with different loss choices and the various boosting algorithms line up in one picture:
AdaBoost = forward stagewise additive model + exponential loss + classifiers as base learners
LogitBoost = forward stagewise additive model + binomial log-likelihood loss + Newton-step updates
Gradient Boosting (regression) = forward stagewise additive model + squared loss + residual fitting
Gradient Boosting (classification) = forward stagewise additive model + log loss + pseudo-residual fitting
XGBoost = forward stagewise additive model + any loss + second-order Taylor expansion + regularization
This unified view gets at what the boosting family really is: every member is a special case of the forward stagewise additive model, and they differ only in the loss they choose and the optimization strategy they use.
3.6 Chapter summary
This article worked through boosting’s mathematical base: the forward stagewise additive model and the loss-function framework. The additive model $f(x) = \sum_{m=1}^M \beta_m b(x; \gamma_m)$ breaks a complicated function into a combination of simple basis functions, and the forward stagewise algorithm makes the computation tractable through a greedy strategy. The loss you pick determines how boosting behaves: exponential loss gives AdaBoost, log-likelihood loss gives LogitBoost, any differentiable loss gives gradient boosting. In the squared-loss case in particular, the negative gradient equals the residual exactly, which is the proof behind the “fit the residuals” intuition. Understanding this framework is the prerequisite for mastering the concrete algorithms that follow — AdaBoost, the Gradient Boosting Machine, XGBoost and LightGBM.
Key references:
Friedman, J., Hastie, T., & Tibshirani, R. (2000). “Additive Logistic Regression: A Statistical View of Boosting.” The Annals of Statistics, 28(2), 337-407. (Section 3.2, 4.1-4.2)
Friedman, J. H. (2001). “Greedy Function Approximation: A Gradient Boosting Machine.” The Annals of Statistics, 29(5), 1189-1232.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. (Ch. 10, Algorithm 10.2)
Cheng Li, Northeastern University. “A Gentle Introduction to Gradient Boosting.”
Burges, C. J. C. (2010). “From RankNet to LambdaRank to LambdaMART: An Overview.” Microsoft Research Technical Report MSR-TR-2010-82.
Disclaimer: This document was assembled from publicly available academic literature, textbooks and official documentation. All formulas and quotations are attributed. It is intended for academic reference only.
