Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
XGBoost is a scalable tree-boosting system that adds decision trees in stages, choosing each tree to improve a regularized objective. It uses both gradients and Hessians—the first and second derivatives of the loss—to score tree structures and calculate leaf weights. For approximate split finding, the original XGBoost paper introduced a weighted quantile sketch; current XGBoost also documents other tree methods, so the sketch is not a description of every modern training mode.
What XGBoost is—and what boosting does
XGBoost is a tree-boosting system introduced by Tianqi Chen and Carlos Guestrin in a paper published at KDD 2016. In boosting, a model is built additively: each new tree is selected to improve predictions made by the trees already in the model. Rather than treating each tree as an independent predictor, training evaluates how its contribution changes the overall objective.
The objective combines how well predictions fit the training data with a penalty for model complexity. In the tree objective described in the official tutorial, complexity is controlled by penalizing both the number of leaves and the squared values assigned to those leaves. This makes the learner balance fit against a more complicated tree or larger leaf scores. XGBoost’s boosted-trees tutorial derives this objective and its split-scoring formula.
Why XGBoost uses gradients and Hessians
At a boosting step, XGBoost approximates the loss with a second-order Taylor expansion around the model’s current predictions. For each training example, the first derivative is the gradient, which indicates the direction and local rate of change in loss; the second derivative is the Hessian, which describes local curvature. These statistics let the learner estimate the effect of adding a tree without repeatedly optimizing the original loss from scratch for every candidate structure.
#1 Best Overall
For a fixed tree structure, the tutorial gives a closed-form solution for each leaf’s weight using the gradient and Hessian totals for the examples assigned to that leaf. The same quantities support split evaluation: a candidate split is judged by the improvement from separating examples into child leaves, with a complexity cost for adding the branch. In practical terms, gradients say which way a prediction should move, while Hessians help account for how sensitive the loss is to that movement.
This is why XGBoost is not simply a process of fitting trees to residuals. Its structure and leaf scores are chosen through a regularized objective informed by first- and second-order loss information.
What the weighted quantile sketch contributes
The weighted quantile sketch is part of the paper’s approximate tree-learning method. A tree learner must choose candidate feature thresholds; exhaustively checking every possible split can be costly. A quantile sketch summarizes feature values so that useful candidate thresholds can be selected approximately. The XGBoost paper’s key distinction is that its sketch accounts for instance weights arising in the objective, rather than acting as ordinary unweighted quantile binning.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Chen and Guestrin describe the contribution this way: “We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning.” The sketch helps make approximate split finding practical, but it is one component of the original system, not a synonym for XGBoost as a whole. The KDD 2016 paper also describes sparsity-aware split finding and systems techniques involving cache access patterns, compression, and sharding.
How current XGBoost tree methods differ
Current XGBoost documentation describes several tree construction methods with different candidate-split strategies. The weighted quantile sketch is especially relevant to understanding the paper’s approximate learner, while the current methods should be distinguished by their documented algorithms:
| Tree method | Documented split strategy | How to interpret it |
|---|---|---|
exact |
Enumerates split candidates. | Uses candidate enumeration rather than the approximate strategies described for the other methods. |
approx |
Uses a quantile sketch and gradient histogram. | Connects most directly to the approximate split-finding explanation. |
hist |
Uses a histogram-optimized approximate greedy algorithm. | A distinct histogram-based approximate method; do not assume it is simply the original weighted quantile sketch. |
The official XGBoost parameter reference describes hist as faster and says auto behaves as hist. Those descriptions identify the methods’ intended approaches, not a guarantee that one will be faster for every dataset or setup. Check the current stable documentation for the release you use, because implementation details and parameter behavior can change. The project’s stable documentation is the starting point.
What the original paper’s scale claim means
The paper abstract says XGBoost “scales beyond billions of examples using far fewer resources than existing systems.” This is the authors’ claim in the 2016 paper, not a current benchmark result or a promise for every workload, hardware configuration, or XGBoost release. The abstract attributes its system’s scalability to combining algorithmic contributions—including sparsity-aware learning and the weighted quantile sketch—with systems design.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
