# XGBoost Reference Guide ## What XGBoost Is XGBoost (Extreme Gradient Boosting) is a scalable, regularized gradient boosted decision tree library. It is designed for both performance (systems optimization) and accuracy (ML principles). It consistently ranks among the top performers on structured/tabular data. --- ## How the Algorithm Works ### Model Structure Prediction is a sum over K trees: `ŷᵢ = Σₖ fₖ(xᵢ)` where each fₖ is a CART (Classification and Regression Tree). Trees "try to complement each other" — the ensemble is stronger than any individual tree. ### Additive Training Trees are learned one at a time (sequentially): `ŷᵢ⁽ᵗ⁾ = ŷᵢ⁽ᵗ⁻¹⁾ + fₜ(xᵢ)` At each step, a new tree fₜ is added that minimizes the objective given all previously learned trees. ### Regularized Objective `obj(θ) = L(θ) + Ω(θ)` - **L(θ):** Training loss (any differentiable loss function) - **Ω(θ):** Tree complexity penalty: `γT + ½λ Σⱼ wⱼ²` - T = number of leaves, γ = leaf count penalty, λ = L2 weight regularization This explicit regularization controls bias-variance tradeoff more formally than heuristic approaches (e.g., impurity-based splitting alone). ### Tree Learning via Taylor Expansion Objective at step t is approximated using second-order Taylor expansion of the loss: `obj⁽ᵗ⁾ ≈ Σᵢ [gᵢfₜ(xᵢ) + ½hᵢfₜ²(xᵢ)] + Ω(fₜ)` - gᵢ = first derivative (gradient) of loss w.r.t. ŷᵢ⁽ᵗ⁻¹⁾ - hᵢ = second derivative (hessian) This formulation allows optimizing **any differentiable loss** with the same tree solver. ### Structure Score and Split Gain For a given tree structure, optimal leaf weights and minimum objective are: `obj* = −½ Σⱼ Gⱼ²/(Hⱼ+λ) + γT` **Split gain** (used to evaluate candidate splits): `Gain = ½ [G_L²/(H_L+λ) + G_R²/(H_R+λ) − (G_L+G_R)²/(H_L+H_R+λ)] − γ` If Gain < γ, the split is rejected — this is the principled basis for pruning. ### Key Design Principles - Loss function flexibility: any differentiable loss via gradient/hessian - Formal regularization (γ, λ, α) prevents overfitting more robustly than heuristics - Principled optimization: tree structure and leaf weights are jointly optimized - Second-order information (Hessian) avoids line search, enabling faster convergence ### XGBoost vs Random Forests Both use tree ensembles. Key difference: Random Forests train trees independently (parallel); XGBoost trains sequentially, each tree correcting previous errors. --- ## Key Parameters ### General Parameters | Parameter | Default | Description | |-----------|---------|-------------| | `booster` | `gbtree` | Booster type: `gbtree`, `gblinear`, `dart` | | `device` | `cpu` | Compute device: `cpu`, `cuda`, `cuda:` | | `nthread` | max available | Parallel threads for training | | `verbosity` | 1 | 0=silent, 1=warning, 2=info, 3=debug | | `validate_parameters` | False | Enable input parameter validation | ### Tree Booster Parameters | Parameter | Default | Description | |-----------|---------|-------------| | `eta` / `learning_rate` | 0.3 | Shrinkage step size to prevent overfitting; typical range 0.01–0.3 | | `max_depth` | 6 | Maximum tree depth; typical range 3–10 | | `gamma` / `min_split_loss` | 0 | Minimum loss reduction required for a split (higher = more conservative) | | `min_child_weight` | 1 | Minimum sum of instance hessian in a child node (controls overfitting) | | `subsample` | 1 | Fraction of training rows sampled per tree; 0.5–0.9 reduces overfitting | | `colsample_bytree` | 1 | Fraction of columns sampled per tree | | `colsample_bylevel` | 1 | Fraction of columns sampled per tree level | | `colsample_bynode` | 1 | Fraction of columns sampled per split node | | `lambda` / `reg_lambda` | 1 | L2 regularization on leaf weights | | `alpha` / `reg_alpha` | 0 | L1 regularization on leaf weights (produces sparsity) | | `max_delta_step` | 0 | Maximum weight update per leaf; useful for imbalanced classification | | `tree_method` | `auto` | Algorithm: `exact`, `approx`, `hist` (hist recommended for large data) | | `grow_policy` | `depthwise` | `depthwise` (BFS) or `lossguide` (best-first, like LightGBM) | | `max_leaves` | 0 | Max leaves to add per tree (lossguide only; 0 = unlimited) | | `max_bin` | 256 | Max discrete bins for continuous features (hist tree method) | | `sampling_method` | `uniform` | `uniform` or `gradient_based` (GPU only) | ### Learning Task Parameters | Parameter | Default | Description | |-----------|---------|-------------| | `objective` | `reg:squarederror` | Learning task; determines loss function | | `eval_metric` | task-dependent | Evaluation metric(s): `rmse`, `mae`, `logloss`, `auc`, `error`, `ndcg`, etc. | | `seed` | 0 | Random seed for reproducibility | | `base_score` | auto | Global bias / initial prediction (auto-estimated from data) | ### Objective Options **Regression:** `reg:squarederror`, `reg:squaredlogerror`, `reg:logistic`, `reg:pseudohubererror`, `reg:absoluteerror`, `reg:quantileerror`, `reg:gamma`, `reg:tweedie` **Classification:** `binary:logistic` (outputs probability), `binary:logitraw` (outputs margin), `binary:hinge`, `multi:softmax` (outputs class), `multi:softprob` (outputs probability per class) **Specialized:** `count:poisson`, `survival:cox`, `survival:aft`, `rank:ndcg`, `rank:map`, `rank:pairwise` --- ## Practical Guidance **Starting point for tuning:** - Set `n_estimators` high (500–1000) with early stopping - `learning_rate`: 0.05–0.1 - `max_depth`: 4–6 - `subsample`: 0.8, `colsample_bytree`: 0.8 - `gamma`: 0.1–1.0 if overfitting - `min_child_weight`: 3–10 for imbalanced data **For large datasets:** Use `tree_method='hist'` — equivalent to LightGBM's histogram algorithm. **Class imbalance:** Use `scale_pos_weight = neg_count / pos_count` with `binary:logistic`; or set `max_delta_step=1`. **Missing values:** XGBoost handles NaN natively — learns optimal default direction per split during training. **Regularization priority:** Start with `lambda` (L2), then add `alpha` (L1) only if sparsity is desired. `gamma` controls tree complexity directly.