data/knowledge_base/xgboost_guide.md

XGBoost Reference Guide

What XGBoost Is

XGBoost (Extreme Gradient Boosting) is a scalable, regularized gradient boosted decision tree library. It is designed for both performance (systems optimization) and accuracy (ML principles). It consistently ranks among the top performers on structured/tabular data.

---

How the Algorithm Works

Model Structure

Prediction is a sum over K trees: ŷᵢ = Σₖ fₖ(xᵢ) where each fₖ is a CART (Classification and Regression Tree).

Trees "try to complement each other" — the ensemble is stronger than any individual tree.

Additive Training

Trees are learned one at a time (sequentially): ŷᵢ⁽ᵗ⁾ = ŷᵢ⁽ᵗ⁻¹⁾ + fₜ(xᵢ)

At each step, a new tree fₜ is added that minimizes the objective given all previously learned trees.

Regularized Objective

obj(θ) = L(θ) + Ω(θ)

• L(θ): Training loss (any differentiable loss function)
• Ω(θ): Tree complexity penalty: γT + ½λ Σⱼ wⱼ²
  • T = number of leaves, γ = leaf count penalty, λ = L2 weight regularization

This explicit regularization controls bias-variance tradeoff more formally than heuristic approaches (e.g., impurity-based splitting alone).

Tree Learning via Taylor Expansion

Objective at step t is approximated using second-order Taylor expansion of the loss: obj⁽ᵗ⁾ ≈ Σᵢ [gᵢfₜ(xᵢ) + ½hᵢfₜ²(xᵢ)] + Ω(fₜ)

• gᵢ = first derivative (gradient) of loss w.r.t. ŷᵢ⁽ᵗ⁻¹⁾
• hᵢ = second derivative (hessian)

This formulation allows optimizing any differentiable loss with the same tree solver.

Structure Score and Split Gain

For a given tree structure, optimal leaf weights and minimum objective are: obj* = −½ Σⱼ Gⱼ²/(Hⱼ+λ) + γT

Split gain (used to evaluate candidate splits): Gain = ½ [G_L²/(H_L+λ) + G_R²/(H_R+λ) − (G_L+G_R)²/(H_L+H_R+λ)] − γ

If Gain < γ, the split is rejected — this is the principled basis for pruning.

Key Design Principles

• Loss function flexibility: any differentiable loss via gradient/hessian
• Formal regularization (γ, λ, α) prevents overfitting more robustly than heuristics
• Principled optimization: tree structure and leaf weights are jointly optimized
• Second-order information (Hessian) avoids line search, enabling faster convergence

XGBoost vs Random Forests

Both use tree ensembles. Key difference: Random Forests train trees independently (parallel); XGBoost trains sequentially, each tree correcting previous errors.

---

Key Parameters

General Parameters

Parameter	Default	Description
booster	gbtree	Booster type: gbtree, gblinear, dart
device	cpu	Compute device: cpu, cuda, cuda:<n>
nthread	max available	Parallel threads for training
verbosity	1	0=silent, 1=warning, 2=info, 3=debug
validate_parameters	False	Enable input parameter validation

Tree Booster Parameters

Parameter	Default	Description
eta / learning_rate	0.3	Shrinkage step size to prevent overfitting; typical range 0.01–0.3
max_depth	6	Maximum tree depth; typical range 3–10
gamma / min_split_loss	0	Minimum loss reduction required for a split (higher = more conservative)
min_child_weight	1	Minimum sum of instance hessian in a child node (controls overfitting)
subsample	1	Fraction of training rows sampled per tree; 0.5–0.9 reduces overfitting
colsample_bytree	1	Fraction of columns sampled per tree
colsample_bylevel	1	Fraction of columns sampled per tree level
colsample_bynode	1	Fraction of columns sampled per split node
lambda / reg_lambda	1	L2 regularization on leaf weights
alpha / reg_alpha	0	L1 regularization on leaf weights (produces sparsity)
max_delta_step	0	Maximum weight update per leaf; useful for imbalanced classification
tree_method	auto	Algorithm: exact, approx, hist (hist recommended for large data)
grow_policy	depthwise	depthwise (BFS) or lossguide (best-first, like LightGBM)
max_leaves	0	Max leaves to add per tree (lossguide only; 0 = unlimited)
max_bin	256	Max discrete bins for continuous features (hist tree method)
sampling_method	uniform	uniform or gradient_based (GPU only)

Learning Task Parameters

Parameter	Default	Description
objective	reg:squarederror	Learning task; determines loss function
eval_metric	task-dependent	Evaluation metric(s): rmse, mae, logloss, auc, error, ndcg, etc.
seed	0	Random seed for reproducibility
base_score	auto	Global bias / initial prediction (auto-estimated from data)

Objective Options

Regression: reg:squarederror, reg:squaredlogerror, reg:logistic, reg:pseudohubererror, reg:absoluteerror, reg:quantileerror, reg:gamma, reg:tweedie

Classification: binary:logistic (outputs probability), binary:logitraw (outputs margin), binary:hinge, multi:softmax (outputs class), multi:softprob (outputs probability per class)

Specialized: count:poisson, survival:cox, survival:aft, rank:ndcg, rank:map, rank:pairwise

---

Practical Guidance

Starting point for tuning:

• Set n_estimators high (500–1000) with early stopping
• learning_rate: 0.05–0.1
• max_depth: 4–6
• subsample: 0.8, colsample_bytree: 0.8
• gamma: 0.1–1.0 if overfitting
• min_child_weight: 3–10 for imbalanced data

For large datasets: Use tree_method='hist' — equivalent to LightGBM's histogram algorithm.

Class imbalance: Use scale_pos_weight = neg_count / pos_count with binary:logistic; or set max_delta_step=1.

Missing values: XGBoost handles NaN natively — learns optimal default direction per split during training.

Regularization priority: Start with lambda (L2), then add alpha (L1) only if sparsity is desired. gamma controls tree complexity directly.