Spaces:
Running
Running
File size: 6,276 Bytes
eefa095 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 | # Hyperparameter Ranges — Practical Reference
Practical starting ranges for tree-based and linear models on tabular data. These are empirically validated defaults — tune from here rather than searching blind.
---
## XGBoost
| Parameter | Typical Range | Notes |
|-----------|--------------|-------|
| `n_estimators` | 100–1000 | Use early stopping to find optimal |
| `learning_rate` | 0.01–0.3 | Lower LR needs more trees; 0.05–0.1 is a good start |
| `max_depth` | 3–8 | Deeper trees overfit; 4–6 is usually optimal |
| `min_child_weight` | 1–10 | Higher = more conservative, reduces overfitting |
| `subsample` | 0.6–1.0 | Row sampling per tree; 0.8 is a safe default |
| `colsample_bytree` | 0.5–1.0 | Feature sampling per tree; 0.8 is a safe default |
| `gamma` | 0–5 | Minimum loss reduction to split; 0 means always split |
| `reg_alpha` (L1) | 0–1 | Useful for sparse feature sets |
| `reg_lambda` (L2) | 0–10 | Default 1; increase to regularize |
| `scale_pos_weight` | `neg/pos` | For imbalanced classification; set to ratio of negative/positive samples |
**Search order**: Fix `n_estimators` high with early stopping → tune `max_depth` + `min_child_weight` → tune `subsample` + `colsample_bytree` → tune `learning_rate` (lower it, add more trees).
---
## LightGBM
| Parameter | Typical Range | Notes |
|-----------|--------------|-------|
| `n_estimators` | 100–2000 | With early stopping |
| `learning_rate` | 0.01–0.2 | Start at 0.05 |
| `num_leaves` | 20–300 | Main complexity control; `< 2^max_depth` |
| `max_depth` | -1 (unlimited) or 6–12 | Prefer controlling via `num_leaves` |
| `min_child_samples` | 10–100 | Minimum samples per leaf; prevents overfitting on small data |
| `subsample` | 0.6–1.0 | Row sampling (`bagging_fraction`) |
| `colsample_bytree` | 0.5–1.0 | Feature sampling (`feature_fraction`) |
| `reg_alpha` | 0–1 | L1 regularization |
| `reg_lambda` | 0–10 | L2 regularization |
| `is_unbalance` | True/False | Auto-adjusts weights for imbalance |
**Key difference from XGBoost**: `num_leaves` is the primary complexity control. With `max_depth=6`, XGBoost has max 64 leaves; LightGBM can have 300+ at the same depth setting.
---
## CatBoost
| Parameter | Typical Range | Notes |
|-----------|--------------|-------|
| `iterations` | 100–2000 | Equivalent to `n_estimators` |
| `learning_rate` | 0.01–0.3 | Auto-tuned if not set |
| `depth` | 4–10 | CatBoost trees are symmetric (oblivious); depth 6–8 is typical |
| `l2_leaf_reg` | 1–10 | L2 regularization on leaves |
| `border_count` | 32–255 | Quantization borders for numeric features |
| `bagging_temperature` | 0–1 | 0 = no bagging, 1 = full Bayesian bootstrap |
| `random_strength` | 0–10 | Noise for split selection; helps avoid overfitting |
**CatBoost advantage**: Pass categorical column indices via `cat_features` — no manual encoding needed. It handles them via target statistics internally.
---
## Random Forest
| Parameter | Typical Range | Notes |
|-----------|--------------|-------|
| `n_estimators` | 100–500 | More trees = better, diminishing returns after 300 |
| `max_depth` | None or 10–30 | `None` (fully grown) is often best; limit for speed |
| `min_samples_split` | 2–20 | Minimum samples to split a node |
| `min_samples_leaf` | 1–10 | Minimum samples in a leaf |
| `max_features` | `"sqrt"`, `"log2"`, 0.5 | `sqrt` is standard for classification, `1/3` for regression |
| `max_samples` | 0.5–1.0 | Bootstrap sample size (if bootstrap=True) |
| `class_weight` | `"balanced"` | For imbalanced classification |
**Note**: Random Forest is less sensitive to hyperparameters than gradient boosting. Default settings often work well. Focus tuning budget on `max_features` and `min_samples_leaf`.
---
## Logistic Regression / Ridge / Lasso
| Parameter | Typical Range | Notes |
|-----------|--------------|-------|
| `C` (LogReg) | 0.001–100 | Inverse of regularization strength; log scale search |
| `alpha` (Ridge/Lasso) | 0.001–100 | Regularization strength; log scale search |
| `solver` | `"lbfgs"`, `"saga"` | `saga` for L1 or large datasets |
| `max_iter` | 100–1000 | Increase if convergence warnings appear |
| `penalty` | `"l1"`, `"l2"`, `"elasticnet"` | L1 gives sparsity; L2 is default |
**Always scale features** before fitting linear models. Use `StandardScaler` or `MinMaxScaler`.
---
## SVM
| Parameter | Typical Range | Notes |
|-----------|--------------|-------|
| `C` | 0.01–100 | Regularization; higher = less regularization |
| `gamma` | `"scale"`, `"auto"`, 0.001–1 | RBF kernel width; `scale` is a good default |
| `kernel` | `"rbf"`, `"linear"`, `"poly"` | `rbf` for nonlinear, `linear` for high-dim sparse |
**Scale features** before SVM. SVMs do not scale to large datasets (>50k rows) — use SGDClassifier instead.
---
## Optuna Search Spaces
```python
def objective(trial):
params = {
"n_estimators": trial.suggest_int("n_estimators", 100, 1000),
"learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
"max_depth": trial.suggest_int("max_depth", 3, 8),
"subsample": trial.suggest_float("subsample", 0.6, 1.0),
"colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),
"min_child_weight": trial.suggest_int("min_child_weight", 1, 10),
}
```
Use `log=True` for parameters that span orders of magnitude (`learning_rate`, `C`, `alpha`).
---
## General Tuning Principles
1. **Use early stopping** for gradient boosting — set `n_estimators` high (1000+) and let early stopping find the right count. Avoids overfitting and saves compute.
2. **Search in log space** for regularization parameters and learning rates — the difference between 0.001 and 0.01 matters as much as 0.1 vs 1.0.
3. **Fix one group at a time** — tune tree structure (`max_depth`, `num_leaves`) first, then sampling params, then regularization, then learning rate.
4. **More data beats more tuning** — hyperparameter tuning typically gives 1–3% improvement; getting more/better data often gives 5–20%.
5. **Don't tune on test set** — use cross-validation or a held-out validation set for tuning. Report test set performance only once, at the end.
|