Publish the Covertype XGBoost classifier

Replaces the breast-cancer demo with a 400-tree model trained on all 581,012 UCI Covertype rows. Test accuracy 0.9109 and macro F1 0.8942, scored once against a majority baseline and a small XGBoost baseline.

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -7,57 +7,72 @@ tags:
   - tabular
   - bleu-js
 datasets:
-  - breast_cancer
+  - covertype
 metrics:
   - accuracy
   - f1
-  - roc_auc
 model-index:
   - name: bleu-xgboost-classifier
     results:
       - task:
-          type: binary-classification
+          type: multiclass-classification
         dataset:
-          name: Breast Cancer Wisconsin (Diagnostic)
-          type: breast_cancer
+          name: UCI Covertype
+          type: covertype
         metrics:
           - type: accuracy
-            value: 0.9473684210526315
-          - type: f1
-            value: 0.958904109589041
-          - type: roc_auc
-            value: 0.9923941798941799
+            value: 0.9108887033897576
+          - type: macro_f1
+            value: 0.8941719351718758
+          - type: weighted_f1
+            value: 0.9115377043223944
 ---
 
 # bleu-xgboost-classifier
 
-XGBoost binary classifier trained by the Bleu.js SDK trainer `bleujs.ml.HybridTrainer`.
+Multiclass XGBoost model that predicts forest cover type from cartographic measurements. Trained for the Bleu.js open-source SDK on the full public UCI Covertype table: **581,012 rows**, **54 features**, **7 cover types**.
 
-This is a small public-dataset model. It is not the hosted Bleu.js API. That API is `https://api.bleujs.org`. The default `pip install bleu-js` package does not include this model. Load the weights below, or train your own data with `pip install "bleu-js[ml]"`.
+This is not the hosted Bleu.js API. That API is `https://api.bleujs.org`. `pip install bleu-js` does not download these weights.
 
-## Result
+## Held-out test result
 
-Held-out test set only. Train/test split is 80/20, `random_state=42`, stratified.
+The test split is 20% of the table (116,203 rows), stratified, `random_state=42`. It was scored once. Tree count was chosen on an inner 10% slice of the training rows only. Early stopping did not fire, so the published model uses the full cap of **400 trees**.
 
-| Metric | Test score |
-| --- | --- |
-| Accuracy | 0.9474 (108/114) |
-| F1 | 0.9589 |
-| ROC AUC | 0.9924 |
+| Model | Accuracy | Macro F1 | Weighted F1 |
+| --- | ---: | ---: | ---: |
+| This model | 0.9109 | 0.8942 | 0.9115 |
+| Small XGBoost baseline (depth 3, 50 trees, no class weights) | 0.7319 | 0.5328 | 0.7167 |
+| Majority class | 0.4876 | 0.0937 | 0.3197 |
 
-455 training rows, 114 test rows, 30 numeric features. Class `1` is benign and class `0` is malignant, which is the scikit-learn encoding of this dataset. The benign rate is about 0.63 in both splits.
+Log loss on the test rows is 0.2391 for this model and 0.6358 for the small baseline.
 
-These scores are for this split of this dataset. They are not a medical-device claim and they do not transfer to other data.
+Per-class F1 on the same test rows:
 
-## Data and training
+| Cover type | F1 | Support |
+| --- | ---: | ---: |
+| Spruce/Fir | 0.9057 | 42,368 |
+| Lodgepole Pine | 0.9164 | 56,661 |
+| Ponderosa Pine | 0.9371 | 7,151 |
+| Cottonwood/Willow | 0.8942 | 549 |
+| Aspen | 0.7673 | 1,899 |
+| Douglas-fir | 0.8961 | 3,473 |
+| Krummholz | 0.9425 | 4,102 |
 
-- **Dataset:** Breast Cancer Wisconsin (Diagnostic), from the UCI Machine Learning Repository, loaded with `sklearn.datasets.load_breast_cancer`.
-- **Trainer:** `HybridTrainer(model_type="xgboost")` in `src/bleujs/ml.py`. Quantum features were off.
-- **Hyperparameters:** `max_depth=6`, `learning_rate=0.1`, `n_estimators=100`, `objective=binary:logistic`.
-- **XGBoost:** 3.2.0. No feature scaler. The trainer fits the classifier on the raw feature values.
-- **File:** `model.json` is the XGBoost native save, not a pickle. `metrics.json` records the split, feature names, and scores.
+Aspen is the weak class. Lodgepole Pine is almost half the table, which is why accuracy alone is a poor summary and macro F1 is the number to quote.
 
-## Load the weights
+The largest gain features are Cache la Poudre wilderness, several soil-type indicators, and elevation. The full list is in `metrics.json`.
+
+## What the weights are
+
+- `model.json` — XGBoost native format, not a pickle. About 29 MB.
+- `metrics.json` — split, hyperparameters, baselines, per-class scores, feature gain.
+- `train.py` — the program that produced this file. Run it again to reproduce the scores.
+
+Settings: `max_depth=8`, `learning_rate=0.1`, `subsample=0.8`, `colsample_bytree=0.8`, `min_child_weight=5`, `reg_lambda=1.0`, `tree_method=hist`, `objective=multi:softprob`, balanced sample weights, XGBoost 3.2.0.
+
+`bleujs.ml.HybridTrainer` fits a binary classifier (`binary:logistic`). This model has seven classes, so it is trained with XGBoost directly. The SDK trainer remains the path for your own binary tables: `pip install "bleu-js[ml]"`.
+
+## Load
 
 ```bash
 pip install xgboost huggingface_hub
@@ -74,30 +89,19 @@ path = hf_hub_download(
 model = xgb.XGBClassifier()
 model.load_model(path)
 
-# X is (n_samples, 30), in the feature order stored in metrics.json
+# X shape (n_samples, 54), column order in metrics.json
+# class 0 is Spruce/Fir ... class 6 is Krummholz (UCI codes 1..7)
 pred = model.predict(X)
-proba = model.predict_proba(X)
-```
-
-## Train the same way on your own table
-
-```bash
-pip install "bleu-js[ml]"
 ```
 
-```python
-from bleujs.ml import HybridTrainer
-
-trainer = HybridTrainer(model_type="xgboost")
-model = trainer.train(X_train, y_train)
-```
+## Data
 
-Python 3.11, 3.12, or 3.13. Source: [HelloblueAI/Bleu.js](https://github.com/HelloblueAI/Bleu.js).
+Covertype, UCI Machine Learning Repository, loaded with `sklearn.datasets.fetch_covtype`. Four wilderness areas in Roosevelt National Forest, Colorado. Copyright Jock A. Blackard and Colorado State University. The model card, training script, and published weights are MIT. The dataset keeps its own terms.
 
-## License
+## Out of scope
 
-MIT. The Wisconsin breast cancer data is distributed with scikit-learn from the UCI repository.
+These scores are for this split of this Colorado table. They are not a map product, not a claim about other forests, and not a leaderboard entry. No search was run against the test split.
 
 ## Contact
 
-support@helloblue.ai
+support@helloblue.ai · [HelloblueAI/Bleu.js](https://github.com/HelloblueAI/Bleu.js)

metrics.json

@@ -1,65 +1,297 @@
 {
-  "dataset": "breast_cancer",
-  "dataset_name": "Breast Cancer Wisconsin (Diagnostic)",
-  "source": "sklearn.datasets.load_breast_cancer",
-  "origin": "UCI Machine Learning Repository, via scikit-learn",
-  "task": "binary classification",
-  "positive_class": "benign",
-  "negative_class": "malignant",
-  "feature_names": [
-    "mean radius",
-    "mean texture",
-    "mean perimeter",
-    "mean area",
-    "mean smoothness",
-    "mean compactness",
-    "mean concavity",
-    "mean concave points",
-    "mean symmetry",
-    "mean fractal dimension",
-    "radius error",
-    "texture error",
-    "perimeter error",
-    "area error",
-    "smoothness error",
-    "compactness error",
-    "concavity error",
-    "concave points error",
-    "symmetry error",
-    "fractal dimension error",
-    "worst radius",
-    "worst texture",
-    "worst perimeter",
-    "worst area",
-    "worst smoothness",
-    "worst compactness",
-    "worst concavity",
-    "worst concave points",
-    "worst symmetry",
-    "worst fractal dimension"
+  "dataset": "UCI Covertype",
+  "source": "sklearn.datasets.fetch_covtype",
+  "citation": "Blackard, Jock A. Covertype. UCI Machine Learning Repository.",
+  "n_rows": 581012,
+  "n_features": 54,
+  "class_names": [
+    "Spruce/Fir",
+    "Lodgepole Pine",
+    "Ponderosa Pine",
+    "Cottonwood/Willow",
+    "Aspen",
+    "Douglas-fir",
+    "Krummholz"
   ],
+  "label_encoding": "model class i is UCI cover type i+1",
   "split": {
     "test_size": 0.2,
+    "inner_val_size": 0.1,
     "random_state": 42,
     "stratify": true
   },
-  "trainer": "bleujs.ml.HybridTrainer",
-  "quantum_components": false,
+  "n_train": 464809,
+  "n_test": 116203,
+  "n_estimators_selected_on_inner_val": 400,
   "hyperparameters": {
-    "max_depth": 6,
+    "max_depth": 8,
     "learning_rate": 0.1,
-    "n_estimators": 100,
-    "objective": "binary:logistic"
+    "subsample": 0.8,
+    "colsample_bytree": 0.8,
+    "min_child_weight": 5,
+    "reg_lambda": 1.0,
+    "tree_method": "hist",
+    "objective": "multi:softprob",
+    "num_class": 7,
+    "random_state": 42,
+    "n_jobs": -1,
+    "n_estimators": 400,
+    "sample_weight": "balanced"
   },
   "xgboost_version": "3.2.0",
-  "metrics_on_held_out_test": {
-    "accuracy": 0.9473684210526315,
-    "f1": 0.958904109589041,
-    "roc_auc": 0.9923941798941799,
-    "n_train": 455,
-    "n_test": 114,
-    "n_features": 30,
-    "positive_rate_train": 0.6263736263736264,
-    "positive_rate_test": 0.631578947368421
-  }
+  "trainer_note": "XGBClassifier. bleujs.ml.HybridTrainer targets binary classification with binary:logistic, so this multiclass model is trained with the same library and a documented multiclass objective.",
+  "test_metrics": {
+    "accuracy": 0.9108887033897576,
+    "macro_f1": 0.8941719351718758,
+    "weighted_f1": 0.9115377043223944,
+    "log_loss": 0.23914723098278046
+  },
+  "simple_xgboost_baseline": {
+    "accuracy": 0.7318916034870012,
+    "macro_f1": 0.5327755702159982,
+    "weighted_f1": 0.7166892879546547,
+    "log_loss": 0.6357948184013367
+  },
+  "majority_baseline": {
+    "accuracy": 0.48760359026875383,
+    "macro_f1": 0.09365083038028243,
+    "weighted_f1": 0.3196513678755306,
+    "log_loss": 8.168821334838867
+  },
+  "per_class_f1": {
+    "Spruce/Fir": 0.9057362118202865,
+    "Lodgepole Pine": 0.9163635050597145,
+    "Ponderosa Pine": 0.9370629370629371,
+    "Cottonwood/Willow": 0.8941798941798942,
+    "Aspen": 0.767287784679089,
+    "Douglas-fir": 0.8960845413900556,
+    "Krummholz": 0.9424886720111537
+  },
+  "per_class_support": {
+    "Spruce/Fir": 42368,
+    "Lodgepole Pine": 56661,
+    "Ponderosa Pine": 7151,
+    "Cottonwood/Willow": 549,
+    "Aspen": 1899,
+    "Douglas-fir": 3473,
+    "Krummholz": 4102
+  },
+  "feature_importance_gain": [
+    {
+      "feature": "Wilderness_Area_3",
+      "gain": 249.5696563720703
+    },
+    {
+      "feature": "Soil_Type_38",
+      "gain": 129.66807556152344
+    },
+    {
+      "feature": "Soil_Type_11",
+      "gain": 120.6527328491211
+    },
+    {
+      "feature": "Soil_Type_2",
+      "gain": 113.96896362304688
+    },
+    {
+      "feature": "Elevation",
+      "gain": 106.00982666015625
+    },
+    {
+      "feature": "Soil_Type_9",
+      "gain": 95.48499298095703
+    },
+    {
+      "feature": "Soil_Type_29",
+      "gain": 79.35009765625
+    },
+    {
+      "feature": "Soil_Type_3",
+      "gain": 69.28760528564453
+    },
+    {
+      "feature": "Wilderness_Area_0",
+      "gain": 67.1938247680664
+    },
+    {
+      "feature": "Soil_Type_39",
+      "gain": 65.25533294677734
+    },
+    {
+      "feature": "Soil_Type_37",
+      "gain": 62.631927490234375
+    },
+    {
+      "feature": "Soil_Type_17",
+      "gain": 60.56175231933594
+    },
+    {
+      "feature": "Soil_Type_1",
+      "gain": 53.69615936279297
+    },
+    {
+      "feature": "Soil_Type_34",
+      "gain": 49.5882453918457
+    },
+    {
+      "feature": "Soil_Type_21",
+      "gain": 46.41263198852539
+    },
+    {
+      "feature": "Wilderness_Area_2",
+      "gain": 43.265750885009766
+    },
+    {
+      "feature": "Soil_Type_12",
+      "gain": 40.007808685302734
+    },
+    {
+      "feature": "Soil_Type_16",
+      "gain": 38.801177978515625
+    },
+    {
+      "feature": "Soil_Type_31",
+      "gain": 34.57275390625
+    },
+    {
+      "feature": "Wilderness_Area_1",
+      "gain": 31.31096839904785
+    },
+    {
+      "feature": "Soil_Type_5",
+      "gain": 30.942785263061523
+    },
+    {
+      "feature": "Soil_Type_4",
+      "gain": 28.343896865844727
+    },
+    {
+      "feature": "Soil_Type_30",
+      "gain": 24.622051239013672
+    },
+    {
+      "feature": "Soil_Type_10",
+      "gain": 24.064062118530273
+    },
+    {
+      "feature": "Soil_Type_0",
+      "gain": 23.00113296508789
+    },
+    {
+      "feature": "Soil_Type_32",
+      "gain": 22.943172454833984
+    },
+    {
+      "feature": "Soil_Type_22",
+      "gain": 22.0693302154541
+    },
+    {
+      "feature": "Soil_Type_28",
+      "gain": 21.283544540405273
+    },
+    {
+      "feature": "Soil_Type_23",
+      "gain": 19.36112403869629
+    },
+    {
+      "feature": "Hillshade_9am",
+      "gain": 18.579858779907227
+    },
+    {
+      "feature": "Soil_Type_19",
+      "gain": 17.969446182250977
+    },
+    {
+      "feature": "Horizontal_Distance_To_Hydrology",
+      "gain": 17.332067489624023
+    },
+    {
+      "feature": "Soil_Type_36",
+      "gain": 16.103273391723633
+    },
+    {
+      "feature": "Horizontal_Distance_To_Roadways",
+      "gain": 15.786291122436523
+    },
+    {
+      "feature": "Soil_Type_25",
+      "gain": 15.337019920349121
+    },
+    {
+      "feature": "Horizontal_Distance_To_Fire_Points",
+      "gain": 14.771939277648926
+    },
+    {
+      "feature": "Soil_Type_13",
+      "gain": 14.268491744995117
+    },
+    {
+      "feature": "Soil_Type_18",
+      "gain": 12.966172218322754
+    },
+    {
+      "feature": "Soil_Type_33",
+      "gain": 11.3596830368042
+    },
+    {
+      "feature": "Vertical_Distance_To_Hydrology",
+      "gain": 11.144617080688477
+    },
+    {
+      "feature": "Aspect",
+      "gain": 10.377111434936523
+    },
+    {
+      "feature": "Soil_Type_8",
+      "gain": 9.88647174835205
+    },
+    {
+      "feature": "Hillshade_Noon",
+      "gain": 9.631880760192871
+    },
+    {
+      "feature": "Soil_Type_27",
+      "gain": 9.097264289855957
+    },
+    {
+      "feature": "Slope",
+      "gain": 8.322596549987793
+    },
+    {
+      "feature": "Hillshade_3pm",
+      "gain": 8.168411254882812
+    },
+    {
+      "feature": "Soil_Type_14",
+      "gain": 7.7810516357421875
+    },
+    {
+      "feature": "Soil_Type_20",
+      "gain": 7.463456630706787
+    },
+    {
+      "feature": "Soil_Type_15",
+      "gain": 7.146697998046875
+    },
+    {
+      "feature": "Soil_Type_26",
+      "gain": 7.062173366546631
+    },
+    {
+      "feature": "Soil_Type_24",
+      "gain": 4.399511814117432
+    },
+    {
+      "feature": "Soil_Type_35",
+      "gain": 3.7608084678649902
+    },
+    {
+      "feature": "Soil_Type_7",
+      "gain": 1.7688950300216675
+    },
+    {
+      "feature": "Soil_Type_6",
+      "gain": 1.6422888040542603
+    }
+  ]
 }

train.py

@@ -0,0 +1,188 @@
+"""Train and score the Bleu.js forest-cover XGBoost classifier.
+
+Protocol, fixed before the test set is scored:
+- Data: sklearn.datasets.fetch_covtype (UCI Covertype, full table)
+- Labels shifted from 1..7 to 0..6
+- Outer split: stratified 80/20, random_state=42. The 20% is scored once.
+- Inner split: stratified 90/10 of the training rows, random_state=42,
+  used only to pick n_estimators with early stopping.
+- Final fit: all training rows, n_estimators = best_iteration + 1,
+  sample_weight = balanced class weights from the training rows.
+- Baselines, same test rows, no retuning: majority class; a depth-3
+  XGBoost with 50 trees and no class weights.
+"""
+
+import json
+from pathlib import Path
+
+import numpy as np
+import xgboost as xgb
+from sklearn.datasets import fetch_covtype
+from sklearn.metrics import (
+    accuracy_score,
+    classification_report,
+    f1_score,
+    log_loss,
+)
+from sklearn.model_selection import train_test_split
+from sklearn.utils.class_weight import compute_sample_weight
+
+OUT = Path(__file__).resolve().parent
+RANDOM_STATE = 42
+CLASS_NAMES = [
+    "Spruce/Fir",
+    "Lodgepole Pine",
+    "Ponderosa Pine",
+    "Cottonwood/Willow",
+    "Aspen",
+    "Douglas-fir",
+    "Krummholz",
+]
+FEATURE_NAMES = [
+    "Elevation",
+    "Aspect",
+    "Slope",
+    "Horizontal_Distance_To_Hydrology",
+    "Vertical_Distance_To_Hydrology",
+    "Horizontal_Distance_To_Roadways",
+    "Hillshade_9am",
+    "Hillshade_Noon",
+    "Hillshade_3pm",
+    "Horizontal_Distance_To_Fire_Points",
+    *[f"Wilderness_Area_{i}" for i in range(4)],
+    *[f"Soil_Type_{i}" for i in range(40)],
+]
+
+
+def scores(y_true, pred, proba):
+    return {
+        "accuracy": float(accuracy_score(y_true, pred)),
+        "macro_f1": float(f1_score(y_true, pred, average="macro")),
+        "weighted_f1": float(f1_score(y_true, pred, average="weighted")),
+        "log_loss": float(log_loss(y_true, proba, labels=list(range(7)))),
+    }
+
+
+def main():
+    bunch = fetch_covtype(as_frame=False)
+    X = np.asarray(bunch.data, dtype=np.float32)
+    y = np.asarray(bunch.target, dtype=np.int32) - 1
+    if X.shape[1] != len(FEATURE_NAMES) or y.min() != 0 or y.max() != 6:
+        raise SystemExit(f"Unexpected covertype shape {X.shape} labels {y.min()}..{y.max()}")
+
+    X_train, X_test, y_train, y_test = train_test_split(
+        X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y
+    )
+    X_fit, X_val, y_fit, y_val = train_test_split(
+        X_train, y_train, test_size=0.1, random_state=RANDOM_STATE, stratify=y_train
+    )
+
+    common = dict(
+        max_depth=8,
+        learning_rate=0.1,
+        subsample=0.8,
+        colsample_bytree=0.8,
+        min_child_weight=5,
+        reg_lambda=1.0,
+        tree_method="hist",
+        objective="multi:softprob",
+        num_class=7,
+        random_state=RANDOM_STATE,
+        n_jobs=-1,
+    )
+    search = xgb.XGBClassifier(n_estimators=400, early_stopping_rounds=20, **common)
+    search.fit(
+        X_fit,
+        y_fit,
+        sample_weight=compute_sample_weight("balanced", y_fit),
+        eval_set=[(X_val, y_val)],
+        verbose=False,
+    )
+    n_trees = int(search.best_iteration) + 1
+
+    model = xgb.XGBClassifier(n_estimators=n_trees, **common)
+    model.fit(
+        X_train,
+        y_train,
+        sample_weight=compute_sample_weight("balanced", y_train),
+        verbose=False,
+    )
+
+    simple = xgb.XGBClassifier(
+        n_estimators=50,
+        max_depth=3,
+        learning_rate=0.1,
+        tree_method="hist",
+        objective="multi:softprob",
+        num_class=7,
+        random_state=RANDOM_STATE,
+        n_jobs=-1,
+    )
+    simple.fit(X_train, y_train, verbose=False)
+
+    proba = model.predict_proba(X_test)
+    pred = proba.argmax(axis=1)
+    simple_proba = simple.predict_proba(X_test)
+    simple_pred = simple_proba.argmax(axis=1)
+    majority = int(np.bincount(y_train).argmax())
+    majority_pred = np.full_like(y_test, majority)
+    majority_proba = np.zeros_like(proba)
+    majority_proba[:, majority] = 1.0
+
+    report = classification_report(
+        y_test, pred, target_names=CLASS_NAMES, output_dict=True, digits=4
+    )
+    gain = model.get_booster().get_score(importance_type="gain")
+    importance = sorted(
+        (
+            {"feature": FEATURE_NAMES[int(k[1:])], "gain": float(v)}
+            for k, v in gain.items()
+        ),
+        key=lambda row: row["gain"],
+        reverse=True,
+    )
+
+    payload = {
+        "dataset": "UCI Covertype",
+        "source": "sklearn.datasets.fetch_covtype",
+        "citation": "Blackard, Jock A. Covertype. UCI Machine Learning Repository.",
+        "n_rows": int(X.shape[0]),
+        "n_features": int(X.shape[1]),
+        "class_names": CLASS_NAMES,
+        "label_encoding": "model class i is UCI cover type i+1",
+        "split": {
+            "test_size": 0.2,
+            "inner_val_size": 0.1,
+            "random_state": RANDOM_STATE,
+            "stratify": True,
+        },
+        "n_train": int(len(y_train)),
+        "n_test": int(len(y_test)),
+        "n_estimators_selected_on_inner_val": n_trees,
+        "hyperparameters": {**common, "n_estimators": n_trees, "sample_weight": "balanced"},
+        "xgboost_version": xgb.__version__,
+        "trainer_note": (
+            "XGBClassifier. bleujs.ml.HybridTrainer targets binary classification "
+            "with binary:logistic, so this multiclass model is trained with the "
+            "same library and a documented multiclass objective."
+        ),
+        "test_metrics": scores(y_test, pred, proba),
+        "simple_xgboost_baseline": scores(y_test, simple_pred, simple_proba),
+        "majority_baseline": scores(y_test, majority_pred, majority_proba),
+        "per_class_f1": {name: float(report[name]["f1-score"]) for name in CLASS_NAMES},
+        "per_class_support": {name: int(report[name]["support"]) for name in CLASS_NAMES},
+        "feature_importance_gain": importance,
+    }
+    model.save_model(OUT / "model.json")
+    (OUT / "metrics.json").write_text(json.dumps(payload, indent=2))
+    print(json.dumps({
+        "n_trees": n_trees,
+        "test": payload["test_metrics"],
+        "simple": payload["simple_xgboost_baseline"],
+        "majority": payload["majority_baseline"],
+        "per_class_f1": payload["per_class_f1"],
+    }, indent=2))
+
+
+if __name__ == "__main__":
+    main()