t22000t/house-prices-tabular-models

House Prices - Tabular Models (8 architectures + interpretability)

Pre-trained models for the t22000t/house-prices-tabular dataset, covering all eight architectures from the tabular-data-modelling-pipeline: two gradient-boosted machines (CatBoost, XGBoost) and six deep-learning models (CANN, CANN-GBM, FT-Transformer, TabM, LocalGLMnet, DRN).

v3 release - adds the full interpretability stack: SHAP TreeExplainer for the GBMs, Captum Integrated Gradients for the DL models, per-layer attention for FT-Transformer, residual analysis for CANN / CANN-GBM, per-row coefficients for LocalGLMnet, and full distributional outputs (mean, variance, quantiles, VaR) for DRN. See INTERPRETABILITY.md and dashboard_dl_interpretability.html.

Results

All 8 architectures trained with default hyperparameters (no Optuna tuning), 3-seed ensembles for the DL models, gamma family + log link.

Rank	Model	Test Gini	Test MAE (USD)	Test RMSE	A/E ratio	n params	Training time
1	XGBoost	0.2049	17,204	29,716	0.999	462 trees	0.3 s
2	CatBoost	0.1996	29,223	42,467	1.161	499 trees	2.1 s
3	LocalGLMnet	0.1991	23,420	41,406	0.988	22,620	4.5 s
4	DRN	0.1962	27,928	50,086	0.981	53,010	4.8 s
5	CANN	0.1941	24,906	40,935	1.024	52,815	8.2 s
6	CANN-GBM	0.1940	32,932	48,574	1.193	52,815	4.2 s
7	FT-Transformer	0.0368	187,771	203,270	3,337.26	483,267	171.7 s
8	TabM	0.0332	187,802	203,299	7,436.72	410,364	126.8 s
-	Stacked ensemble (NNLS)	0.2049	17,204	29,716	0.999	(9 weights)	-

• Test set: 304 rows (20% of 1,460)
• Target: SalePrice (USD)
• Loss: Gamma deviance via reg:gamma (XGBoost) / Tweedie:variance_power=1.99 (CatBoost) / explicit gamma NLL (DL)
• Cap: 99.5th percentile (= $555,355; 6 rows winsorised)
• Random seed: 42

Why FT-Transformer and TabM underfit

With only ~1,150 training rows, transformer-class models have nothing like enough data to learn meaningful attention patterns. They converge to near-constant predictions (MAE ≈ mean target). This is the expected behaviour for these architectures on small tabular datasets - the literature consistently shows they need 10k+ rows to outperform GBMs.

For a comparison where these architectures actually compete, see the companion drop on Bike Sharing (17k rows): t22000t/bike-sharing-tabular-models.

Interpretability (new in v3)

Multiple methods applied side by side so findings can be triangulated:

Method	Applies to	What it measures
SHAP TreeExplainer	CatBoost, XGBoost	Per-row Shapley contribution of each feature to the model's log-prediction.
Native importance (CatBoost / XGBoost)	CatBoost, XGBoost	Loss reduction (CatBoost) / gain (XGBoost).
Captum Integrated Gradients	All 6 DL architectures	Gradient-based attribution for continuous features, averaged over 50 integration steps.
FT-Transformer attention	FT-Transformer	Per-layer multi-head self-attention weights, averaged across heads.
CANN / CANN-GBM residual analysis	CANN, CANN-GBM	Distribution of the neural network's correction to the GBM/GLM base. Here: mean=0.005, std=0.107 -> the NN added almost no correction beyond what the base already captured.
LocalGLMnet coefficients	LocalGLMnet	Per-row linear coefficients - exposes how the effective regression formula varies across the input space.
DRN distributional output	DRN	Full predicted gamma distribution per row. Here: mean shape ≈ 1.04, CoV ≈ 0.98, VaR95 ≈ $1.18M, VaR99 ≈ $2.5M. The wide upper tail is consistent with House Prices' heavy right skew.

Cross-method consensus (the high-confidence finding)

Both CatBoost and XGBoost ranked the following four features in their top-5 most important for predicting SalePrice. Cross-method agreement is a strong signal - independent methods identifying the same drivers means the finding is unlikely to be a method-specific artefact:

Feature	CatBoost	XGBoost	Interpretation
TotalSF (derived)	0.096	0.138	Total square footage (basement + 1st + 2nd floor). Dominant predictor across both models.
OverallQual	0.043	0.103	Material and finish quality 1-10. Strongly monotonic with price.
GrLivArea	0.043	0.032	Above-grade living area sq ft.
YearRemodAdd	0.038	0.025	Year of most recent remodel - matters more than YearBuilt.

Note the derived feature TotalSF (defined in configs/example_house_prices.py as TotalBsmtSF + 1stFlrSF + 2ndFlrSF) is more predictive than any of its components individually - the pipeline's derived_features mechanism paid off here.

Full breakdown - per-method top-10 tables, LocalGLMnet coefficient distributions, sign-stability analysis - is in INTERPRETABILITY.md. For the interactive plots see dashboard_dl_interpretability.html.

Files

File	What it is
catboost.cbm	Trained CatBoost (Tweedie:variance_power=1.99)
xgboost.json	Trained XGBoost Booster (reg:gamma)
cann_member{0,1,2}.pt	CANN 3-seed ensemble
cann_gbm_member{0,1,2}.pt	CANN-GBM 3-seed ensemble
ft_transformer_member{0,1,2}.pt	FT-Transformer 3-seed ensemble
tabm_member{0,1,2}.pt	TabM 3-seed ensemble
localglmnet_member{0,1,2}.pt	LocalGLMnet 3-seed ensemble
drn_member{0,1,2}.pt	DRN 3-seed ensemble
evaluation_summary.csv	Per-model train/test metrics
ensemble_weights.json	NNLS weights over the 8 base predictions
dashboard_dl_models.html	Performance dashboard (Lorenz, calibration, A/P scatter)
dashboard_dl_interpretability.html	Interpretability dashboard (SHAP, IG, attention, residuals)
feature_importance.csv	Consolidated importances across CatBoost + XGBoost
localglmnet_coefficients.csv	LocalGLMnet per-row coefficients (304 rows × 16 continuous features)
drn_distributional_outputs.csv	DRN per-row distributional moments (mean, variance, VaR95, VaR99)
INTERPRETABILITY.md	Human-readable interpretability summary report
figures/fig_dl_*.png	Standalone publication figures (incl. attention heatmap)
model_summary.json	Structured run record

Loading and inference

CatBoost (single best non-XGB baseline)

from huggingface_hub import hf_hub_download
from catboost import CatBoostRegressor
import pandas as pd

path = hf_hub_download("t22000t/house-prices-tabular-models", "catboost.cbm")
model = CatBoostRegressor()
model.load_model(path)

df = pd.read_csv("hf://datasets/t22000t/house-prices-tabular/train.csv")
features = [  # see configs/example_house_prices.py for the full list
    "LotArea", "YearBuilt", "YearRemodAdd", "TotalBsmtSF", "1stFlrSF",
    "2ndFlrSF", "GrLivArea", "FullBath", "BedroomAbvGr", "TotRmsAbvGrd",
    "GarageCars", "GarageArea", "OverallQual", "OverallCond",
    "MSZoning", "Street", "LotShape", "Neighborhood", "BldgType",
    "HouseStyle", "RoofStyle", "ExterQual", "Foundation", "Heating",
    "CentralAir", "KitchenQual", "SaleType", "SaleCondition",
]
preds = model.predict(df[features])

XGBoost (best overall on this dataset)

from huggingface_hub import hf_hub_download
import xgboost as xgb

path = hf_hub_download("t22000t/house-prices-tabular-models", "xgboost.json")
booster = xgb.Booster()
booster.load_model(path)

XGBoost predictions need the exact feature order used at training time. Clone the pipeline repo to reproduce the preprocessing path.

Deep-learning models (CANN, CANN-GBM, FT-T, TabM, LocalGLMnet, DRN)

Each architecture ships as a 3-seed ensemble of PyTorch state-dicts. Reconstructing them requires the matching architecture class from the pipeline package. Easiest path:

git clone https://github.com/timothy22000/tabular_data_modelling_pipeline
cd tabular_data_modelling_pipeline
pip install -e ".[all]"

Then load each ensemble member's state dict into the right module (see modelling/models/cann.py etc).

Training configuration

Setting	Value
Pipeline	tabular-data-modelling-pipeline v0.1.0
Architectures	All 8 (catboost, xgboost, cann, cann_gbm, ft_transformer, tabm, localglmnet, drn)
Hyperparameters	Defaults - no Optuna tuning
DL ensemble size	3 seeds per architecture
Family / link	Gamma / log
XGBoost objective	reg:gamma
CatBoost loss	Tweedie:variance_power=1.99 (CatBoost has no native Gamma loss)
Train/test split	Random 80/20, seed 42
Cap percentile	99.5
Hardware	Apple M-series, MPS device for DL
Total wall-clock	~5.4 min

To reproduce:

git clone https://github.com/timothy22000/tabular_data_modelling_pipeline
cd tabular_data_modelling_pipeline
pip install -e ".[all]"
python scripts/download_data.py --dataset house_prices

OMP_NUM_THREADS=1 python train.py \
    --config configs/example_house_prices.py \
    --input data/house_prices.csv \
    --skip-tuning --skip-interpretability \
    --architectures catboost xgboost cann cann_gbm ft_transformer tabm localglmnet drn

(OMP_NUM_THREADS=1 is only needed on macOS arm64; Linux is unaffected.)

Limitations

• No Optuna tuning. Defaults only. With tuning, expect ~0.02-0.04 Gini lift across the board.
• FT-Transformer & TabM underfit. Dataset is too small (~1.2k training rows) - see the companion Bike Sharing drop for a fair comparison.
• No interpretability artefacts. Captum attributions and partial-dependence plots were skipped to keep wall-clock manageable. The pipeline supports them - drop --skip-interpretability to compute on re-run.
• Random split, not stratified. SalePrice is right-tailed; a quantile-stratified split would give a more representative test set.
• Trained on Kaggle's train.csv only (test.csv is unlabelled). Not directly comparable to the Kaggle leaderboard.

Intended use

• Baseline for tabular DL research on small-N regression problems.
• Teaching end-to-end tabular pricing pipelines (8 architectures with one CLI).
• Sanity check for reimplementations of any of the 8 architectures on gamma-family data.

Citation

@software{tabular_data_modelling_pipeline,
  author = {Mun, Timothy},
  title  = {tabular-data-modelling-pipeline},
  url    = {https://github.com/timothy22000/tabular_data_modelling_pipeline},
  year   = {2026}
}

@article{decock2011ames,
  author  = {De Cock, Dean},
  title   = {Ames, Iowa: Alternative to the Boston Housing Data},
  journal = {Journal of Statistics Education},
  volume  = {19},
  number  = {3},
  year    = {2011}
}

Please also cite the individual architecture papers - see the main repo README for the full reference list.

License

MIT for the model code and pipeline. Underlying dataset under Kaggle competition terms (free use with attribution).

Related

• 📂 Dataset: t22000t/house-prices-tabular
• 🤖 Companion: t22000t/bike-sharing-tabular-models - Poisson family, 17k rows
• 📦 Pipeline: tabular-data-modelling-pipeline