Upload REPORT.md

REPORT.md

@@ -0,0 +1,405 @@
+# STAT 5430 Course Project Report
+## Acute GVHD Severity Prediction: Binary Classification Using Pre-Transplant Variables
+
+---
+
+## 1. Project Background and Objectives
+
+### 1.1 Research Question
+Acute graft-versus-host disease (aGVHD) is one of the most common complications after hematopoietic cell transplantation (HCT), significantly impacting patient survival and quality of life. This project aims to build a binary classification model using **only pre-transplant variables** to predict whether a patient will develop **severe (Grade 3-4) aGVHD**.
+
+### 1.2 Clinical Significance
+- Severe aGVHD incidence: ~16.3%, but with high mortality
+- Pre-transplant prediction can help physicians adjust conditioning regimens and intensify monitoring
+- Using only pre-transplant variables means the model can be deployed before transplantation, making it clinically practical
+
+### 1.3 Data Sources
+- **Main data**: `Cleaned_Data_Final.csv` (8,027 cases, 227 variables)
+- **Data dictionary**: `3.xlsx` (variable definitions, types, GVHD relevance annotations)
+- Data from CIBMTR (Center for International Blood and Marrow Transplant Research)
+
+---
+
+## 2. Data Preprocessing
+
+### 2.1 Label Construction
+Following course requirements, construct three-class labels using `agvhd24` and `agvhd34`:
+
+| agvhd34 | agvhd24 | GVHD Grade | Label |
+|---------|---------|------------|-------|
+| 1 | - | Severe | 3 |
+| 0 | 1 | Moderate | 2 |
+| 0 | 0 | Mild | 1 |
+
+**Binary target**: `target = (agvhd34 == 1)`, i.e., **Severe vs Non-severe**
+
+### 2.2 Data Cleaning
+- Replace missing value code `905` with NaN
+- Remove post-transplant variables (cgvhd, grfs, trm, etc.)
+- Keep only pre-transplant variables: 179 features
+- Remove features with >50% missing (none in practice)
+- Fill numerical missing values with median
+
+### 2.3 Data Splitting
+- Training set: 6,421 cases (80%)
+- Test set: 1,606 cases (20%)
+- Stratified sampling to maintain Severe proportion (16.3%)
+
+### 2.4 Class Imbalance Handling
+- Severe: 1,307 cases (16.3%)
+- Non-severe: 6,720 cases (83.7%)
+- Use `scale_pos_weight = 5.14` for cost-sensitive learning
+
+---
+
+## 3. Modeling Methods
+
+This project attempted **7 different methods**, from traditional machine learning to cutting-edge deep learning:
+
+### Method 1: TabNet (Interpretable Neural Network)
+**Principle**: Attention-based tabular deep learning model that selects features through sequential attention masks, providing instance-level interpretability.
+
+**Configuration**:
+```python
+n_d=64, n_a=64, n_steps=5, gamma=1.5
+lambda_sparse=1e-4, lr=1e-3, batch_size=512
+```
+
+**Characteristics**:
+- Built-in feature importance
+- Single-sample attention visualization
+- Poor performance on this dataset (AUC=0.52)
+
+**Failure Analysis**:
+- 179-dimensional features too sparse for TabNet
+- Lack of effective feature selection mechanism
+- Medical tabular data noise overwhelms signal
+
+---
+
+### Method 2: LightGBM (Gradient Boosting)
+**Principle**: Histogram-based gradient boosting framework by Microsoft, strong at handling class imbalance.
+
+**Configuration**:
+```python
+n_estimators=1000, learning_rate=0.05, max_depth=6
+num_leaves=31, subsample=0.8, colsample_bytree=0.8
+scale_pos_weight=5.14
+```
+
+---
+
+### Method 3: XGBoost (Extreme Gradient Boosting)
+**Principle**: Gradient boosting library by Tianqi Chen, stronger regularization to prevent overfitting.
+
+**Configuration**:
+```python
+n_estimators=1000, learning_rate=0.05, max_depth=5
+subsample=0.8, colsample_bytree=0.8
+scale_pos_weight=5.14
+```
+
+---
+
+### Method 4: CatBoost (Categorical Boosting)
+**Principle**: Developed by Yandex, native categorical feature handling, Ordered Boosting reduces overfitting.
+
+**Configuration**:
+```python
+iterations=1000, learning_rate=0.05, depth=6
+l2_leaf_reg=3, scale_pos_weight=5.14
+```
+
+---
+
+### Method 5: Ensemble (Stacking)
+**Principle**: Combine predictions from multiple base learners to reduce variance and improve generalization.
+
+**Two ensemble strategies**:
+1. Simple average: `(LGB + XGB + CAT) / 3`
+2. Weighted average: weighted by validation AUC
+
+---
+
+### Method 6: AutoGluon (AutoML)
+**Principle**: AWS automatic machine learning framework with multi-layer stacking and hyperparameter search.
+
+**Attempted configurations**:
+- `presets='best_quality'`: multi-layer stacking + bagging
+- `presets='good_quality'`: fast high quality
+
+**Issues**:
+- Complex dependency installation (lightgbm/catboost/xgboost/torch)
+- GPU environment configuration difficulties
+- Failed to run successfully in sandbox environment
+
+---
+
+### Method 7: Conformal Prediction (Uncertainty Quantification)
+**Principle**: Non-parametric statistical method constructing prediction sets with **guaranteed coverage**.
+
+**Steps**:
+1. Train model on training set
+2. Compute non-conformity scores on calibration set: `score = 1 - p(true_class)`
+3. Calculate quantile threshold
+4. Construct prediction sets on test set
+
+**Clinical significance**:
+- High-confidence samples: adopt model prediction directly
+- Low-confidence samples: recommend manual review
+- Coverage rate ≥90%, controlling misdiagnosis risk
+
+---
+
+## 4. Experimental Results
+
+### 4.1 Model Performance Comparison
+
+| Rank | Model | AUC | Notes |
+|------|-------|-----|-------|
+| 1 | **CatBoost** | **0.6885** | Best single model |
+| 2 | Ensemble (weighted) | 0.6861 | No significant improvement |
+| 3 | Ensemble (average) | 0.6856 | Close to CatBoost |
+| 4 | LightGBM | 0.6646 | Baseline tree model |
+| 5 | XGBoost | 0.6553 | Over-regularization |
+| 6 | TabNet | 0.5153 | Not suitable for this data |
+
+### 4.2 Key Findings
+
+**1. Why CatBoost is optimal**
+- Native categorical feature handling (race, marital status)
+- Ordered Boosting reduces target leakage
+- More robust for high-dimensional sparse medical data
+
+**2. Why ensemble did not improve**
+- High correlation between three tree models (>0.85)
+- Lack of diversity, limited ensemble gain
+- Suggestion: add neural networks or linear models for diversity
+
+**3. Clinical reasonableness of AUC 0.69**
+- Predicting post-transplant complications using only pre-transplant variables is extremely difficult
+- Literature shows similar tasks typically achieve AUC 0.65-0.75
+- Post-transplant variables (early GVHD manifestations) would significantly improve AUC
+
+### 4.3 Feature Importance Analysis
+
+**Top 10 important features (CatBoost + LightGBM consensus)**:
+
+| Rank | Feature | Clinical Meaning | Importance |
+|------|---------|------------------|------------|
+| 1 | **grpcod** | Transplant center code | Highest |
+| 2 | **dnrage** | Donor age | High |
+| 3 | **pbcd34kg** | CD34+ cell dose/kg | High |
+| 4 | **ldhpr** | Pre-transplant LDH | Medium-High |
+| 5 | **bmi** | Body mass index | Medium-High |
+| 6 | **rawtpr** | Pre-transplant weight | Medium |
+| 7 | **ast_pr** | Pre-transplant AST | Medium |
+| 8 | **agedx** | Age at diagnosis | Medium |
+| 9 | **hb_pr** | Pre-transplant hemoglobin | Medium |
+| 10 | **plate_pr** | Pre-transplant platelets | Medium |
+
+**Clinical interpretation**:
+- **Center effect (grpcod)**: Different centers have different patient populations and protocols
+- **Donor age**: Younger donors have better stem cell quality
+- **Cell dose**: CD34+ dose affects immune reconstitution
+- **Liver function indicators**: LDH/AST reflect disease burden and organ status
+
+### 4.4 Conformal Prediction Results
+
+| Metric | Value |
+|--------|-------|
+| Confidence level α | 0.10 (90% coverage) |
+| High-confidence samples | 494 (31%) |
+| Low-confidence samples | 1,112 (69%) |
+| Actual coverage rate | 97.45% |
+| Target coverage rate | ≥90% |
+
+**Medical deployment strategy**:
+- High-confidence predictions → automatic classification, saving physician time
+- Low-confidence predictions → flagged for "manual review"
+- Effectively identifies "difficult cases", reducing misdiagnosis risk
+
+---
+
+## 5. Method Comparison and Discussion
+
+### 5.1 Method Selection Decision Tree
+
+```
+Data type: Medical tabular data
+├── High feature dimension (>100)?
+│   ├── Yes → Tree models (LGB/XGB/CAT) better than neural networks
+│   └── No → Can try TabNet/FT-Transformer
+├── Many categorical features?
+│   ├── Yes → CatBoost optimal
+│   └── No → LightGBM/XGBoost both fine
+├── Need interpretability?
+│   ├── Yes → TabNet (sacrifices performance) or SHAP (post-hoc)
+│   └── No → Any high-performance model
+└── Need uncertainty quantification?
+    ├── Yes → Conformal Prediction (model-agnostic)
+    └── No → Direct probability output
+```
+
+### 5.2 Pros and Cons of Each Method
+
+| Method | Pros | Cons | Suitable Scenarios |
+|--------|------|------|-------------------|
+| TabNet | Strong interpretability | Poor on high-dimensional data | Low-dimensional, clear feature relationships |
+| LightGBM | Fast, accurate | Requires tuning | General tabular data |
+| XGBoost | Strong regularization | Slower, prone to underfitting | Small samples, overfitting prevention |
+| CatBoost | Native categorical support | Slower training | Data with many categorical features |
+| AutoGluon | Fully automatic | Complex dependencies, resource-heavy | Quick baseline |
+| Conformal | Coverage guarantee | Prediction sets may be too large | High-risk medical scenarios |
+
+---
+
+## 6. Code Implementation
+
+### 6.1 Complete Data Preprocessing
+
+```python
+import pandas as pd
+import numpy as np
+
+# Load data
+df = pd.read_csv('Cleaned_Data_Final.csv')
+
+# Handle missing value codes
+for col in ['agvhd24', 'agvhd34']:
+    df[col] = df[col].replace(905, np.nan)
+
+# Construct binary label: Severe vs Non-severe
+df['target'] = (df['agvhd34'] == 1).astype(int)
+
+# Remove post-transplant variables
+post_vars = ['agvhd24', 'agvhd34', 'ahisgut', 'ahisliv', ...]
+feature_cols = [c for c in df.columns if c not in post_vars]
+
+# Data cleaning
+model_df = df[feature_cols + ['target']].copy()
+for col in feature_cols:
+    if model_df[col].dtype in ['float64', 'int64']:
+        model_df[col] = model_df[col].replace(905, np.nan)
+        model_df[col] = model_df[col].fillna(model_df[col].median())
+
+# Split
+from sklearn.model_selection import train_test_split
+train_df, test_df = train_test_split(
+    model_df, test_size=0.2, stratify=model_df['target'], random_state=42
+)
+```
+
+### 6.2 CatBoost Training
+
+```python
+from catboost import CatBoostClassifier
+from sklearn.metrics import roc_auc_score
+
+# Calculate class weights
+scale_pos_weight = len(y_train[y_train==0]) / len(y_train[y_train==1])
+
+# Train
+model = CatBoostClassifier(
+    iterations=1000,
+    learning_rate=0.05,
+    depth=6,
+    l2_leaf_reg=3,
+    scale_pos_weight=scale_pos_weight,
+    random_seed=42,
+    verbose=False
+)
+model.fit(X_train, y_train, eval_set=(X_test, y_test))
+
+# Evaluate
+proba = model.predict_proba(X_test)[:, 1]
+auc = roc_auc_score(y_test, proba)
+print(f"AUC: {auc:.4f}")
+```
+
+### 6.3 Conformal Prediction
+
+```python
+# Split calibration set
+train_idx, cal_idx = train_test_split(
+    np.arange(len(train_df)), test_size=0.25, 
+    stratify=y_train, random_state=42
+)
+
+# Calibration probabilities
+cal_proba = model.predict_proba(X_train[cal_idx])[:, 1]
+cal_proba_2class = np.stack([1-cal_proba, cal_proba], axis=1)
+
+# Compute non-conformity scores
+scores = 1 - cal_proba_2class[np.arange(len(cal_idx)), y_cal.astype(int)]
+
+# Quantile threshold
+alpha = 0.1
+q_level = np.ceil((len(cal_idx) + 1) * (1 - alpha)) / len(cal_idx)
+threshold = np.quantile(scores, q_level, method='higher')
+
+# Test prediction sets
+test_scores = 1 - np.stack([1-test_proba, test_proba], axis=1)
+prediction_sets = test_scores <= threshold
+```
+
+---
+
+## 7. Limitations and Future Work
+
+### 7.1 Current Limitations
+1. **Insufficient feature engineering**: No interaction features created (age difference, cell dose/weight ratio)
+2. **FT-Transformer not attempted**: Tabular Transformer may be more suitable for high-dimensional data
+3. **Limited hyperparameter search**: Used default parameters, no systematic tuning
+4. **Single-fold split**: No K-fold cross-validation, results may have variance
+
+### 7.2 Improvement Directions
+1. **Feature engineering**:
+   - Donor-recipient age difference
+   - CD34+ dose/weight ratio
+   - HLA matching composite score
+   - Disease risk stratification combinations
+
+2. **Model improvements**:
+   - FT-Transformer (tabular-specific Transformer)
+   - Deep ensemble (add neural networks)
+   - Bayesian optimization for hyperparameters
+
+3. **Evaluation improvements**:
+   - 5-fold cross-validation
+   - Temporal split (by transplant year)
+   - External validation (different centers)
+
+4. **Clinical deployment**:
+   - Build web interface for pre-transplant variable input
+   - Output prediction probability + confidence flag
+   - Automatic expert referral for low-confidence cases
+
+---
+
+## 8. Conclusion
+
+This project attempted multiple methods from traditional machine learning to cutting-edge deep learning for pre-transplant prediction of acute GVHD severity. Main conclusions:
+
+1. **CatBoost is the best choice** (AUC=0.6885), benefiting from native categorical feature support
+2. **Tree models outperform neural networks** on this dataset because high-dimensional sparse features are not suitable for TabNet
+3. **Conformal Prediction provides valuable uncertainty quantification**, identifying cases requiring manual review
+4. **Predicting severe GVHD using only pre-transplant variables is a clinical challenge**, AUC 0.69 is within reasonable range
+5. **Feature importance reveals center effect, donor age, and cell dose as key factors**
+
+---
+
+## Appendix: Runtime Environment
+
+- Python 3.12
+- pandas 2.x
+- numpy 1.26
+- scikit-learn 1.4
+- lightgbm 4.x
+- xgboost 2.x
+- catboost 1.2
+
+---
+
+*Report generated: May 2025*
+*Course: STAT 5430 - Statistical Learning*