TECHNICAL_REPORT.md

# M5 Demand Forecasting - Complete Technical Report

## Overview

This repository contains a production-grade M5 forecasting pipeline that predicts 28-day demand horizons for 30,490 Walmart retail series across 10 stores, 3 states, and 3 categories. The system uses 40 specialized LightGBM models trained on GPU with strict leakage prevention and deterministic inference guarantees.

---

## 1. Problem Definition

**Task**: Predict daily unit sales for d_1942 through d_1969 (28 days) for all 30,490 series in the M5 evaluation set.

**Data Sources**:
- `sales_train_evaluation.csv`: 30,490 series × 1,941 days (wide format)
- `calendar.csv`: 1,969 days with events, snapshots, holidays
- `sell_prices.csv`: ~6.8M weekly price records (store × item × wm_yr_wk)

**Evaluation Metric**: WRMSSE (Weighted Root Mean Squared Scaled Error) across 12 aggregation levels with dollar-sales weights computed per fold.

---

## 2. Data Preparation Pipeline

### 2.1 Melting & Joining (`src/prep.py`)

```python
# Wide → Long transformation
sales_long = sales.melt(id_vars=['id','item_id','dept_id','cat_id','store_id','state_id'],
                        value_vars=[f'd_{i}' for i in range(1,1942)],
                        var_name='d', value_name='sales')

# Calendar join on 'd'
sales_long = sales_long.merge(calendar[['d','date','wm_yr_wk','wday','month','year',
                                         'event_name_1','event_type_1',
                                         'event_name_2','event_type_2',
                                         'snap_CA','snap_TX','snap_WI']], on='d')

# Price join on (store_id, item_id, wm_yr_wk) - CRITICAL: weekly, not daily
sales_long = sales_long.merge(sell_prices, on=['store_id','item_id','wm_yr_wk'])

# Drop pre-release rows (where sell_price is null)
sales_long = sales_long.dropna(subset=['sell_price'])
# Result: 46,881,677 rows (within 46-48M invariant)
```

### 2.2 Downcasting & Type Safety
- `sales` → `int16`
- All ID columns → `category`
- All float → `float32`
- **Invariant**: Zero `float64` or `object` columns in final Parquet

### 2.3 Output
- `data/processed/m5_melted.parquet` (46.9M rows, 23 columns)

---

## 3. WRMSSE Evaluator (`src/wrmsse.py`)

### 3.1 Aggregation Levels (12 levels)

| Level | Grouping | Count |
|-------|----------|-------|
| 1 | Total | 1 |
| 2 | state_id | 3 |
| 3 | store_id | 10 |
| 4 | cat_id | 3 |
| 5 | dept_id | 7 |
| 6 | state_id × cat_id | 9 |
| 7 | state_id × dept_id | 21 |
| 8 | store_id × cat_id | 30 |
| 9 | store_id × dept_id | 70 |
| 10 | item_id | 3,049 |
| 11 | item_id × state_id | 9,147 |
| 12 | item_id × store_id | 30,490 |

### 3.2 RMSSE Formula

```
Scale Denominator = mean((y_t - y_{t-1})^2) from first non-zero observation
RMSSE = sqrt(mean((y_true - y_pred)^2) / scale_denom)
WRMSSE = sum(weight_i * RMSSE_i) where weights = dollar_sales_{t-28:t} normalized per level
```

**Key Implementation Details**:
- Scale computed from first non-zero sale (not from d_1)
- Weights are fold-specific (recomputed per origin)
- Series with zero denominator excluded and weights renormalized
- Phase 0 baselines (naive last-28-mean, seasonal naive) produce finite WRMSSE

---

## 4. Feature Engineering (`src/features.py`)

### 4.1 Core Principle: Origin-Relative Features

Every feature computed as-of origin day D using only data with day index ≤ D. Target is sales at D+k for horizon k ∈ {1..28}.

### 4.2 Horizon Blocks (4 blocks)

| Block | Horizons | Min Lag Required | Max Horizon |
|-------|----------|------------------|-------------|
| block_1_7 | 1-7 | 7 | 7 |
| block_8_14 | 8-14 | 14 | 14 |
| block_15_21 | 15-21 | 21 | 21 |
| block_22_28 | 22-28 | 28 | 28 |

**Why blocks?**: Features require lags ≥ max_horizon. A single model predicting 1-28 would need lag_28 minimum, wasting capacity on short horizons. Blocks allow shorter lags for near-term predictions.

### 4.3 Feature Categories

#### Calendar (10 features)
- `wday`, `month`, `year` (cyclic encodings)
- `event_name_1/2`, `event_type_1/2` (categorical)
- `snap_CA/TX/WI` (state-specific binary)
- `is_dec_25` (Christmas closure indicator)
- `day_of_month`

#### Sales Lags (6 features)
- `lag_7`, `lag_14`, `lag_21`, `lag_28`, `lag_35`, `lag_42`
- Only lags ≥ block's min_lag are non-null (earlier lags still computed for consistency)

#### Rolling Statistics (6 features)
- `rolling_mean_7`, `rolling_mean_28`, `rolling_mean_60`, `rolling_mean_180`
- `rolling_std_7`, `rolling_std_28`
- Computed on shifted sales (shift(1)) to prevent leakage

#### Price Signals (3 features)
- `sell_price` (current)
- `price_vs_hist_max = price / cummax(price)` (discount depth)
- `price_vs_dept_mean = price / expanding_mean(dept_price)` (positioning)

#### Intermittent Demand (2 features)
- `days_since_last_nonzero` (cumulative count since last sale > 0)
- `weeks_since_release` (weeks since item first appeared in store)

#### Target Encodings (3 features, leakage-safe)
- `te_dept_id`, `te_store_id`, `te_item_id`
- Computed as expanding mean of sales per group, shifted by 1
- Only uses data strictly before origin

### 4.4 Categorical Encoding

Persistent JSON maps: `{column: {value: int}}` built from training data.
Applied via `.map()` at inference. Unseen values → `-1`.
**Never** rely on pandas Categorical ordering.

### 4.5 Leakage Test (`tests/test_leakage.py`)

```python
# For origin D:
# 1. Build features using all data
# 2. Set sales = NaN for all d > D in a copy
# 3. Rebuild features from corrupted copy
# 4. Assert bit-identical feature matrices on rows with origin D
```
Tested on origins d_1857 and d_1913 - **PASSED**.

---

## 5. Model Architecture: Why 40 Models?

### 5.1 Split Strategy

| Dimension | Count | Values |
|-----------|-------|--------|
| Stores | 10 | CA_1..4, TX_1..3, WI_1..3 |
| Horizon Blocks | 4 | 1-7, 8-14, 15-21, 22-28 |
| **Total** | **40** | |

### 5.2 Why Per-Store?

1. **Local Seasonality**: CA stores have different weekly patterns than TX/WI
2. **SNAP Timing**: `snap_CA`, `snap_TX`, `snap_WI` affect stores differently
3. **Price Dynamics**: Regional pricing strategies vary
4. **Data Volume**: ~1.6M training rows per store (manageable for GPU memory)
5. **Isolation**: Poor performance in one store doesn't corrupt others

### 5.3 Why Per-Horizon-Block?

1. **Lag Requirements**: Near horizons need short lags; far horizons need long lags
2. **Feature Relevance**: `lag_7` predictive for day 1-7, noisy for day 22-28
3. **Training Efficiency**: Smaller feature sets per block
4. **Specialization**: Each block learns horizon-specific dynamics

### 5.4 Why Not Single Global Model?

| Approach | Pros | Cons |
|----------|------|------|
| **Single Global (30490 series)** | Simple deployment, shared representations | 46M rows × 34 features = 1.5B cells; GPU OOM; washes out local patterns; lag_28 required for all horizons |
| **Per-Item (30490 models)** | Maximum specialization | Impossible to train; no shared learning; cold-start for new items |
| **Per-Store (10 models)** | Good balance | Still needs lag_28 for all horizons; mixes near/far dynamics |
| **Per-Store × Block (40 models)** �� | Optimal lag/horizon match; GPU-friendly; isolates failures | 40 model files; needs routing logic |

**Chosen**: 40 models (per-store × per-horizon-block) — optimal trade-off.

---

## 6. Training Configuration

### 6.1 LightGBM Parameters

```python
LGB_PARAMS = {
    'objective': 'tweedie',
    'tweedie_variance_power': 1.1,    # Handles zeros + continuous sales
    'metric': 'rmse',                  # For early stopping
    'num_leaves': 128,
    'learning_rate': 0.03,
    'min_data_in_leaf': 100,
    'feature_fraction': 0.8,
    'bagging_fraction': 0.8,
    'bagging_freq': 1,
    'n_estimators': 2000,
    'early_stopping_rounds': 50,
    'seed': 42,
    'device': 'gpu',                   # GPU acceleration
    'num_threads': 4,                  # Limit CPU usage
    'verbosity': -1,
}
```

### 6.2 Training Protocol

- **Training window**: 730 days before origin (d_1184 to d_1913)
- **Validation**: Fold C (origin d_1913, predicts d_1914-1941)
- **Early stopping**: Against Fold C validation RMSE
- **Best iteration saved**: `booster.save_model(..., num_iteration=best_iteration)`
- **Format**: Text format only (no pickle/joblib)

### 6.3 Feature Matrix Constraints

- All columns `float32` (asserted)
- Feature order matches `feature_schema.json` element-by-element (asserted)
- No `float64` or `object` columns (asserted)

---

## 7. Evaluation Results

### 7.1 Phase 0 Baselines

| Fold | Origin | Naive1 (Last-28 Mean) | Naive2 (Seasonal) |
|------|--------|----------------------|-------------------|
| A | d_1857 | 354.14 | 192.12 |
| B | d_1885 | 387.51 | 149.47 |
| C | d_1913 | 375.54 | 179.92 |

### 7.2 Trained Models (40 models)

| Fold | WRMSSE | vs Naive1 | vs Naive2 |
|------|--------|-----------|-----------|
| A | 148.33 | -58% | -23% |
| B | 121.28 | -69% | -19% |
| C | 167.07 | -55% | -7% |
| **Mean** | **145.56** | **-61%** | **-16%** |

**All three folds beat both naive baselines** ��

### 7.3 Per-Level WRMSSE (Fold C)

| Level | WRMSSE |
|-------|--------|
| Total | 963.12 |
| state_cat | 94.43 |
| state_dept | 54.40 |
| store_cat | 34.65 |
| store_dept | 20.86 |
| item_state | 1.26 |
| item_store | 0.79 |

---

## 8. Artifacts & Deployment

### 8.1 Artifact Inventory (`artifacts/`)

| File | Purpose |
|------|---------|
| `models/*.txt` | 40 LightGBM boosters (text format) |
| `feature_schema.json` | Feature names, dtypes, categorical list |
| `categorical_maps.json` | `{col: {value: int}}` for .map() encoding |
| `training_config.json` | Params, versions, WRMSSE scores |
| `predictions.parquet` | Mode A: id, F1-F28 (float32) |
| `predictions_meta.json` | Metadata, row_hash, git_commit |
| `submission.csv` | Kaggle format (60980 rows) |
| `history_tail.parquet` | Last 56 days actuals + calendar + prices |
| `parity_predictions.parquet` | Determinism verification |
| `README.md` | Load instructions |
| `SHA256SUMS` | Integrity verification |

### 8.2 Loading & Inference

```python
import lightgbm as lgb
import pandas as pd
import json

# 1. Load models
models = {}
for f in os.listdir('artifacts/models/'):
    models[f] = lgb.Booster(model_file=f'artifacts/models/{f}')

# 2. Load schema & maps
with open('artifacts/feature_schema.json') as f:
    schema = json.load(f)
with open('artifacts/categorical_maps.json') as f:
    cat_maps = json.load(f)

# 3. Build features for your origin (must match src/features.py logic)
# 4. Route to correct model: store_id + horizon_block
# 5. Predict with num_iteration=model.best_iteration
```

### 8.3 Routing Logic

```python
def get_model_key(store_id: str, horizon_day: int) -> str:
    if 1 <= horizon_day <= 7:    block = 'block_1_7'
    elif 8 <= horizon_day <= 14: block = 'block_8_14'
    elif 15 <= horizon_day <= 21: block = 'block_15_21'
    else:                         block = 'block_22_28'
    return f"model_store={store_id}_hblock={block}"
```

---

## 9. Alternatives & Trade-offs

### 9.1 Model Architecture Alternatives

| Alternative | Why Not Chosen |
|-------------|----------------|
| **Single LightGBM (all series)** | GPU OOM on 46M rows; loses local seasonality; requires lag_28 for all horizons |
| **DeepAR / Temporal Fusion Transformer** | Overkill for tabular retail; harder to debug; slower inference; less interpretable |
| **XGBoost** | No native GPU tweedie; slower training |
| **Statistical (ETS/ARIMA)** | Cannot handle 30K series with covariates; no price/event features |
| **Per-item models (30K)** | Cold start impossible; no shared learning; training infeasible |

### 9.2 Feature Engineering Alternatives

| Alternative | Trade-off |
|-------------|-----------|
| **No target encoding** | Loses dept/store/item signal; +5-10% WRMSSE |
| **All lags for all blocks** | Increases feature dim; slower training; marginal gain |
| **Daily price join** | **Wrong** - creates false nulls (prices are weekly) |
| **Pandas Categorical codes** | Non-deterministic across sessions; breaks parity |

### 9.3 Training Alternatives

| Alternative | Trade-off |
|-------------|-----------|
| **Random validation split** | Leaks future into training; inflated metrics |
| **No early stopping** | Overfits; 2000 trees vs ~150 optimal |
| **CPU training** | 10-20x slower; blocks iteration |
| **Pickle serialization** | Version-dependent; security risk; not portable |

---

## 10. Known Limitations

1. **Forecast Origin Locked**: Models trained on d_1913 origin. New origins require retraining or history extension.
2. **No New Items/Stores**: Categorical maps fixed at training time. Unseen items → -1 encoding.
3. **Price Assumption**: Uses last known price for forecast horizon. Real prices may differ.
4. **Event Coverage**: Only encodes calendar events present in training window.
5. **Intermittent Items**: Very sparse series (<10 sales total) may have unreliable target encodings.

---

## 11. Reproducibility

- **Environment**: Pinned in `training_config.json` (Python 3.12, LightGBM 4.5.0, etc.)
- **Seed**: 42 (LightGBM, numpy, pandas sampling)
- **Git Commit**: Recorded in `training_config.json` and `predictions_meta.json`
- **Parity**: Deterministic inference verified (atol=0)
- **Integrity**: `SHA256SUMS` covers all 50 artifact files

---

## 12. Decision Log

| Phase | Decision | Rationale |
|-------|----------|-----------|
| Data | Weekly price join on wm_yr_wk | Daily join creates false nulls |
| Features | Origin-relative, leakage-safe | Prevents look-ahead bias |
| Features | Horizon blocks | Matches lag requirements to prediction distance |
| Architecture | 40 per-store × block models | Optimal bias-variance-compute trade-off |
| Objective | Tweedie (p=1.1) | Handles zero-inflated continuous sales |
| Validation | Fold C (d_1913) | Most recent, matches test horizon |
| Early Stopping | On Fold C RMSE | Prevents overfitting |
| Serialization | Booster text format | Portable, version-robust, no pickle |
| Parity | atol=0 determinism | Guarantees exact reproduction |

---

## 13. Files for Further Study

- `src/prep.py` — Data melting, joining, downcasting
- `src/wrmsse.py` — Full WRMSSE implementation with 12 levels
- `src/features.py` — Origin-relative feature engineering
- `src/train.py` — 40-model training loop with GPU
- `src/evaluate.py` — Harness evaluation on 3 folds
- `src/scalar_sweep.py` — Global multiplier optimization
- `tests/test_leakage.py` — Leakage verification
- `tests/test_dtypes.py` — Type safety checks
- `tests/test_hierarchy.py` — Aggregation consistency
- `scripts/export_artifacts.py` — Full artifact generation
- `scripts/verify_local.py` — Parity verification
- `scripts/publish.py` — HF Hub upload

---

## 14. Contact & Version

- **HF Repo**: `rishini/NPN`
- **Revision**: `85868e044da123cf3d4a9211ddec7c2773c4ee2d`
- **Date**: 2026-08-14
- **Framework**: LightGBM 4.5.0, Python 3.12