File size: 3,375 Bytes
00ae263 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | # M5 Forecasting: Statistical Models Comparison
## Performance Summary
| Model | RMSE | MAE | MAPE | Model Size | Repo |
|----------------|-----------|-----------|---------|------------|------|
| **SARIMAX** | 2759.70 | 2260.25 | 4.98% | 85.30 MB | [rishini/NPN-sarimax](https://huggingface.co/rishini/NPN-sarimax) |
| **Prophet** | 4860.67 | 4038.73 | 8.73% | 0.18 MB | [rishini/NPN-prophet](https://huggingface.co/rishini/NPN-prophet) |
| **ARIMA** | 6459.70 | 4852.62 | 10.48% | 6.56 MB | [rishini/NPN-arima](https://huggingface.co/rishini/NPN-arima) |
| **LightGBM** (per-series) | N/A (WRMSSE=145.56) | | | 106.1 MB | [rishini/NPN](https://huggingface.co/rishini/NPN) |
## Key Findings
### 1. SARIMAX Wins (Best Accuracy)
- **Lowest RMSE**: 2,759.70 (4.98% MAPE)
- **Best fit**: SARIMAX(2,1,1)(1,1,1,7) captures weekly seasonality
- **Exogenous boost**: SNAP indicators and event dummies improve predictions
- **Trade-off**: Largest model file (85MB) due to complex state space representation
### 2. Prophet (Best Interpretability)
- **RMSE**: 4,860.67 (8.73% MAPE)
- **Strengths**: Fast training, automatic seasonality detection, built-in uncertainty intervals
- **Weaknesses**: Underperforms on aggregate-level predictions
- **Trade-off**: Smallest model (180KB), fastest to deploy
### 3. ARIMA (Baseline Simplicity)
- **RMSE**: 6,459.70 (10.48% MAPE)
- **Best config**: ARIMA(3,1,1) with deterministic trend
- **Strengths**: Simple, interpretable, smallest non-Prophet model
- **Weaknesses**: No seasonality, no exogenous variables, poorest fit
### 4. LightGBM (Per-Series Champion)
- **WRMSSE**: 145.56 (beats naive baselines by 55-69%)
- **Advantage**: Predicts all 30,490 series individually
- **Trade-off**: Not directly comparable (different granularity)
## Why SARIMAX Outperforms?
1. **Weekly Seasonality**: Retail demand has strong 7-day cycles (weekends higher)
2. **Exogenous Signals**: SNAP eligibility and events directly impact demand
3. **Autocorrelation**: Captures persistence in sales patterns
4. **Differencing**: (d=1) removes trend, focusing on changes
## Why These Models Don't Beat LightGBM?
| Aspect | LightGBM | Statistical Models |
|--------|----------|-------------------|
| Granularity | 30,490 individual series | 1 aggregate series |
| Features | 34 engineered features | 6-12 basic features |
| Flexibility | Non-linear relationships | Linear/AR/MA assumptions |
| Cross-series learning | Store/dept/item interactions | No sharing across series |
| Deployment | 40 per-store models | 3 aggregate models |
The statistical models operate on **aggregate** sales (total ~34,000/day), while LightGBM models each of the **30,490 individual series** with specialized features. The statistical approach serves as a solid baseline but cannot match the per-series precision of gradient boosting.
## Model Selection Guide
Use **SARIMAX** when:
- You need the best statistical baseline
- Exogenous variables (events, promotions) are important
- Weekly seasonality dominates
Use **Prophet** when:
- Fast experimentation is needed
- Interpretability is key
- Multiple seasonalities exist
Use **ARIMA** when:
- Simple baseline is sufficient
- No strong seasonality
- Minimal computational budget
Use **LightGBM** when:
- Maximum accuracy is required
- Per-series predictions needed
- GPU is available
|