M5 Forecasting: Statistical Models Comparison
Performance Summary
| Model | RMSE | MAE | MAPE | Model Size | Repo |
|---|---|---|---|---|---|
| SARIMAX | 2759.70 | 2260.25 | 4.98% | 85.30 MB | rishini/NPN-sarimax |
| Prophet | 4860.67 | 4038.73 | 8.73% | 0.18 MB | rishini/NPN-prophet |
| ARIMA | 6459.70 | 4852.62 | 10.48% | 6.56 MB | rishini/NPN-arima |
| LightGBM (per-series) | N/A (WRMSSE=145.56) | 106.1 MB | rishini/NPN |
Key Findings
1. SARIMAX Wins (Best Accuracy)
- Lowest RMSE: 2,759.70 (4.98% MAPE)
- Best fit: SARIMAX(2,1,1)(1,1,1,7) captures weekly seasonality
- Exogenous boost: SNAP indicators and event dummies improve predictions
- Trade-off: Largest model file (85MB) due to complex state space representation
2. Prophet (Best Interpretability)
- RMSE: 4,860.67 (8.73% MAPE)
- Strengths: Fast training, automatic seasonality detection, built-in uncertainty intervals
- Weaknesses: Underperforms on aggregate-level predictions
- Trade-off: Smallest model (180KB), fastest to deploy
3. ARIMA (Baseline Simplicity)
- RMSE: 6,459.70 (10.48% MAPE)
- Best config: ARIMA(3,1,1) with deterministic trend
- Strengths: Simple, interpretable, smallest non-Prophet model
- Weaknesses: No seasonality, no exogenous variables, poorest fit
4. LightGBM (Per-Series Champion)
- WRMSSE: 145.56 (beats naive baselines by 55-69%)
- Advantage: Predicts all 30,490 series individually
- Trade-off: Not directly comparable (different granularity)
Why SARIMAX Outperforms?
- Weekly Seasonality: Retail demand has strong 7-day cycles (weekends higher)
- Exogenous Signals: SNAP eligibility and events directly impact demand
- Autocorrelation: Captures persistence in sales patterns
- Differencing: (d=1) removes trend, focusing on changes
Why These Models Don't Beat LightGBM?
| Aspect | LightGBM | Statistical Models |
|---|---|---|
| Granularity | 30,490 individual series | 1 aggregate series |
| Features | 34 engineered features | 6-12 basic features |
| Flexibility | Non-linear relationships | Linear/AR/MA assumptions |
| Cross-series learning | Store/dept/item interactions | No sharing across series |
| Deployment | 40 per-store models | 3 aggregate models |
The statistical models operate on aggregate sales (total ~34,000/day), while LightGBM models each of the 30,490 individual series with specialized features. The statistical approach serves as a solid baseline but cannot match the per-series precision of gradient boosting.
Model Selection Guide
Use SARIMAX when:
- You need the best statistical baseline
- Exogenous variables (events, promotions) are important
- Weekly seasonality dominates
Use Prophet when:
- Fast experimentation is needed
- Interpretability is key
- Multiple seasonalities exist
Use ARIMA when:
- Simple baseline is sufficient
- No strong seasonality
- Minimal computational budget
Use LightGBM when:
- Maximum accuracy is required
- Per-series predictions needed
- GPU is available