| # M5 Forecasting: Statistical Models Comparison |
|
|
| ## Performance Summary |
|
|
| | Model | RMSE | MAE | MAPE | Model Size | Repo | |
| |----------------|-----------|-----------|---------|------------|------| |
| | **SARIMAX** | 2759.70 | 2260.25 | 4.98% | 85.30 MB | [rishini/NPN-sarimax](https://huggingface.co/rishini/NPN-sarimax) | |
| | **Prophet** | 4860.67 | 4038.73 | 8.73% | 0.18 MB | [rishini/NPN-prophet](https://huggingface.co/rishini/NPN-prophet) | |
| | **ARIMA** | 6459.70 | 4852.62 | 10.48% | 6.56 MB | [rishini/NPN-arima](https://huggingface.co/rishini/NPN-arima) | |
| | **LightGBM** (per-series) | N/A (WRMSSE=145.56) | | | 106.1 MB | [rishini/NPN](https://huggingface.co/rishini/NPN) | |
|
|
| ## Key Findings |
|
|
| ### 1. SARIMAX Wins (Best Accuracy) |
| - **Lowest RMSE**: 2,759.70 (4.98% MAPE) |
| - **Best fit**: SARIMAX(2,1,1)(1,1,1,7) captures weekly seasonality |
| - **Exogenous boost**: SNAP indicators and event dummies improve predictions |
| - **Trade-off**: Largest model file (85MB) due to complex state space representation |
|
|
| ### 2. Prophet (Best Interpretability) |
| - **RMSE**: 4,860.67 (8.73% MAPE) |
| - **Strengths**: Fast training, automatic seasonality detection, built-in uncertainty intervals |
| - **Weaknesses**: Underperforms on aggregate-level predictions |
| - **Trade-off**: Smallest model (180KB), fastest to deploy |
|
|
| ### 3. ARIMA (Baseline Simplicity) |
| - **RMSE**: 6,459.70 (10.48% MAPE) |
| - **Best config**: ARIMA(3,1,1) with deterministic trend |
| - **Strengths**: Simple, interpretable, smallest non-Prophet model |
| - **Weaknesses**: No seasonality, no exogenous variables, poorest fit |
|
|
| ### 4. LightGBM (Per-Series Champion) |
| - **WRMSSE**: 145.56 (beats naive baselines by 55-69%) |
| - **Advantage**: Predicts all 30,490 series individually |
| - **Trade-off**: Not directly comparable (different granularity) |
|
|
| ## Why SARIMAX Outperforms? |
|
|
| 1. **Weekly Seasonality**: Retail demand has strong 7-day cycles (weekends higher) |
| 2. **Exogenous Signals**: SNAP eligibility and events directly impact demand |
| 3. **Autocorrelation**: Captures persistence in sales patterns |
| 4. **Differencing**: (d=1) removes trend, focusing on changes |
|
|
| ## Why These Models Don't Beat LightGBM? |
|
|
| | Aspect | LightGBM | Statistical Models | |
| |--------|----------|-------------------| |
| | Granularity | 30,490 individual series | 1 aggregate series | |
| | Features | 34 engineered features | 6-12 basic features | |
| | Flexibility | Non-linear relationships | Linear/AR/MA assumptions | |
| | Cross-series learning | Store/dept/item interactions | No sharing across series | |
| | Deployment | 40 per-store models | 3 aggregate models | |
|
|
| The statistical models operate on **aggregate** sales (total ~34,000/day), while LightGBM models each of the **30,490 individual series** with specialized features. The statistical approach serves as a solid baseline but cannot match the per-series precision of gradient boosting. |
|
|
| ## Model Selection Guide |
|
|
| Use **SARIMAX** when: |
| - You need the best statistical baseline |
| - Exogenous variables (events, promotions) are important |
| - Weekly seasonality dominates |
|
|
| Use **Prophet** when: |
| - Fast experimentation is needed |
| - Interpretability is key |
| - Multiple seasonalities exist |
|
|
| Use **ARIMA** when: |
| - Simple baseline is sufficient |
| - No strong seasonality |
| - Minimal computational budget |
|
|
| Use **LightGBM** when: |
| - Maximum accuracy is required |
| - Per-series predictions needed |
| - GPU is available |
|
|
