process.md

NBA ML Prediction System - Process Guide

Prerequisites

Before starting, ensure you have:

• Python 3.10+ installed
• Virtual environment activated: .\venv\Scripts\activate
• All dependencies installed: pip install -r requirements.txt

---

Step 1: Collect Training Data (COMPREHENSIVE)

Purpose: Fetch 10 seasons of ALL NBA stats from the API including:

• Games, Team Stats, Player Stats (basic)
• Advanced Metrics (NET_RTG, PACE, PIE, TS%, eFG%)
• Clutch Stats (performance in close games)
• Hustle Stats (deflections, charges, loose balls)
• Defense Stats

File: src/data_collector.py

Command:

python -m src.data_collector

Duration: ~2-4 hours (has resume capability if interrupted)

Output Files (in data/raw/):

• all_games.parquet - Game results
• all_team_stats.parquet - Basic team stats
• all_team_advanced.parquet - NET_RTG, PACE, PIE, TS%
• all_team_clutch.parquet - Close game performance
• all_team_hustle.parquet - Deflections, charges
• all_team_defense.parquet - Defensive metrics
• all_player_stats.parquet - Player averages
• all_player_advanced.parquet - PER, USG%, TS%
• all_player_clutch.parquet - Player clutch stats
• all_player_hustle.parquet - Player hustle metrics

---

Step 2: Generate Features

Purpose: Create ~50+ features including ELO, rolling stats, momentum, rest/fatigue

File: src/feature_engineering.py

Command:

python -m src.feature_engineering --process

Duration: ~30-60 minutes

Output Files:

• data/processed/game_features.parquet

Features Generated:

• ELO ratings (team_elo, opponent_elo, elo_diff, elo_win_prob)
• Rolling stats (PTS/AST/REB/FG_PCT last 5/10/20 games)
• Defensive stats (STL, BLK, DREB rolling)
• Momentum (wins_last5, hot_streak, cold_streak, plus_minus)
• Rest/fatigue (days_rest, back_to_back, games_last_week)
• Season averages (all stats)
• Team advanced metrics (NET_RTG, PACE, clutch, hustle)
• Player aggregations (top players avg, star concentration)

---

Step 3: Build Dataset

Purpose: Split data into train/val/test and prepare for training

File: src/preprocessing.py

Command:

python -m src.preprocessing --build

Output Files:

• data/processed/game_dataset.joblib

What It Does:

• Automatically detects ALL numeric features
• Splits by season (no data leakage)
• Scales and imputes missing values

---

Step 4: Train Model

Purpose: Train XGBoost + LightGBM ensemble on ALL features

File: src/models/game_predictor.py

Command:

python -m src.models.game_predictor --train

Expected Output:

Loading dataset...
Training XGBoost model...
Training LightGBM model...
Training complete!

=== Test Metrics ===
Test Accuracy: 0.67XX
Test Brier Score: 0.21XX
✓ Target accuracy (>65%) achieved!

=== Top Features ===
                feature  xgb_importance  lgb_importance  avg_importance
0              elo_diff          0.XXX           0.XXX            0.XXX
1          elo_win_prob          0.XXX           0.XXX            0.XXX
...

Saved model to models/game_predictor.joblib

Output Files:

• models/game_predictor.joblib

---

Step 5: Generate Visualizations

Purpose: Create analysis charts saved to graphs/

File: src/visualization.py

Command:

python -m src.visualization

Output Files (in graphs/):

• mvp_race.png
• mvp_stat_comparison.png
• championship_odds_pie.png
• strength_vs_experience.png

---

Step 6: Run the Dashboard

Purpose: Launch Streamlit web interface

File: app/app.py

Command:

streamlit run app/app.py

Opens: http://localhost:8501

Pages:

• 🔴 Live Games - Real-time scores with predictions
• 🎮 Game Predictions - Predict any matchup
• 📈 Model Accuracy - Track prediction accuracy
• 🏆 MVP Race - Top candidates
• 👑 Championship Odds - Team probabilities
• 📊 Team Explorer - Stats & injuries

---

Quick Reference

Step	Command	Duration
1	python -m src.data_collector	2-4 hours
2	python -m src.feature_engineering --process	30-60 min
3	python -m src.preprocessing --build	1-2 min
4	python -m src.models.game_predictor --train	2-5 min
5	python -m src.visualization	10 sec
6	streamlit run app/app.py	Immediate

---

Live Data Features (NEW)

View Live Scoreboard

python -m src.live_data_collector

Shows today's NBA games with live scores.

Continuous Learning

# Ingest completed games
python -m src.continuous_learner --ingest

# Full update cycle (ingest + features + retrain)
python -m src.continuous_learner --update

# Update without retraining
python -m src.continuous_learner --update --no-retrain

Check Prediction Accuracy

python -m src.prediction_tracker

Shows accuracy stats from ChromaDB.

---

Data Flow

NBA API
   ↓
[Step 1: data_collector.py]
   ↓
data/raw/*.parquet (10+ files)
   ↓
[Step 2: feature_engineering.py]
   ↓
data/processed/game_features.parquet (~50+ features)
   ↓
[Step 3: preprocessing.py]
   ↓
data/processed/game_dataset.joblib (train/val/test splits)
   ↓
[Step 4: game_predictor.py]
   ↓
models/game_predictor.joblib (trained ensemble)
   ↓
[Step 6: app.py] → Web Dashboard
   ↓
ChromaDB (prediction tracking)

---

Troubleshooting

ModuleNotFoundError: No module named 'src'

Ensure you're in the project root directory.

API Rate Limit Errors

The data collector handles this with exponential backoff. Just let it retry.

Resume Interrupted Collection

Just run the command again - it has checkpoint capability and will skip completed data.

ChromaDB Connection Issues

Check your API key in src/config.py under ChromaDBConfig.