Spaces:
Running
Running
Machine Learning Best Practices
Core engineering principles for reliable, production-ready ML systems drawn from Google's Rules of Machine Learning and Google ML Crash Course.
Data Splitting Strategy
Three-Part Split (Standard)
Always divide data into three disjoint subsets:
| Split | Typical Size | Purpose |
|---|---|---|
| Training | ~70% | Model learns parameters |
| Validation | ~15% | Hyperparameter tuning and model selection during development |
| Test | ~15% | Single final evaluation after all decisions are made |
- Shuffle data before splitting to remove ordering bias.
- Never use test-set results to guide model changes — each time validation or test results inform a model tweak, those sets become less reliable as unbiased estimators.
- Refresh validation/test sets periodically when the dataset evolves.
Requirements for Valid Test and Validation Sets
- Large enough for statistically significant conclusions.
- Representative of both the training distribution and real-world deployment conditions.
- Zero examples duplicated in the training set.
- Free from any temporal or structural leakage between splits.
Cross-Validation
Use k-fold stratified cross-validation when data is scarce. It is more rigorous than a single holdout split and produces more stable performance estimates. Stratify by class label to preserve class proportions in each fold.
Avoiding Data Leakage
Data leakage causes models to appear much stronger in evaluation than they are in production. Common sources:
- Target leakage: Features that contain information about the label that would not be available at prediction time (e.g., a "diagnosis confirmed" flag included as a feature when predicting diagnosis).
- Train/test contamination: Preprocessing steps (scaling, imputation, encoding) fitted on the full dataset before splitting. Always fit transformers on training data only, then apply to validation/test.
- Temporal leakage: Using future data to predict past events. When data has a time dimension, the split must respect chronological order — never shuffle before splitting a time series.
- Indirect leakage: Features indirectly correlated with the label through a confounder (e.g., hospital specialization predicting diagnosis category because certain hospitals only treat certain diseases).
- Dynamic table leakage (production): Features joined from tables that change over time create skew — what the model saw at training time differs from what is available at serving time. Snapshot or log serving-time feature values to close this gap.
Rule: audit every feature to confirm it would be available at the exact moment of prediction in production, with no knowledge of events after the prediction timestamp.
Overfitting: Detection and Prevention
Detection
- Track training loss vs. validation loss across epochs/iterations. Divergence (training loss continues falling while validation loss plateaus or rises) is the primary signal.
- Monitor metric delta between training data, holdout data, next-day data, and live predictions (training-serving skew). Large gaps signal engineering errors or overfitting to stale signals.
- Use learning curves: plot performance vs. training set size. High variance (large gap between train and val curves) indicates overfitting; high bias (both curves plateaued at poor performance) indicates underfitting.
Prevention
- Regularization: L2 (weight decay) or L1 (sparsity) penalties on model parameters. Apply stronger regularization to features that cover few examples (rare features overfit easily).
- Early stopping: Halt training when validation metric stops improving.
- Dropout / data augmentation: For neural networks.
- Simpler models first (Google Rule #4, #14): Start with interpretable linear/logistic models. They are easier to debug, less prone to overfitting, and set a calibrated baseline. Add complexity only when simpler models demonstrably underfit.
- Hold out positional features: In ranking models, do not cross position with content features — position artificially inflates interaction rates and causes position-dependent overfitting.
- Importance-weight sampled data instead of dropping it: Arbitrary data removal introduces bias; use sampling weights (weight = 1/sampling_rate) to preserve calibration.
Cross-Validation
- k-fold CV: Split training data into k folds; train k models each using k-1 folds, evaluate on the held-out fold. Average performance across folds.
- Stratified k-fold: Maintain class proportions in every fold. Required for imbalanced datasets.
- Time-series CV (walk-forward): For temporal data, each fold trains on past data and validates on the immediately following window. Never allow future data into the training window.
- After cross-validation selects a configuration, retrain on the full training set and evaluate once on the test set.
Training-Serving Skew
Production models fail silently when serving conditions differ from training conditions.
- Schema skew: Training and serving data arrive in different formats or with different feature ranges. Apply identical schema validation and data contracts to both pipelines.
- Feature skew: Feature engineering code diverges between offline training and online serving. Reuse the same code (Rule #32) — never maintain separate implementations.
- Log serving-time features (Rule #29): Save actual feature values used during prediction and pipe them back to training logs. This is the only way to guarantee training and serving use identical data.
- Test on post-cutoff data (Rule #33): Always evaluate the final model on data with timestamps after the training cutoff. Performance should degrade gracefully, not collapse — a collapse signals temporal leakage.
Measuring Skew (Rule #37)
Track these gaps in sequence:
- Training data vs. holdout data
- Holdout data vs. next-day data
- Next-day data vs. live predictions
Large gaps at any level indicate engineering errors or time-sensitive features that have become stale.
Data Drift and Monitoring
- Define a data schema: Set expected ranges, allowed categorical values, and distribution bounds for every feature. Alert when incoming data violates schema.
- Silent failures (Rule #10): Degraded data quality (stale tables, reduced feature coverage, changed upstream sources) may not cause obvious errors but will slowly erode model performance. Instrument feature coverage and distribution statistics.
- Slice-level monitoring: Monitor metrics separately across important data subsets (region, user cohort, product category). Aggregate metrics can mask degraded performance on critical slices.
- Model freshness (Rule #8): Determine how quickly performance degrades on stale models (day-old, week-old). Set retraining schedules and staleness alerts accordingly.
- Live quality signals: When serving data lacks ground-truth labels, use human raters, user feedback signals, or gradual rollout (validate on a query subset before full deployment) to catch quality regressions.
Reproducibility
- Seed all random number generators at the start of every experiment (data shuffling, train/test split, model initialization, dropout). Record the seed with results.
- Version datasets: Tag the exact dataset snapshot used for each experiment. Re-running on a refreshed dataset invalidates comparisons.
- Deterministic splits via hashing (production): Use invariant hash keys (e.g., query + log date, not current timestamp) to assign examples to train/eval splits. This prevents data leakage across time periods and ensures consistent distribution across experimental runs.
- Pin dependency versions: Record library versions (scikit-learn, XGBoost, etc.) alongside model artifacts. Different versions can produce different results for the same code.
- Infrastructure tests (Rule #5): Validate data ingestion and feature pipelines independently from ML logic. Confirm that the model score in training environment matches the score in serving environment on the same example.
Production Readiness Checklist
- Data schema validated for training and serving inputs
- No leakage between train/validation/test splits confirmed
- Serving-time features logged and matched to training features
- Performance gate (AUC or metric threshold) checked before export (Rule #9)
- Training-serving skew measured and within acceptable bounds
- RNG seeds recorded; results are reproducible from seed + dataset snapshot
- Monitoring in place for feature distribution drift and slice-level performance
- Model freshness threshold defined and staleness alert configured
- Gradual rollout plan defined; rollback procedure documented