data/knowledge_base/data_leakage.md

Data Leakage

What Is Data Leakage

Data leakage occurs when information from outside the training distribution — typically from the target variable or future time periods — is used as a feature during training. The model learns a signal that won't exist at prediction time, producing deceptively good training/CV metrics but poor real-world performance.

Leakage is one of the most common reasons a model performs well in development but fails in production.

---

Types of Leakage

1. Target Leakage

A feature is directly or indirectly derived from the target, but would not be known at prediction time.

Example: Predicting loan default. Feature collection_letter_sent = 1 is added because it correlates strongly with default. But a collection letter is sent because the person defaulted — this feature wouldn't exist at the time you're trying to predict default.

How to detect: Features with suspiciously high correlation to the target. Features that "shouldn't" be available at prediction time. Test set performance dramatically worse than validation performance after deployment.

2. Train-Test Contamination

Preprocessing steps that depend on the full dataset (including test data) are applied before splitting. The test set "leaks" statistics into the training pipeline.

Common mistakes:

• Fitting StandardScaler on the full dataset before splitting
• Imputing missing values using the mean of the full dataset
• Fitting SelectKBest or PCA on the full dataset

Correct approach: Fit all preprocessing on training data only. Apply (transform) to validation/test data using training statistics.

# Wrong
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # uses full dataset
X_train, X_test = train_test_split(X_scaled)

# Correct
X_train, X_test = train_test_split(X)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)  # only transform, no fit

Always use sklearn Pipelines to prevent this — the pipeline fits only on training data within each CV fold.

3. Temporal Leakage

For time-series data, using future information to predict the past.

Examples:

• Shuffling before splitting (standard train_test_split on time-series)
• Using a 7-day rolling average that includes the current day's value
• Using data from t+1 to predict t

Correct approach: Always sort by time, split chronologically. Use TimeSeriesSplit for cross-validation. When engineering lag features, ensure lags are strictly in the past.

4. Group Leakage

Multiple rows from the same entity (patient, user, session) appear in both train and test. The model memorizes entity-level patterns rather than generalizing.

Example: Medical dataset with 10 measurements per patient. Standard split puts some measurements from the same patient in train and test. The model learns patient-specific patterns (blood type, age) that inflate performance.

Correct approach: Use GroupKFold or GroupShuffleSplit, ensuring all rows from an entity are in the same fold.

5. Feature Leakage via Aggregation

Aggregated features (e.g., "average purchase amount for this customer") computed over the full dataset include test set rows.

Correct approach: Compute aggregations only from training data, or use proper fold-aware feature engineering.

---

Detection Methods

Check feature-target correlations

Unusually high correlation (>0.9) between a single feature and the target is suspicious — especially for features that "shouldn't" be that informative.

Compare validation to production performance

If a model that scores 0.95 AUC in CV drops to 0.65 in production, leakage is the first thing to investigate.

Permutation importance after deployment

If features that were important in training have near-zero importance on new data, they may have been leaky.

Remove suspicious features and retrain

If removing a feature causes minimal drop in CV performance but the feature seemed too powerful, it was likely leaky.

---

Pipelines Prevent Leakage

sklearn Pipeline ensures that all fitting (scaler, imputer, encoder, selector) happens only on the training fold within each CV split.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.ensemble import GradientBoostingClassifier

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("model", GradientBoostingClassifier()),
])

# Safe: Pipeline fits imputer+scaler only on each training fold
scores = cross_val_score(pipe, X, y, cv=StratifiedKFold(5))

---

Common Scenarios and Fixes

Scenario	Leaky approach	Fix
Scaling	fit_transform(X_full)	Fit only on X_train
Imputation	Mean of full dataset	Mean of training fold only
Encoding	Frequency encoding on full dataset	Pipeline or training-only counts
Time series	train_test_split(shuffle=True)	Chronological split
Multi-row entities	Random split	GroupKFold
Target encoding	Encode with full y	Fold-aware target encoding

---

Key Rule

Any transformation that uses information from the target or from data points outside the current training fold is leakage. When in doubt, ask: "Would I have access to this feature at the moment I need to make a prediction in production?" If no — it's leakage.