Data leakage occurs when a machine learning model gains access to information during training that will not be available during real-world inference. This can lead to misleadingly high performance metrics, even if the code runs without errors.
In a controlled experiment with 6000 objects and 20 features, a simple decision tree with a depth of 1 outperformed a random forest of 300 trees. The decision tree achieved higher accuracy because it was provided with a leaked feature containing information about the target variable, which matched the correct answer in 96.3% of cases.
Standard train-test splits do not guarantee the absence of leakage. For instance, using StandardScaler on an entire dataset before splitting allows information about the distribution of test data to influence the preprocessing of training data.
Data leakage can also arise from improper data partitioning. When dealing with user actions or time series, random splitting can cause information from the future or from the same user to appear in both training and testing sets, violating the logic of real-world prediction scenarios.
To prevent leakage, developers should use techniques like GroupShuffleSplit for user-based data or chronological splitting for time series. Proper pipelines, such as those in scikit-learn, ensure that preprocessing parameters are calculated only on training folds during cross-validation.
