A recent experiment demonstrated that changing the random_state parameter during data splitting significantly impacts machine learning model performance. In a test using a dataset of 1200 objects with 20 features, a single decision tree model showed accuracy fluctuations ranging from 64% to 79.33% across 100 different random splits.
The study highlights that random_state influences two distinct sources of variability: the composition of training and testing sets, and the internal stochastic processes of algorithms like Random Forest. For the latter, a separate test on 30 runs showed an accuracy variance of 4.33 percentage points when only the model's internal seed was modified.
Researchers warn against selecting a random_state based on the highest test metric, as this constitutes data leakage from the test set. Instead, cross-validation is recommended to obtain more stable performance estimates. Using a 5-fold StratifiedKFold, the average accuracy for logistic regression was 73.83%, while the decision tree achieved 71.58%.
The experiment concluded that comparing models based on a single random split is unreliable. Even when models are evaluated under identical conditions, their relative performance can shift depending on the specific data distribution in the test subset. Therefore, minor differences in accuracy between algorithms on a single split do not necessarily indicate a stable performance advantage.
