Train, Validation and Test Split Best Practices

A training set is used to fit model parameters. Validation data help choose hyperparameters, features, thresholds, or model variants. A final test set estimates performance after those choices are settled. Repeatedly tuning against the test set makes it part of model selection and can produce an optimistic estimate. The split strategy should reflect how predictions will be used. Random stratified splits can be useful when future examples are independent and come from a similar population. If the same person, household, device, or source appears in multiple rows, group-aware splitting may be needed to estimate performance on new groups. For future prediction, time-based splitting usually better reflects deployment than shuffling historical events. Scikit-learn documents GroupKFold and TimeSeriesSplit for such settings. All preprocessing that learns from data—such as scaling, imputation, feature selection, or vocabulary building—should be fitted using training data within each fold, then applied to held-out data. Otherwise information from validation or test can leak into training. Duplicate and near-duplicate examples across partitions can also inflate evaluation. Check split membership after deduplication and before augmentation or oversampling. There is no universal ratio such as 70/15/15. Choose sizes based on data volume, class balance, uncertainty, and the evaluation goal. Preserve a final holdout where feasible, report the split method and random seed, and consider confidence intervals or repeated cross-validation for development. A held-out test is still only an estimate for the population and time period it represents.