Cross-Validation

k folds, k scores

A single train/test split gives a noisy estimate: a lucky or unlucky split can move the score by several points. k-fold cross-validation splits the training data into k parts, trains on k-1 and validates on the remaining one, k times, and averages the scores. Use it to compare models and tune settings, report the mean and spread, and keep a final test set aside for one last check. For time series use time-ordered splits; for grouped data (several rows per customer) use group-aware folds.

Single splits versus 5-fold CV, run

I ran this with Python 3, numpy 2.5.3 and scikit-learn 1.9.1, using fixed random seeds. The same decision tree scores anywhere from 0.881 to 0.951 on four different random splits. Five-fold cross-validation gives a mean of 0.917 with a standard deviation of 0.016.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
single = [round(DecisionTreeClassifier(random_state=0).fit(a, c).score(b, d), 3)
          for a, b, c, d in [train_test_split(X, y, random_state=s) for s in range(4)]]
print("four different single splits:", single)
scores = cross_val_score(DecisionTreeClassifier(random_state=0), X, y, cv=5)
print("5-fold scores:", scores.round(3), "mean", round(scores.mean(), 3), "std", round(scores.std(), 3))

Output:

four different single splits: [0.881, 0.951, 0.916, 0.944]
5-fold scores: [0.904 0.921 0.912 0.947 0.903] mean 0.917 std 0.016

Report mean and spread

Write 0.917 plus or minus 0.016 rather than a single number, so readers see how stable it is.

त्वरित जाँच: Why use cross-validation instead of one split?

• It averages over several splits, giving a more reliable estimate
• It makes training faster
• It removes the need for data
• It guarantees zero error