Cross-Validation Using K-Fold With Scikit-Learn - GeeksforGeeks (2024)

Cross-validation involves repeatedly splitting data into training and testing sets to evaluate the performance of a machine-learning model. One of the most commonly used cross-validation techniques is K-Fold Cross-Validation. In this article, we will explore the implementation of K-Fold Cross-Validation using Scikit-Learn, a popular Python machine-learning library.

What is K-Fold Cross Validation?

In K-Fold cross-validation, the input data is divided into ‘K’ number of folds, hence the name K Fold. The model undergoes training with K-1 folds and is evaluated on the remaining fold. This procedure is performed K times, where each fold is utilized as the testing set one time. The performance metrics are averaged across K iterations to offer a more reliable evaluation of the model’s performance.

Example: Suppose we specified the fold as 10 (k = 10), then the K-Fold cross-validation splits the input data into 10 folds, which means we have 10 sets of data to train and test our model. So for every iteration, the model uses one fold as test data and the remaining as training data (9 folds). Every time, it picks a different fold for evaluation, and the result is an array of evaluation scores for each fold.

K-Fold With Scikit-Learn

Let’s look at how to implement K-Fold cross-validation using Scikit-Learn. To achieve this, we need to import the KFold class from sklearn.model_selection. Let’s look at the KFold class from Scikit-Learn, its parameters, and its methods.

sklearn.model_selection.KFold(n_splits=5, *, shuffle=False, random_state=None)

PARAMETERS:

• n_splits (int, default=5): Number of folds. Must be at least 2.
• shuffle (bool, default=False): Whether to shuffle the data before splitting into batches. Note that the samples within each split will not be shuffled
• random_state (int, default=None): When shuffle is True, random_state affects the ordering of the indices, which controls the randomness of each fold. Otherwise, this parameter has no effect.

METHODS:

• get_metadata_routing(): Get metadata routing of this object.
• get_n_splits(X=None, y=None, groups=None): Returns the number of splitting iterations in the cross-validator. Here X,y and groups are objects.
• split(X, y=None, groups=None): Generate indices to split data into training and test set. Here X is an array which holds number of samples and number of features, y is the target variable for supervised learning problems, groups is the samples used while splitting the data into training / test set.

Let’s create a synthetic regression dataset to analyse how the K-Fold split works. The code is as follows:

import numpy as npfrom sklearn import datasetsfrom sklearn.model_selection import KFold# synthetic regression datasetX, y = datasets.make_regression( n_samples=10, n_features=1, n_informative=1, noise=0, random_state=0)# KFold splitkf = KFold(n_splits=4)for i, (train_index, test_index) in enumerate(kf.split(X)): print(f"Fold {i}:") print(f" Training dataset index: {train_index}") print(f" Test dataset index: {test_index}")

Output:

Fold 0:
 Training dataset index: [3 4 5 6 7 8 9]
 Test dataset index: [0 1 2]
Fold 1:
 Training dataset index: [0 1 2 6 7 8 9]
 Test dataset index: [3 4 5]
Fold 2:
 Training dataset index: [0 1 2 3 4 5 8 9]
 Test dataset index: [6 7]
Fold 3:
 Training dataset index: [0 1 2 3 4 5 6 7]
 Test dataset index: [8 9]

In the above code we created a synthetic regression dataset by using make_regression() method from sklearn. Here X is the input set and y is the target data (label). The KFold class divides the input data into four folds using the split() method. Hence, it has a total of four iterations (4 folds). Hope you noticed that for the entire iterations, the train index and test index are different, and it also considered the entire data for training. Let’s check the number of splits using the get_n_splits() method.

kf.get_n_splits(X)

Output:

4

Visualizing K-Fold Cross-Validation Behavior

We can create a classification dataset and visualize the behaviour of K-Fold cross-validation. The code is as follows:

from sklearn.datasets import make_classification# define datasetX, y = make_classification( n_samples=100, n_features=20, n_informative=15, n_redundant=5)# prepare the K-Fold cross-validation proceduren_splits = 10cv = KFold(n_splits=n_splits)

Using the make_classification() method, we created a synthetic binary classification dataset of 100 samples with 20 features and prepared a K-Fold cross-validation procedure for the dataset with 10 folds. Then we displayed the training and test data for each fold. You can notice how the data is divided among the training and test sets for each fold.

Let’s visualise K-Fold cross validation behavior in Sklearn. The code is as follows:

import matplotlib.pyplot as pltfrom matplotlib.patches import Patchimport numpy as npdef plot_kfold(cv, X, y, ax, n_splits, xlim_max=100): """ Plots the indices for a cross-validation object. Parameters: cv: Cross-validation object X: Feature set y: Target variable ax: Matplotlib axis object n_splits: Number of folds in the cross-validation xlim_max: Maximum limit for the x-axis """ # Set color map for the plot cmap_cv = plt.cm.coolwarm cv_split = cv.split(X=X, y=y) for i_split, (train_idx, test_idx) in enumerate(cv_split): # Create an array of NaNs and fill in training/testing indices indices = np.full(len(X), np.nan) indices[test_idx], indices[train_idx] = 1, 0 # Plot the training and testing indices ax_x = range(len(indices)) ax_y = [i_split + 0.5] * len(indices) ax.scatter(ax_x, ax_y, c=indices, marker="_", lw=10, cmap=cmap_cv, vmin=-0.2, vmax=1.2) # Set y-ticks and labels y_ticks = np.arange(n_splits) + 0.5 ax.set(yticks=y_ticks, yticklabels=range(n_splits), xlabel="X index", ylabel="Fold", ylim=[n_splits, -0.2], xlim=[0, xlim_max]) # Set plot title and create legend ax.set_title("KFold", fontsize=14) legend_patches = [Patch(color=cmap_cv(0.8), label="Testing set"), Patch(color=cmap_cv(0.02), label="Training set")] ax.legend(handles=legend_patches, loc=(1.03, 0.8))# Create figure and axisfig, ax = plt.subplots(figsize=(6, 3))plot_kfold(cv, X, y, ax, n_splits)plt.tight_layout()fig.subplots_adjust(right=0.6)

Output

in the above code, we used matplotlib to visualize the sample plot for indices of a k-fold cross-validation object. We generated training or test visualizations for each CV split. Here, we filled the indices with training or test groups using Numpy and plotted the indices using the scatter() method. The cmap parameter specifies the color of the training and test sets, and the lw parameter sets the width of each fold. Finally, by using the set() method, we formatted the X and Y axes.

Logistic Regression Model & K-Fold Cross Validating

Now let’s create a logistic regression model and cross-validate it using K-Fold. The code is as follows:

from numpy import meanfrom numpy import stdfrom sklearn.linear_model import LogisticRegressionfrom sklearn.model_selection import cross_val_score# create modellog_reg = LogisticRegression()# evaluate modelscores = cross_val_score( log_reg, X, y, scoring='accuracy', cv=cv, n_jobs=-1)# accuracyprint('Accuracy: %.3f ,\nStandard Deviations :%.3f' % (mean(scores), std(scores)))

Output:

Accuracy: 0.810 ,
Standard Deviations :0.114

In the above code, we make use of the cross_val_score() method to evaluate a score by k-fold cross-validation. Here, we passed the logistic regression model and evaluation procedure (K-Fold) as parameters. The accuracy is the evaluation metric (scoring parameter) that we used to score the dataset.

Cross-Validating Different Regression Models Using K-Fold (California Housing Dataset)

Now it’s time to cross-validate different regression models using K-Fold, and we can analyze the performance of each model. Let’s make use of the California Housing dataset from Sklearn. The code is as follows:

from sklearn.datasets import fetch_california_housing# fetch california housing datahousing = fetch_california_housing()print("Dataset Shape:", housing.data.shape, housing.target.shape)print("Dataset Features:", housing.feature_names)

Output

Dataset Shape: (20640, 8) (20640,)
Dataset Features: ['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 
'Population', 'AveOccup', 'Latitude', 'Longitude']

Here we make use of the fetch_california_housing() method from the sklearn dataset. The dataset consists of 20,640 samples and 9 features (including the label).

Here, the dataset contains only numerical features, and there are no missing values. So we don’t need to deal with text attributes or missing values; all we need to do is scale the features.

Let’s scale the features and apply K-Fold to the dataset.

from sklearn.preprocessing import StandardScalerfrom sklearn.model_selection import KFoldimport numpy as npX_housing = housing.datay_housing = housing.target# Scaling the datascaler = StandardScaler()X_scaler = scaler.fit_transform(X_housing)# K-Fold splitcnt = 0n_splits = 10kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)for train_index, test_index in kf.split(X_scaler, y_housing): print(f'Fold:{cnt}, Train set: {len(train_index)}, \ Test set:{len(test_index)}') cnt += 1

Output

Fold:0, Train set: 18576, Test set:2064
Fold:1, Train set: 18576, Test set:2064
Fold:2, Train set: 18576, Test set:2064
Fold:3, Train set: 18576, Test set:2064
Fold:4, Train set: 18576, Test set:2064
Fold:5, Train set: 18576, Test set:2064
Fold:6, Train set: 18576, Test set:2064
Fold:7, Train set: 18576, Test set:2064
Fold:8, Train set: 18576, Test set:2064
Fold:9, Train set: 18576, Test set:2064

Here, we scaled the features using the StandardScaler() method from Sklearn and passed the scaled features to the fit_transform() method. Then we prepared the K-Fold validation procedure, where we set the folds as 10 and mixed the dataset by setting the shuffle parameter as true.

Let’s visualise the split using matplotlib.

fig, ax = plt.subplots(figsize=(6, 3))plot_kfold(kf, X_scaler, y_housing, ax, n_splits, xlim_max=2000)# Make the legend fitplt.tight_layout()fig.subplots_adjust(right=0.7)

Output

We make use of the same plot_cv_indices() method (explained above) to visualize the data split. Hope you noticed that in the above plot diagram, the training and test sets got shuffled up. This is because we set the shuffle parameter in K-Fold as true. This helps in considering data from different section.

Now let’s create different regression models and apply K-fold cross validation. The code is as follows:

from sklearn.linear_model import LinearRegressionfrom sklearn.tree import DecisionTreeRegressorfrom sklearn.ensemble import RandomForestRegressorfrom sklearn.model_selection import cross_val_scoredef cross_validation(reg_model, housing_prepared, housing_labels, cv): scores = cross_val_score( reg_model, housing_prepared, housing_labels, scoring="neg_mean_squared_error", cv=cv) rmse_scores = np.sqrt(-scores) print("Scores:", rmse_scores) print("Mean:", rmse_scores.mean()) print("StandardDeviation:", rmse_scores.std())print("----- Linear Regression Model Cross Validation ------")lin_reg = LinearRegression()cross_validation(lin_reg, X_scaler, y_housing, kf)print("")print("----- Decision Tree Regression Model Cross Validation ------")tree_reg = DecisionTreeRegressor()cross_validation(tree_reg, X_scaler, y_housing, kf)print("")print("----- Random Forest Regression Model Cross Validation ------")forest_reg = RandomForestRegressor()cross_validation(forest_reg, X_scaler, y_housing, kf)

Output

----- Linear Regression Model Cross Validation ------
Scores: [0.74766431 0.74372259 0.6936579 0.75776228 0.69926807 0.72690314
 0.74241379 0.68908607 0.75124511 0.74163695]
Mean: 0.7293360220706322
StandardDeviation: 0.02440550831772841
----- Decision Tree Regression Model Cross Validation ------
Scores: [0.69024329 0.71299152 0.72902583 0.74687543 0.73311366 0.70912615
 0.71031728 0.70438177 0.71907938 0.74508813]
Mean: 0.7200242426779767
StandardDeviation: 0.01731035436143824
----- Random Forest Regression Model Cross Validation ------
Scores: [0.50050277 0.49624521 0.47534694 0.522097 0.48679587 0.51611116
 0.48861124 0.46187822 0.50740703 0.50927282]
Mean: 0.4964268280240172
StandardDeviation: 0.017721367101897926

In the above code, we created three different regression models (Linear, Decision Tree and Random Forest regression) and identified the prediction error using cross-validation for each model. The cross_val_score() method makes use of neg_mean_squared_error as an evaluation metric (scoring parameter) and K-fold as the cross-validation procedure. Here, we randomly split the training set into 10 distinct subsets called folds. So the K-Fold cross-validation feature can train and evaluate the model 10 times by picking a different fold each time and training on the other 9 folds.

You can notice that the decision tree has a mean prediction error of $72002, whereas the linear regression score is $72933. The Random Forest Regressor seems to be a promising model with a prediction error of $49642.

Once you have identified a promising model, you can fine tune the particular model and increase the model performance .

Advantages & Disadvantages of K-Fold Cross Validation

Advantages of K-Fold Cross Validation

• It has a great positive impact on reducing underfitting and overfitting. It considers most of the data for training and validation.
• Model performance analysis on each fold helps to understand the variation of input data and also provides more insights to fine-tune the model.
• It can efficiently handle unbalanced data and be used for hyperparameter tuning.

Disadvantage of K-Fold Cross Validation

• The approach can be computationally expensive.

Additional Information

Apart from K-Fold cross-validation, there are a few other variations of K-Fold techniques. A few of them are:

• Repeated K-Fold: It can be used when one requires to run KFold n times, producing different splits in each repetition.
• Stratified K-Fold: It is variation of K-Fold which returns startified sample
• Group K-Fold:It is a variation of k-fold which ensures that the same group is not represented in both testing and training sets.
• StratifiedGroupKFold: It is a cross-validation scheme that combines both StratifiedKFold and GroupKFold.

Conclusions

We have discussed the importance of K-Fold cross-validation technique in machine learning and gone through how it can be implemented using Sklearn. Hope you understood how K-fold methodology can increase model performance by avoiding overfitting and underfitting. We also analyzed the performance of different regression models, which helped us choose the most promising model for prediction.

Frequently Asked Questions (FAQs)

Q. What is K in K fold cross validation?

K represents the number of folds, or subsets, into which the dataset is divided for cross-validation. Based on the K value, the dataset splits and creates k number of folds. For example, if k = 10, then it becomes 10-fold cross-validation.

Q. Is there any formal rule to choose K value?

There is no formal rule, but normally one performs k-fold cross-validation using k = 5 or k = 10, as these values suffer neither from excessively high bias nor from very high variability.

Q. What are the factors to consider while choosing K value?

• The value of k is chosen in such a way that the data samples (train/test) should be large enough to represent the broader dataset.
• Choosing k as 10 results in a model with low bia and modest variance.
• Choosing k as n (n = size of the dataset) ensures each sample is used in a holdout dataset. This is called leave-one-out cross-validation.

Q. What is cross_val_score() method?

The corss_val_score() method evaluates a score by cross-validation, and it returns the array of scores of the estimator for each run of the cross-validation. Here we can specify the cross-validation procedure to validate the model (in our case, it is k-fold).