如何在 scikit-learn 中正确执行交叉验证？

我正在尝试对 k-nn 分类器进行交叉验证，但我对以下两种方法中哪一种正确执行交叉验证感到困惑。

training_scores = defaultdict(list)
validation_f1_scores = defaultdict(list)
validation_precision_scores = defaultdict(list)
validation_recall_scores = defaultdict(list)
validation_scores = defaultdict(list)

def model_1(seed, X, Y):
    np.random.seed(seed)
    scoring = ['accuracy', 'f1_macro', 'precision_macro', 'recall_macro']
    model = KNeighborsClassifier(n_neighbors=13)

    kfold = StratifiedKFold(n_splits=2, shuffle=True, random_state=seed)
    scores = model_selection.cross_validate(model, X, Y, cv=kfold, scoring=scoring, return_train_score=True)
    print(scores['train_accuracy'])
    training_scores['KNeighbour'].append(scores['train_accuracy'])
    print(scores['test_f1_macro'])
    validation_f1_scores['KNeighbour'].append(scores['test_f1_macro'])
    print(scores['test_precision_macro'])
    validation_precision_scores['KNeighbour'].append(scores['test_precision_macro'])
    print(scores['test_recall_macro'])
    validation_recall_scores['KNeighbour'].append(scores['test_recall_macro'])
    print(scores['test_accuracy'])
    validation_scores['KNeighbour'].append(scores['test_accuracy'])

    print(np.mean(training_scores['KNeighbour']))
    print(np.std(training_scores['KNeighbour']))
    #rest of print statments

看来第二个模型中的 for 循环是多余的。

def model_2(seed, X, Y):
    np.random.seed(seed)
    scoring = ['accuracy', 'f1_macro', 'precision_macro', 'recall_macro']
    model = KNeighborsClassifier(n_neighbors=13)

    kfold = StratifiedKFold(n_splits=2, shuffle=True, random_state=seed)
    for train, test in kfold.split(X, Y):
        scores = model_selection.cross_validate(model, X[train], Y[train], cv=kfold, scoring=scoring, return_train_score=True)
        print(scores['train_accuracy'])
        training_scores['KNeighbour'].append(scores['train_accuracy'])
        print(scores['test_f1_macro'])
        validation_f1_scores['KNeighbour'].append(scores['test_f1_macro'])
        print(scores['test_precision_macro'])
        validation_precision_scores['KNeighbour'].append(scores['test_precision_macro'])
        print(scores['test_recall_macro'])
        validation_recall_scores['KNeighbour'].append(scores['test_recall_macro'])
        print(scores['test_accuracy'])
        validation_scores['KNeighbour'].append(scores['test_accuracy'])

    print(np.mean(training_scores['KNeighbour']))
    print(np.std(training_scores['KNeighbour']))
    # rest of print statments

我在用StratifiedKFold我不确定我是否需要像 model_2 函数中那样的 for 循环cross_validate当我们传递时，函数已经使用了 splitcv=kfold作为一个论点。

我没有打电话fit方法，这样可以吗？做cross_validate自动调用还是我需要调用fit打电话之前cross_validate?

最后，如何创建混淆矩阵？我是否需要为每次折叠创建它，如果是，如何计算最终/平均混淆矩阵？

The 文档 https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html在此类问题上可以说是你最好的朋友；从这个简单的例子来看，很明显你不应该使用for循环也不调用fit。调整示例以使用KFold像你一样做：

from sklearn.model_selection import KFold, cross_validate
from sklearn.datasets import load_boston
from sklearn.tree import DecisionTreeRegressor

X, y = load_boston(return_X_y=True)
n_splits = 5
kf = KFold(n_splits=n_splits, shuffle=True)

model = DecisionTreeRegressor()
scoring=('r2', 'neg_mean_squared_error')

cv_results = cross_validate(model, X, y, cv=kf, scoring=scoring, return_train_score=False)
cv_results

Result:

{'fit_time': array([0.00901461, 0.00563478, 0.00539804, 0.00529385, 0.00638533]),
 'score_time': array([0.00132656, 0.00214362, 0.00134897, 0.00134444, 0.00176597]),
 'test_neg_mean_squared_error': array([-11.15872549, -30.1549505 , -25.51841584, -16.39346535,
        -15.63425743]),
 'test_r2': array([0.7765484 , 0.68106786, 0.73327311, 0.83008371, 0.79572363])}

如何创建混淆矩阵？我需要为每个折叠创建它吗

没有人能告诉你，如果你need为每个折叠创建一个混淆矩阵 - 这是你的选择。如果您选择这样做，最好跳过cross_validate并“手动”执行该过程 - 请参阅我的答案如何显示每个交叉验证折叠的混淆矩阵和报告（召回率、精度、fmeasure） https://stackoverflow.com/questions/53531167/how-to-display-confusion-matrix-and-report-recall-precision-fmeasure-for-eac/55050146#55050146.

如果是，如何计算最终/平均混淆矩阵？

不存在“最终/平均”混淆矩阵；如果你想计算比k如链接答案中所述（每个 k 倍一个），您需要有一个单独的验证集......