您打算使用“ROC” - ROC 曲线下的区域来选择最佳调整参数,但您没有指定twoClassSummary() https://www.rdocumentation.org/packages/caret/versions/6.0-86/topics/defaultSummary它保存了这个指标。这就是警告的内容
通知你
Warning message:
In train.default(x, y, weights = w, ...) :
The metric "ROC" was not in the result set. Accuracy will be used instead.
执行车削:
library(tidyverse)
library(caret)
library(glmnet)
library(mlbench)
data(PimaIndiansDiabetes, package="mlbench")
set.seed(2323)
train.data <- PimaIndiansDiabetes
set.seed(2323)
model <- train(
diabetes ~., data = train.data, method = "glmnet",
trControl = trainControl("cv",
number = 10,
classProbs = TRUE,
savePredictions = TRUE,
summaryFunction = twoClassSummary),
tuneLength = 10,
metric="ROC" #ROC metric is in twoClassSummary
)
既然你指定了classProbs = TRUE
and savePredictions = TRUE
您可以根据预测计算任何指标。
计算精度:
model$pred %>%
filter(alpha == model$bestTune$alpha, #filter predictions for best tuning parameters
lambda == model$bestTune$lambda) %>%
group_by(Resample) %>% #group by fold
summarise(acc = sum(pred == obs)/n()) #calculate metric
#output
`summarise()` ungrouping output (override with `.groups` argument)
# A tibble: 10 x 2
Resample acc
<chr> <dbl>
1 Fold01 0.740
2 Fold02 0.753
3 Fold03 0.818
4 Fold04 0.776
5 Fold05 0.779
6 Fold06 0.753
7 Fold07 0.766
8 Fold08 0.792
9 Fold09 0.727
10 Fold10 0.789
这给出了每倍的度量。为了获得平均性能
model$pred %>%
filter(alpha == model$bestTune$alpha,
lambda == model$bestTune$lambda) %>%
group_by(Resample) %>%
summarise(acc = sum(pred == obs)/n()) %>%
pull(acc) %>%
mean
#output
0.769566
当 ROC 用作选择指标时,超参数会在所有决策阈值上进行优化。在许多情况下,使用默认决策阈值 0.5 时,所选模型的表现可能不是最优的。
插入符有一个功能thresholder() https://www.rdocumentation.org/packages/caret/versions/6.0-86/topics/thresholder
它将根据超过指定决策阈值的重新采样数据计算许多指标。
thresholder(model, seq(0, 1, length.out = 10)) #in reality I would use length.out = 100
#output
alpha lambda prob_threshold Sensitivity Specificity Pos Pred Value Neg Pred Value Precision Recall F1 Prevalence Detection Rate Detection Prevalence Balanced Accuracy Accuracy
1 0.1 0.03607775 0.0000000 1.000 0.00000000 0.6510595 NaN 0.6510595 1.000 0.7886514 0.6510595 0.6510595 1.0000000 0.5000000 0.6510595
2 0.1 0.03607775 0.1111111 0.994 0.02621083 0.6557464 0.7380952 0.6557464 0.994 0.7901580 0.6510595 0.6471463 0.9869617 0.5101054 0.6562714
3 0.1 0.03607775 0.2222222 0.986 0.15270655 0.6850874 0.8711111 0.6850874 0.986 0.8082906 0.6510595 0.6419344 0.9375256 0.5693533 0.6952837
4 0.1 0.03607775 0.3333333 0.964 0.32421652 0.7278778 0.8406807 0.7278778 0.964 0.8290127 0.6510595 0.6276316 0.8633459 0.6441083 0.7408578
5 0.1 0.03607775 0.4444444 0.928 0.47364672 0.7674158 0.7903159 0.7674158 0.928 0.8395895 0.6510595 0.6041866 0.7877990 0.7008234 0.7695147
6 0.1 0.03607775 0.5555556 0.862 0.59002849 0.7970454 0.7053968 0.7970454 0.862 0.8274687 0.6510595 0.5611928 0.7043575 0.7260142 0.7669686
7 0.1 0.03607775 0.6666667 0.742 0.75740741 0.8521972 0.6114289 0.8521972 0.742 0.7926993 0.6510595 0.4830827 0.5677204 0.7497037 0.7473855
8 0.1 0.03607775 0.7777778 0.536 0.90284900 0.9156149 0.5113452 0.9156149 0.536 0.6739140 0.6510595 0.3489918 0.3828606 0.7194245 0.6640636
9 0.1 0.03607775 0.8888889 0.198 0.98119658 0.9573810 0.3967404 0.9573810 0.198 0.3231917 0.6510595 0.1289474 0.1354751 0.5895983 0.4713602
10 0.1 0.03607775 1.0000000 0.000 1.00000000 NaN 0.3489405 NaN 0.000 NaN 0.6510595 0.0000000 0.0000000 0.5000000 0.3489405
Kappa J Dist
1 0.0000000 0.00000000 1.0000000
2 0.0258717 0.02021083 0.9738516
3 0.1699809 0.13870655 0.8475624
4 0.3337322 0.28821652 0.6774055
5 0.4417759 0.40164672 0.5329805
6 0.4692998 0.45202849 0.4363768
7 0.4727251 0.49940741 0.3580090
8 0.3726156 0.43884900 0.4785352
9 0.1342372 0.17919658 0.8026597
10 0.0000000 0.00000000 1.0000000
现在根据您所需的指标选择一个阈值并使用它。通常用于不平衡数据的指标科恩的卡帕 https://en.wikipedia.org/wiki/Cohen%27s_kappa, 尤登的J https://en.wikipedia.org/wiki/Youden%27s_J_statistic or 马修斯相关系数(MCC) https://en.wikipedia.org/wiki/Matthews_correlation_coefficient。这里有一个体面的paper https://www.semanticscholar.org/paper/Predictive-Accuracy-%3A-A-Misleading-Performance-for-Akosa/8eff162ba887b6ed3091d5b6aa1a89e23342cb5c就此事。
请注意,由于该数据用于查找最佳阈值,因此通过这种方式获得的性能将存在乐观偏差。为了评估所选决策阈值的性能,最好使用几个独立的测试集。换句话说,我建议嵌套重采样,您可以使用内部折叠优化参数和阈值并评估外部折叠。
这是一个解释 https://stackoverflow.com/questions/62183291/statistical-test-with-test-data/62193116#62193116关于如何使用插入符号和回归的嵌套重采样。需要进行一些修改才能使其与具有优化阈值的分类一起使用。
请注意,这不是选择最佳决策阈值的唯一方法。另一种方法是先验地选择所需的度量(例如 MCC)并将决策阈值视为要与所有其他超参数联合调整的超参数。我相信创建自定义模型时插入符号不支持这一点。