R：将 LIME 应用于 Quanteda 文本模型的问题

这是我的修改版本上一个问题 https://stackoverflow.com/q/50273919/6327771：我正在尝试在我的设备上运行 LIMEquanteda依赖的文本模型特朗普和克林顿推文数据 https://www.kaggle.com/benhamner/clinton-trump-tweets/data#。我按照 Thomas Pedersen 在他的文章中给出的示例运行它了解石灰 https://cran.r-project.org/web/packages/lime/vignettes/Understanding_lime.html和提供的有用的答案@黄伟煌 https://stackoverflow.com/a/50275665/6327771:

library(dplyr)
library(stringr)
library(quanteda)
library(lime)

#data prep
tweet_csv <- read_csv("tweets.csv")

# creating corpus and dfm for train and test sets

get_matrix <- function(df){
  corpus <- quanteda::corpus(df)
  dfm <- quanteda::dfm(corpus, remove_url = TRUE, remove_punct = TRUE,     remove = stopwords("english"))
}

set.seed(32984)
trainIndex <- sample.int(n = nrow(tweet_csv), size =     floor(.8*nrow(tweet_csv)), replace = F)

train_dfm <- get_matrix(tweet_csv$text[trainIndex])
train_raw <- tweet_csv[, c("text", "tweet_num")][as.vector(trainIndex), ]
train_labels <- tweet_csv$author[as.vector(trainIndex)] == "realDonaldTrump"

test_dfm <- get_matrix(tweet_csv$text[-trainIndex])
test_raw <- tweet_csv[, c("text", "tweet_num")][-as.vector(trainIndex), ]
test_labels <- tweet_csv$author[-as.vector(trainIndex)] == "realDonaldTrump"

#### make sure that train & test sets have exactly same features
test_dfm <- dfm_select(test_dfm, train_dfm)

### Naive Bayes model using quanteda::textmodel_nb ####
nb_model <- quanteda::textmodel_nb(train_dfm, train_labels)
nb_preds <- predict(nb_model, test_dfm) #> 0.5


# select only correct predictions
predictions_tbl <- data.frame(predict_label = nb_preds$nb.predicted,
                          actual_label = test_labels,
                          tweet_name = rownames(nb_preds$posterior.prob)
) %>%
  mutate(tweet_num = 
       as.integer(
         str_trim(
           str_replace_all(tweet_name, "text", ""))
     )) 


correct_pred <- predictions_tbl %>%
  filter(actual_label == predict_label) 

# pick a sample of tweets for explainer 
tweets_to_explain <- test_raw %>%
  filter(tweet_num %in% correct_pred$tweet_num) %>% 
  head(4)



### set up correct model class and predict functions 
class(nb_model)

model_type.textmodel_nb_fitted <- function(x, ...) {
  return("classification")
}


# have to modify the textmodel_nb_fitted so that 

predict_model.textmodel_nb_fitted <- function(x, newdata, type, ...) {
  X <- corpus(newdata)
  X <- dfm_select(dfm(X), x$data$x)   
  res <- predict(x, newdata = X, ...)
  switch(
    type,
    raw = data.frame(Response = res$nb.predicted, stringsAsFactors = FALSE),
    prob = as.data.frame(res$posterior.prob, check.names = FALSE)
  )  
}


### run the explainer - no problems here 
explainer <- lime(tweets_to_explain$text, # lime returns error on different features in explainer and explanations, even if I use the same dataset in both. Raised an issue on Github and asked a question on SO
              model = nb_model,
              preprocess = get_matrix) 

但是当我运行解释器时......

corr_explanation <- lime::explain(tweets_to_explain$text, 
                              explainer, 
                              n_labels = 1,
                              n_features = 6,
                              cols = 2,
                              verbose = 0)

...我收到以下错误：

UseMethod(“语料库”) 中的错误： 没有适用于“语料库”的方法应用于类“c('dfm', 'dgCMatrix', 'CsparseMatrix', 'dsparseMatrix', 'generalMatrix', 'dCsparseMatrix', 'dMatrix', 'sparseMatrix', 'compMatrix '、'矩阵'、'xMatrix'、'mMatrix'、'Mnumeric'、'replValueSp')"

又回到了申请corpus() to newdata:

5.corpus(newdata) 
4.predict_model.textmodel_nb_fitted(x = explainer$model, newdata = permutations_tokenized, 
type = o_type) 
3.predict_model(x = explainer$model, newdata = permutations_tokenized, 
type = o_type) 
2.explain.character(tweets_to_explain$text, explainer, n_labels = 1, 
n_features = 6, cols = 2, verbose = 0) 
1.lime::explain(tweets_to_explain$text, explainer, n_labels = 1, 
n_features = 6, cols = 2, verbose = 0) 

但我不明白为什么这会导致任何问题，因为新数据是文本向量？

感谢您的任何提示

corpus不必运行。尝试重新定义predict_model.textmodel_nb_fitted如下，唯一的修改是添加dfm_select step:

predict_model.textmodel_nb_fitted <- function(x, newdata, type, ...) {
  X <- dfm_select(dfm(newdata), x$data$x)   
  res <- predict(x, newdata = X, ...)
  switch(
    type,
    raw = data.frame(Response = res$nb.predicted, stringsAsFactors = FALSE),
    prob = as.data.frame(res$posterior.prob, check.names = FALSE)
  )  
}

As your traceback()输出显示，corpus抛出错误。为了调试，我插入了print(str(newdata))在第一行predict_model.textmodel_nb_fitted功能。这表明newdata已经是一个dfm对象，因此可以直接传递给predict.textmodel_nb_fitted（处理后dfm_select).

在较新的版本中quanteda, textmodel_nb()返回类的对象textmodel_nb,textmodel, and list。这首先需要一个相应的方法model_type:

model_type.textmodel_nb <- function(x, ...) {
  return("classification")
}

然后我们还必须写一个textmodel_nb方法用于predict_model:

predict_model.textmodel_nb <- function(x, newdata, type, ...) {
  X <- dfm_select(dfm(newdata), x$x)   
  res <- predict(x, newdata = X, ...)
  switch(
    type,
    raw = data.frame(Response = res$nb.predicted, stringsAsFactors = FALSE),
    prob = as.data.frame(res$posterior.prob, check.names = FALSE)
  )  
}

请注意，第二个参数dfm_select与其中不同的是predict_model.textmodel_nb_fitted（来自答案的原始版本）。这是因为该结构的x对象——输出textmodel_nb()- 已经改变。