Huggingface 分类与预测作斗争

我正在微调 longformer，然后使用进行预测TextClassificationPipeline and model(**inputs)方法。我不确定为什么会得到不同的结果

import pandas as pd
import datasets
from transformers import LongformerTokenizerFast, LongformerForSequenceClassification, Trainer, TrainingArguments, LongformerConfig
import torch.nn as nn
import torch
from torch.utils.data import DataLoader#Dataset, 
import numpy as np
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from tqdm import tqdm
#import wandb
import os
from datasets import Dataset
from transformers import TextClassificationPipeline, AutoTokenizer, AutoModelForSequenceClassification

tokenizer = LongformerTokenizerFast.from_pretrained('folder_path/', max_length = maximum_len)

从保存的位置加载微调模型。使用原始分词器

saved_location='c:/xyz'
model_saved=AutoModelForSequenceClassification.from_pretrained(saved_location)
pipe = TextClassificationPipeline(model=model_saved, tokenizer=tokenizer, device=0)#tokenizer_saved, padding=True, truncation=True)
prediction = pipe(["The text to predict"], return_all_scores=True)
prediction
[[{'label': 'LABEL_0', 'score': 0.7107483148574829},
  {'label': 'LABEL_1', 'score': 0.2892516553401947}]]

第二种方法

inputs = tokenizer("The text to predict", return_tensors="pt").to(device)
outputs = model_saved(**inputs)#, labels=labels)
print (outputs['logits'])
#tensor([[ 0.4552, -0.4438]], device='cuda:0', grad_fn=<AddmmBackward0>)
torch.sigmoid(outputs['logits'])
#tensor([[0.6119, 0.3908]], device='cuda:0', grad_fn=<SigmoidBackward0>)

AutoModelForSequenceClassification返回概率0.71 and 0.29。当我看第二种方法时。它返回逻辑0.4552, -0.4438转换为概率0.6119, 0.3908

#更新1

第一个链接文本分类管道 https://huggingface.co/docs/transformers/v4.17.0/en/main_classes/pipelines#transformers.TextClassificationPipeline克罗诺克的回答如下

function_to_apply (str, optional, defaults to "default") — The function to apply to the model outputs in order to retrieve the scores. Accepts four different values:
"default": if the model has a single label, will apply the sigmoid function on the output. If the model has several labels, will apply the softmax function on the output.
"sigmoid": Applies the sigmoid function on the output.
"softmax": Applies the softmax function on the output.
"none": Does not apply any function on the output.

因为这是一个二元分类问题（单标签），它不应该应用 sigmoid 吗？

我假设model.config.num_labels==2，如果是这样的话，则文本分类管道 https://huggingface.co/docs/transformers/v4.17.0/en/main_classes/pipelines#transformers.TextClassificationPipeline应用 softmax 而不是 sigmoid 来计算概率 (code https://github.com/huggingface/transformers/blob/198c335d219a5eb4d3f124fdd1ce1a9cd9f78a9b/src/transformers/pipelines/text_classification.py#L142).

import torch

logits = torch.tensor([ 0.4552, -0.4438])
print(torch.softmax(logits,0))

Output:

tensor([0.7107, 0.2893])