我对 PyTorch 和 Huggingface-transformers 比较陌生,并对此尝试了 DistillBertForSequenceClassificationKaggle-数据集 https://www.kaggle.com/c/nlp-getting-started.
from transformers import DistilBertForSequenceClassification
import torch.optim as optim
import torch.nn as nn
from transformers import get_linear_schedule_with_warmup
n_epochs = 5 # or whatever
batch_size = 32 # or whatever
bert_distil = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')
#bert_distil.classifier = nn.Sequential(nn.Linear(in_features=768, out_features=1), nn.Sigmoid())
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(bert_distil.parameters(), lr=0.1)
X_train = []
Y_train = []
for row in train_df.iterrows():
seq = tokenizer.encode(preprocess_text(row[1]['text']), add_special_tokens=True, pad_to_max_length=True)
X_train.append(torch.tensor(seq).unsqueeze(0))
Y_train.append(torch.tensor([row[1]['target']]).unsqueeze(0))
X_train = torch.cat(X_train)
Y_train = torch.cat(Y_train)
running_loss = 0.0
bert_distil.cuda()
bert_distil.train(True)
for epoch in range(n_epochs):
permutation = torch.randperm(len(X_train))
j = 0
for i in range(0,len(X_train), batch_size):
optimizer.zero_grad()
indices = permutation[i:i+batch_size]
batch_x, batch_y = X_train[indices], Y_train[indices]
batch_x.cuda()
batch_y.cuda()
outputs = bert_distil.forward(batch_x.cuda())
loss = criterion(outputs[0],batch_y.squeeze().cuda())
loss.requires_grad = True
loss.backward()
optimizer.step()
running_loss += loss.item()
j+=1
if j == 20:
#print(outputs[0])
print('[%d, %5d] running loss: %.3f loss: %.3f ' %
(epoch + 1, i*1, running_loss / 20, loss.item()))
running_loss = 0.0
j = 0
[1, 608] 运行损失:0.689 损失:0.687
[1、1248]运行损失:0.693 损失:0.694
[1, 1888] 运行损失:0.693 损失:0.683
[1, 2528] 运行损失:0.689 损失:0.701
[1、3168]运行损失:0.690 损失:0.684
[1、3808]运行损失:0.689 损失:0.688
[1, 4448] 运行损失:0.689 损失:0.692 等等...
无论我尝试什么,损失从未减少,甚至增加,预测也没有变得更好。在我看来,我忘记了一些事情,所以权重实际上没有更新。有人有主意吗?
氧
我尝试过什么
- Different loss functions
- One-Hot 编码与单个神经元输出
- 不同的学习率和优化器
- 我什至将所有目标更改为只有一个标签,但即便如此,网络也没有收敛。