我正在尝试将标记生成器保存在 Huggingface 中,以便稍后可以从不需要访问互联网的容器中加载它。
BASE_MODEL = "distilbert-base-multilingual-cased"
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.save_vocabulary("./models/tokenizer/")
tokenizer2 = AutoTokenizer.from_pretrained("./models/tokenizer/")
然而,最后一行给出了错误:
OSError: Can't load config for './models/tokenizer3/'. Make sure that:
- './models/tokenizer3/' is a correct model identifier listed on 'https://huggingface.co/models'
- or './models/tokenizer3/' is the correct path to a directory containing a config.json file
变形金刚版本:3.1.0
如何从 Pytorch 中的预训练模型加载保存的分词器 https://stackoverflow.com/questions/58417374/how-to-load-the-saved-tokenizer-from-pretrained-model-in-pytorch不幸的是没有帮助。
Edit 1
感谢@ashwin 下面的回答,我尝试过save_pretrained
相反,我收到以下错误:
OSError: Can't load config for './models/tokenizer/'. Make sure that:
- './models/tokenizer/' is a correct model identifier listed on 'https://huggingface.co/models'
- or './models/tokenizer/' is the correct path to a directory containing a config.json file
the contents of the tokenizer folder is below:
我尝试重命名tokenizer_config.json
to config.json
然后我得到了错误:
ValueError: Unrecognized model in ./models/tokenizer/. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: retribert, t5, mobilebert, distilbert, albert, camembert, xlm-roberta, pegasus, marian, mbart, bart, reformer, longformer, roberta, flaubert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm, ctrl, electra, encoder-decoder