设置vocabulary
明确意味着没有从数据中学习词汇。如果你不设置它,你会得到:
>>> v = CountVectorizer(ngram_range=(1, 2))
>>> pprint(v.fit(["an apple a day keeps the doctor away"]).vocabulary_)
{u'an': 0,
u'an apple': 1,
u'apple': 2,
u'apple day': 3,
u'away': 4,
u'day': 5,
u'day keeps': 6,
u'doctor': 7,
u'doctor away': 8,
u'keeps': 9,
u'keeps the': 10,
u'the': 11,
u'the doctor': 12}
明确的词汇表限制了将从文本中提取的术语;词汇没有改变:
>>> v = CountVectorizer(ngram_range=(1, 2), vocabulary={"keeps", "keeps the"})
>>> v.fit_transform(["an apple a day keeps the doctor away"]).toarray()
array([[1, 1]]) # unigram and bigram found
(请注意,停用词过滤是在 n 元语法提取之前应用的,因此"apple day"
.)