ContextIndex.similar_words(word) http://nltk.org/_modules/nltk/text.html#ContextIndex.similar_words将每个单词的相似度得分计算为每个上下文中频率乘积的总和。Text.similar() http://nltk.org/_modules/nltk/text.html#Text.similar只是计算单词共享的独特上下文的数量。
similar_words()NLTK 2.0 中似乎包含一个错误。请参阅中的定义nltk/text.py http://nltk.org/_modules/nltk/text.html#ContextIndex.similar_words:
def similar_words(self, word, n=20):
scores = defaultdict(int)
for c in self._word_to_contexts[self._key(word)]:
for w in self._context_to_words[c]:
if w != word:
print w, c, self._context_to_words[c][word], self._context_to_words[c][w]
scores[w] += self._context_to_words[c][word] * self._context_to_words[c][w]
return sorted(scores, key=scores.get)[:n]
返回的单词列表应按相似度得分降序排序。将 return 语句替换为：
return sorted(scores, key=scores.get)[::-1][:n]
In similar()，调用similar_words()被注释掉了，可能是由于这个错误。
def similar(self, word, num=20):
if '_word_context_index' not in self.__dict__:
print 'Building word-context index...'
self._word_context_index = ContextIndex(self.tokens,
filter=lambda x:x.isalpha(),
key=lambda s:s.lower())
# words = self._word_context_index.similar_words(word, num)
word = word.lower()
wci = self._word_context_index._word_to_contexts
if word in wci.conditions():
contexts = set(wci[word])
fd = FreqDist(w for w in wci.conditions() for c in wci[w]
if c in contexts and not w == word)
words = fd.keys()[:num]
print tokenwrap(words)
else:
print "No matches"
注意：在一个FreqDist，不同于dict, keys()返回一个排序列表。
Example:
import nltk
text = nltk.Text(word.lower() for word in nltk.corpus.brown.words())
text.similar('woman')
similar_words = text._word_context_index.similar_words('woman')
print ' '.join(similar_words)
Output:
man day time year car moment world family house boy child country
job state girl place war way case question # Text.similar()
#man ('a', 'who') 9 39 # output from similar_words(); see following explanation
#girl ('a', 'who') 9 6
#[...]
man number time world fact end year state house way day use part
kind boy matter problem result girl group # ContextIndex.similar_words()
fd，频率分布similar()，是每个单词的上下文数量的统计：
fd = [('man', 52), ('day', 30), ('time', 30), ('year', 28), ('car', 24), ('moment', 24), ('world', 23) ...]
对于每个上下文中的每个单词，similar_words()计算频率乘积的总和：
man ('a', 'who') 9 39 # 'a man who' occurs 39 times in text;
# 'a woman who' occurs 9 times
# Similarity score for the context is the product:
# score['man'] = 9 * 39
girl ('a', 'who') 9 6
writer ('a', 'who') 9 4
boy ('a', 'who') 9 3
child ('a', 'who') 9 2
dealer ('a', 'who') 9 2
...
man ('a', 'and') 6 11 # score += 6 * 11
...
man ('a', 'he') 4 6 # score += 4 * 6
...
[49 more occurrences of 'man']
