如何确定单词的概率？

我有两份文件。 Doc1 的格式如下：

TOPIC:  0 5892.0
site 0.0371690427699
Internet 0.0261371350984
online 0.0229124236253
web 0.0218940936864
say 0.0159538357094

TOPIC:  1 12366.0
web 0.150331554262
site 0.0517548115801
say 0.0451237263464
Internet 0.0153647096879
online 0.0135856380398

...以此类推，直到主题 99 以相同的模式。

Doc2 的格式为：

0 0.566667 0 0.0333333 0 0 0 0.133333 ..........

等等...每个主题总共有 100 个值，每个值。

现在，我必须找到每个单词的加权平均概率，即：

P(w) = alpha.P(w1)+ alpha.P(w2)+...... +alpha.P(wn)

where alpha = value in the nth position corresponding to the nth topic. 

也就是说对于“say”这个词，概率应该是

P(say) = 0*0.0159 + 0.5666*0.045+....... 

同样，对于每个单词，我都必须计算概率。

For  multiplication, if the word is taken from topic 0, then the 0th value from the doc2 must be considered and so on.

我只用下面的代码对单词的出现次数进行了计数，但从未获取它们的值。所以，我很困惑。

 with open(doc2, "r") as f:
    with open(doc3, "w") as f1:

         words = " ".join(line.strip() for line in f)
         d = defaultdict(int)
         for word in words.split():  
              d[word] += 1
              for key, value in d.iteritems() :
                  f1.write(key+ ' ' + str(value) + ' ')
              print '\n'

我的输出应该如下所示：

 say = "prob of this word calculated by above formula"
 site = "
 internet = " 

等等。

我究竟做错了什么？

假设您忽略 TOPIC 行，请使用 defaultdict 对值进行分组，然后在最后进行计算：

from collections import defaultdict
from itertools import groupby, imap

d = defaultdict(list)
with open("doc1") as f,open("doc2") as f2:
    values = map(float, f2.read().split()) 
    for line in f:
        if line.strip() and not line.startswith("TOPIC"):
            name, val = line.split()
            d[name].append(float(val))

for k,v in d.items():
    print("Prob for {} is {}".format(k ,sum(i*j for i, j in zip(v,values)) ))

另一种方法是边做边计算，每次点击新部分（即带有 TOPIC 的行）时增加计数，以通过索引从值中获取正确的值：

from collections import defaultdict
d = defaultdict(float)
from itertools import  imap

with open("doc1") as f,open("doc2") as f2:
    # create list of all floats from doc2
    values = imap(float, f2.read().split())
    for line in f:
        # if we have a new TOPIC increase the ind to get corresponding ndex from values
        if line.startswith("TOPIC"):
            ind = next(values)
            continue
        # ignore empty lines
        if line.strip():
            # get word and float and multiply the val by corresponding values value
            name, val = line.split()
            d[name] += float(val) * values[ind]

for k,v in d.items():
    print("Prob for {} is {}".format(k ,v) )

使用你的两个 doc1 内容和0 0.566667 0 0.0333333 0doc2 内部输出以下两者：

Prob for web is 0.085187930859
Prob for say is 0.0255701266375
Prob for online is 0.0076985327511
Prob for site is 0.0293277438137
Prob for Internet is 0.00870667394471

您还可以使用 itertools groupby：

from collections import defaultdict
d = defaultdict(float)
from itertools import groupby, imap

with open("doc1") as f,open("doc2") as f2:
    values = imap(float, f2.read().split())
    # lambda x: not(x.strip()) will split into groups on the empty lines
    for ind, (k, v) in enumerate(groupby(f, key=lambda x: not(x.strip()))):
        if not k:
            topic = next(v) 
            #  get matching float from values
            f = next(values)
            # iterate over the group 
            for s in v:
                name, val = s.split()
                d[name] += (float(val) * f)
for k,v in d.iteritems():
    print("Prob for {} is {}".format(k,v))

对于 python3 所有迭代工具 https://docs.python.org/2/library/itertools.html imaps应该改为仅map它还返回 python3 中的迭代器。