我有一些文字:
s="Imageclassificationmethodscan beroughlydividedinto two broad families of approaches:"
我想将其解析为单独的单词。我很快地研究了附魔和nltk,但没有看到任何看起来立即有用的东西。如果我有时间在这方面投入,我会考虑编写一个动态程序,用 enchant 来检查单词是否是英语。我本以为可以在网上做一些事情,我错了吗?
使用 trie 的贪心方法
尝试使用这个生物蟒蛇 https://github.com/biopython/biopython (pip install biopython
):
from Bio import trie
import string
def get_trie(dictfile='/usr/share/dict/american-english'):
tr = trie.trie()
with open(dictfile) as f:
for line in f:
word = line.rstrip()
try:
word = word.encode(encoding='ascii', errors='ignore')
tr[word] = len(word)
assert tr.has_key(word), "Missing %s" % word
except UnicodeDecodeError:
pass
return tr
def get_trie_word(tr, s):
for end in reversed(range(len(s))):
word = s[:end + 1]
if tr.has_key(word):
return word, s[end + 1: ]
return None, s
def main(s):
tr = get_trie()
while s:
word, s = get_trie_word(tr, s)
print word
if __name__ == '__main__':
s = "Imageclassificationmethodscan beroughlydividedinto two broad families of approaches:"
s = s.strip(string.punctuation)
s = s.replace(" ", '')
s = s.lower()
main(s)
Results
>>> if __name__ == '__main__':
... s = "Imageclassificationmethodscan beroughlydividedinto two broad families of approaches:"
... s = s.strip(string.punctuation)
... s = s.replace(" ", '')
... s = s.lower()
... main(s)
...
image
classification
methods
can
be
roughly
divided
into
two
broad
families
of
approaches
Caveats
英语中有一些退化的情况,这不起作用。您需要使用回溯来处理这些问题,但这应该可以帮助您开始。
强制性测试
>>> main("expertsexchange")
experts
exchange
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)